I've been running a headless Ubuntu server for about 10 years or so. At first, it was just a file/print server, so I bought a super low power motherboard/processor to cut down on the energy bill. It's a passively cooled Intel Celeron J3455 "maxed out" with 16BG of RAM.
Since then it's ballooned into a Plex/Shinobi/Photoprism/Samba/Frigate/MQTT/Matrix/Piwigo monster. It has six drives in RAID6 and a 7th for system storage (three of the drives are through a PCI card). I'm planning on moving my server closet, and I'll be upgrading the case into a rack-mount style case. While I'm at it, I figured I could upgrade the hardware as well. I was curious what I should look for in hardware.
I've built a number of gaming PCs in the past, but I've never looked at server hardware. What features should I look for? Also, is there anything specific (besides a general purpose video card) that I can buy to speed up video encoding? It'd be nice to be able to real-time transcode video with Plex.
Great advice from everyone here. For the transcoding side of things you want an 8th gen or newer Intel chip to handle quicksync and have a good level of quality. I've been using a 10th gen i5 for a couple of years now and it's been great. Regularly handles multiple transcodes and has enough cores to do all the other server stuff without an issue. You need Plex Pass to do the hardware transcodes if you don't already have it or can look at switching to Jellyfin.
As mentioned elsewhere, using an HBA is great when you start getting to large numbers of drives. I haven't seen random drops the way I've seen occasionally on the cheap SATA PCI cards. If you get one that's flashed in "IT mode" the drives appear normally to your OS and you can then build software raid however you want. If you don't want to flash it yourself, I've had good luck with stuff from The Art of Server
I know some people like to use old "real" server hardware for reliability or ECC memory but I've personally had good luck with quality consumer hardware and keeping everything running on a UPS. I've learned a lot from serverbuilds.net about compatibility works between some of the consumer gear, and making sense of some of the used enterprise gear that's useful for this hobby. They also have good info on trying to do "budget" build outs.
Most of the drives in my rack have been running for years and were shucked from external drives to save money. I think the key to success here has been keeping them cool and under consistent UPS power. Some of mine are in a disk shelf, and some are in the Rosewill case with the 12 hot swap bays. Drives are sitting at 24-28 degrees Celsius.
Moving to the rack is a slippery slope... You start with one rack mounted server, and soon you're adding a disk shelf and setting up 10 gigabit networking between devices. Give yourself more drive bays than you need now if you can so you have expansion space and not have to completely rearrange the rack 3 years later.
Also if your budget can swing it, it's nice keeping other older hardware around for testing. I leave my "critical" stuff running on one server now so that a reboot when tinkering doesn't take down all the stuff running the house. That one only gets rebooted or has major changes made when it's not in use (and wife isn't watching Plex). The stuff that doesn't quite need to be 24/7 gets tested on the other server that is safe to reboot.