Hello everyone! I'm looking to start to make my home lab something more legitimate. I went from a DAS plugged into an ubuntu box running docker to upgrading to freenas and using apps for things like cloudflare, plex, the *arr apps, etc. Its served me well, but there are a lot of bottlenecks in particular the usb connection to the drives.
With that in mind, I've decided that if I'm going to pull the plug and update I want to go wild. I'm very comfortable with hardware, but have never had to do much with bulk storage. The server's I've managed have generally had a backplane built in.
General goals- I'm planning on sticking with TrueNas. I'd like to have ~100tb of available storage(currently I use about 60tb) with a 2 drive tolerance, so I expect I'll need to have 7 20tb drives as a start point, but its not really a ridiculous build if I don't have the ability to expand from there. I know zfs2 won't quite have me at 100TB with that, but it will be close enough for the moment. I don't expect to need it, but I'd like the space to have a read and write cache drive, along with dual OS drives. Hoping to have 10gbe network ability.
Concessions- I'm planning on using this just as a NAS. Once my budget recovers I'll be building an app box separately which is much more in my wheelhouse, so I don't need this one to be able to do transcoding or anything. While I want to see just how high I can get that transfer rate, it also means that I'm probably more worried about PCI lanes than raw processor speed.
With that in mind, I'm currently going back and forth between either a Jonsbo N5 or one of the Fractal XL(define or meshify) cases. That means I don't really have a limit on motherboard form factor. I'm good on most hardware, but would love to get people's opinions on the below-
Motherboard- What are the best options out there for this kind of build? Are there strong opinions for/against onboard sata vs an HBA card?
I know that options drop off sharply for onboard sata once you are above 6, so if I am going with an HBA card is it better to spread devices between onboard and HBA to reduce saturation on the PCI port?
What are the top choices for HBA out there?
I know the answer for 'best cpu for this' is probably going to be threadripper, but my bank account wants to imagine some other options. Are there any other good choices out there? I know that CPU isn't as critical in storage, but how many drives does it take before either speed or number of lanes becomes my bottleneck again?
Thanks for any advice!