Threadripper Home Server

[tictoc]

I have been back-and-forth with ASRock, testing a number of beta BIOS.  The same issue still exists with x8x8 working, but x4x4x4x4 just acts like x16.  It is going up the chain to a BIOS engineer, so it seems as if there might be some progress in the near future on the bifurcation issues.

Edited May 1, 2021 by tictoc

[axipher]

1 hour ago, tictoc said:

I have been back-and-forth with ASRock, testing a number of beta BIOS.  The same issue still exists with x8x8 working, but x4x4x4x4 just acts like x16.  It is going up the chain to a BIOS engineer, so it seems as if there might be some progress in the near future on the bifurcation issues.

 

That's really crap that it's not working, but glad they are on it at least.

 

I got lucky that my Asus Prime X570-Pro supported the 4x4x4x4 mode on the top slot for my 4x NVME M.2 PCIe card without any messing around.

[ENTERPRISE]

Reading through this just makes me want to nerd out lol. One thing I will upgrade is my switch to 10G...I am so exciting

[tictoc]

10 hours ago, axipher said:

 

That's really crap that it's not working, but glad they are on it at least.

 

I got lucky that my Asus Prime X570-Pro supported the 4x4x4x4 mode on the top slot for my 4x NVME M.2 PCIe card without any messing around.

Niche board with a niche use case, but every other Threadripper board I've ran worked without a hitch.  I'm not sure why this doesn't work on this board.  I don't care about using NVMe RAID mode or booting the OS off the drives (RAID will be handled by the OS), just looking for what has become a pretty standard option on most server motherboards.

 

Right now I don't really need this to work, but I figured I might as well try to iron out all the possible kinks before I load the hardware in the box.

[tictoc]

For all you cable management lovers, here's a little preview of my new 12U network/power rack:

[Avacado]

55 minutes ago, tictoc said:

For all you cable management lovers, here's a little preview of my new 12U network/power rack:

It pleases my OCD

[tictoc]

On 01/05/2021 at 15:35, Avacado said:

It pleases my OCD

Just realized that if I swap the switch and the patch panel in the rack, I can keep the wires from crossing and not have to label the patch panel 2,1,4,3,6,5...

 

Now we have achieved maximum OCD relief.

 

Edited July 2, 2021 by tictoc

[tictoc]

Beta BIOS #6 looks to be a winner.  In addition to the x4 split now working, there are also x8x4x4 and x4x4x8 splits for PCIE2/3 and PCIE4/5. This should make it possible to split all the slots, including the two slots that are x8 electrical.

 

I haven't fully tested all the options but this works.

 

Using some old drives to make sure that the final storage configuration will work.

 

Drive layout:

• 6x SATA HDDs on an HBA in PCIe slot 5
• 4x SATA HDDs on the mini-sas connector (chipset)
• 2x SATA SSDs (chipset)
• 1x SATA SSD (Asmedia SATA DOM)
• 4x NVMe SSD on an ASUS Hyper M.2 X16 v2 in PCIe slot 6
• 2x M.2 NVMe SSD (onboard slots)
• 1x U.2 Optane 905p on a PCIe adapter in PCIE slot 3

All of the above is detected and working with 2 GPUs also installed in PCIe slots 2 and 4.  Only empty slot is the top x4 slot which is wired to the chipset.

Edited July 2, 2021 by tictoc
fixed images

[axipher]

Progress is always nice to see

[AllenG]

That is great news they finally got around to fixing it properly. At least they didn't drop your request to fix it like they did for mine, not sure if the tech i was working with just didn't want to deal with it or if my testing notes weren't up to par. Based on your posted tests, that looks good and like everything is finally functioning properly. Hopefully they actually release this bios now.

 

I guess it's time for me to open another ticket to try and get the version you have... the tech i was dealing with hasn't responded in two weeks now.

 

Rack setup is looking good! Good catch on the arrangement to keep cables fanning out proportionally... i see it backwards like that alot in real life actually, funny when you know it was that simple of a change.

 

[tictoc]

Small update on the bifurcation options.  The implementation in the BETA BIOS that I am running seems to be 100% complete.

The slots that are electrically x16 all work in x4x4x4x4 mode, and the x4x4x8 splits also work correctly.

 

This is a huge improvement especially on this platform.  The 2970WX and the 2990WX CPUs have four NUMA nodes, with two NUMA nodes having direct access to memory and certain PCIe slots.  With proper bifurcation on all the slots, there is much more flexibility when running VMs.  It is now much easier to pin high performance VMs to the proper NUMA node, without having to swap devices from slot to slot.

 

The only real limitation now is due to the fact that slots 3 and 5 are only x8 electrically.  At the end of the day it probably doesn't really matter.  With all the bifurcation options working, the use case where this might matter is going to be a niche, of a niche, of a niche, use case, and there are reasonably priced EPYC options available.      

[tictoc]

I am actually making progree on the new server now. 

 

I have migrated all the storage off my old 2P server, and hooked everything up to a 3200G/x570 board on one of my benches.   I am currently archiving a bunch of data, since part of the plan on the down sizing was to trim down on some of my data hoarding.  I may end up needing a few more drives, because I'm not sure I want to offline enough stuff to fit in the 24TB media (usable) pool + 4TB (online critical backups)  that I was planning on.  I'm probably going to grab a few more HDDs, which will bring the total up to 10 HDDs in the chasis + 1 in the hot-swap.

 

New Heatkiller radiators arrived last week, and the case is stripped down and cleaned.  Next step will be to plastidip or paint the scratched up acrylic side panel. I'll post some pics once that is done, and then it will be time to actually start putting everything together in the case and get it off my bench.    

[BWG]

7 hours ago, tictoc said:

I am actually making progree on the new server now. 

 

I have migrated all the storage off my old 2P server, and hooked everything up to a 3200G/x570 board on one of my benches.   I am currently archiving a bunch of data, since part of the plan on the down sizing was to trim down on some of my data hoarding.  I may end up needing a few more drives, because I'm not sure I want to offline enough stuff to fit in the 24TB media (usable) pool + 4TB (online critical backups)  that I was planning on.  I'm probably going to grab a few more HDDs, which will bring the total up to 10 HDDs in the chasis + 1 in the hot-swap.

 

New Heatkiller radiators arrived last week, and the case is stripped down and cleaned.  Next step will be to plastidip or paint the scratched up acrylic side panel. I'll post some pics once that is done, and then it will be time to actually start putting everything together in the case and get it off my bench.    

 

 

I've had great success wet sanding acrylic from 2000, 3000 grit to a da polisher with auto cutting creme, then final polish. Give it a shot if you have the tools.

[tictoc]

On 13/07/2021 at 10:43, BWG said:

 

 

I've had great success wet sanding acrylic from 2000, 3000 grit to a da polisher with auto cutting creme, then final polish. Give it a shot if you have the tools.

That's not a bad idea.  Same method I use to polish my headlights.

[Joe__Bishop]

why people keep giving me ideas how to spend money on something I clearly don't need but I would rather have. haha

 

Seeing this, making me wanna jump start my next project of centralizing my entire home it into one rack and wiring access via cat cable 

eh, one can only dream about such a wonderful set up

[tictoc]

A few pics of the stripped down case before hardware starts going in.

 

 

[Avacado]

What case is that? Looks like a Caselabs S3. 

[pio]

Here I am, just now getting a rack system this last year.  And here you are, going away from a rack into a Threadripper all in one setup.  

Not sure if you like punishment or what, going with water cooling on a server, but it looks absolutely amazing so far.  Them cable management pictures......man oh man. 

[tictoc]

22 hours ago, Avacado said:

What case is that? Looks like a Caselabs S3. 

 

No Caselabs, just a cheap clone. Thermaltake Core x5.

Edited July 25, 2021 by tictoc

[tictoc]

17 hours ago, pioneerisloud said:

Here I am, just now getting a rack system this last year.  And here you are, going away from a rack into a Threadripper all in one setup.  

Not sure if you like punishment or what, going with water cooling on a server, but it looks absolutely amazing so far.  Them cable management pictures......man oh man. 

 

Most of my gear that runs 24/7 is water-cooled.  As long as you follow some best practices like EPDM tubing, distilled+Mayhems biocide and inhibitor, dual pumps in serial, there is not really anything to worry about or any additional maintenance.

 

I haven't touched my workstation in 16 months other than to top off the fluid one time, and it runs 24/7 @ about 1900-2000W.

[pio]

22 minutes ago, tictoc said:

 

Most of my gear that runs 24/7 is water-cooled.  As long as you follow some best practices like EPDM tubing, distilled+Mayhems biocide and inhibitor, dual pumps in serial, there is not really anything to worry about or any additional maintenance.

 

I haven't touched my workstation in 16 months other than to top off the fluid one time, and it runs 24/7 @ about 1900-2000W.

That's why I like forums like this man, learn something new everyday.   Never thought to run dual pumps in serial.  I had only ever learned of simple loops, and that's all I ran before.  Pretty sure I was following good practices still, my last loop did last like 2 years without draining, and sold the whole rig functional.  But yeah.

I just assumed that air was simpler to maintain on something that's on 24/7 365.  I wasn't thinking leaks or anything silly like that.  Was thinking punishment as in its more parts that could potentially fail and lead to downtime of a "server", that's all.  I guess dual pumps in serial kind of negates that point, since you'd have a backup pump already running so at least things still move.  And the biocide + inhibiter I'm assuming would stop things growing better than a killcoil would.  Neat stuff!

Edited July 25, 2021 by pioneerisloud

[Diffident]

40 minutes ago, pioneerisloud said:

That's why I like forums like this man, learn something new everyday.   Never thought to run dual pumps in serial.  I had only ever learned of simple loops, and that's all I ran before.  Pretty sure I was following good practices still, my last loop did last like 2 years without draining, and sold the whole rig functional.  But yeah.

I just assumed that air was simpler to maintain on something that's on 24/7 365.  I wasn't thinking leaks or anything silly like that.  Was thinking punishment as in its more parts that could potentially fail and lead to downtime of a "server", that's all.  I guess dual pumps in serial kind of negates that point, since you'd have a backup pump already running so at least things still move.  And the biocide + inhibiter I'm assuming would stop things growing better than a killcoil would.  Neat stuff!

Even with only one pump, you can survive a pump failure if you have enough rad space and high rpm fans.  I've woken up to a dead pump with the system still chugging along at 100% load.  I've also had an almost complete blockage, but kept the system going for days before I cleaned it out since we were in the middle of the Pentathlon.

[tictoc]

The trend in HPC, especially with high density muti-GPU nodes, is water-cooling all the things. 

 

6U systems like this dual Epyc w/ 8x NVIDIA GPUs are becoming much more common. https://bizon-tech.com/bizon-zx9000.html

 

[pio]

You guys are making me want to water cool again....... 

Good stuff, good stuff.  Can't wait for more updates!

[tictoc]

The last few months have been very busy, so this project has stalled. 

 

Back at it again, and I will post some better pics, but here's a few quick ones of the current progress.

Radiators are in, half of the drive brackets are in, and all of the fans are installed.