Threadripper Home Server

[tictoc]

On 10/05/2021 at 22:54, AllenG said:

That is great news they finally got around to fixing it properly. At least they didn't drop your request to fix it like they did for mine, not sure if the tech i was working with just didn't want to deal with it or if my testing notes weren't up to par. Based on your posted tests, that looks good and like everything is finally functioning properly. Hopefully they actually release this bios now.

 

I guess it's time for me to open another ticket to try and get the version you have... the tech i was dealing with hasn't responded in two weeks now.

 

Rack setup is looking good! Good catch on the arrangement to keep cables fanning out proportionally... i see it backwards like that alot in real life actually, funny when you know it was that simple of a change.

 

 

@AllenG ASRock Rack has released an official BIOS with the additional PCIe bifurcation options.  I haven't tested it yet, to see how it compares to the beta BIOS I got from ASRock, but I assume it is basically the same.  

 

I will actually be moving forward with this build over the next few days, so if anyone is still interested, there will actually be some progress incoming.

[tictoc]

The latest BIOS looks to be an identical copy to the beta BIOS that ASRock sent me back in May. 

I was actually able to save my settings before flashing to the new BIOS, and all of the settings were intact after the BIOS update.

Edited November 8, 2021 by tictoc

[ENTERPRISE]

16 hours ago, tictoc said:

The latest BIOS looks to be an identical copy to the beta BIOS that ASRock sent me back in May. 

I was actually able to save my setting before flashing to the new BIOS, and all of the settings were intact after the BIOS update.

Loving how this build is progressing so far. Really like your 12U Cabinet cable management. My OCD is satisfied haha.

[tictoc]

Progress has been slow to zero over the last few months.  I did get the pumps installed, and plumbed up the res, pumps, and radiators last night.  Hopefully I'll drop the board in tonight and get the CPU and GPU in the loop for leak testing.

 

The down-sizing from my 42U rack to the 12U rack is mostly complete.  Once I have the server done I'll finish up the rack which will be housing the following gear: two 2U UPS, one 1U UPS, one 24 port patch panel, one 12 port patch panel, 24 port 1G switch, 8 port 10G switch, 1.5U router/firewall, and 1U PDU.  I'll probably throw up a thread on the router/firewall and the build-out for the rack once this machine is up and running.

Edited November 8, 2021 by tictoc

[ENTERPRISE]

29 minutes ago, tictoc said:

Progress has been slow to zero over the last few months.  I did get the pumps installed, and plumbed up the res, pumps, and radiators last night.  Hopefully I'll drop the board in tonight and get the CPU and GPU in the loop for leak testing.

 

The down-sizing from my 42U rack to the 12U rack is mostly complete.  Once I have the server done I'll finish up the rack which will be housing the following gear: two 2U UPS, one 1U UPS, one 24 port patch panel, one 12 port patch panel, 24 port 1G switch, 8 port 10G switch, 1.5U router/firewall, and 1U PDU.  I'll probably throw up a thread on the router/firewall and the build-out for the rack once this machine is up and running.

Darn, you went from 42U to 12U, You must have had a good old clear out !

[tictoc]

No actual build progress, but I have had most of the software stack up and running for testing. 

 

This hit a bit of a stall as I ran into some networking performance regressions with newer kernels.  I had previously been running on the 5.10.xx LTS kernels.  There are some btrfs improvements in newer kernels that haven't been back-ported to the LTS kernel, so I decided to switch over to the stable kernel.  This led me down the rabbit hole of figuring out why my 10Gbps ports were maxing out at 1Gbps.

 

I have bisected the kernel back to the offending commit, and it is a bit of an oddball. It looks to probably be a Threadripper specific issue, due to some timer/clocksource changes.  Reverting this commit: https://lore.kernel.org/all/162437391793.395.1913539101416169640.tip-bot2@tip-bot2/  allows the 10Gbps ports to operate at full speed.  I'm playing around with it this evening, and I'll submit a bug report and/or a patch sometime tomorrow. 

 

I have a few 10Gbps NICs with the same controller as the onboard ports on the X399D8A-2T.  I tested those NICs on two other systems, and those systems did not have the same issue as my Threadripper X399 machine.  The regression occurs with the onboard 10Gbps ports and with the PCIe X550-T2 NIC on the Threadripper machine. 

[J7SC_Orion]

2 hours ago, tictoc said:

No actual build progress, but I have had most of the software stack up and running for testing. 

 

This hit a bit of a stall as I ran into some networking performance regressions with newer kernels.  I had previously been running on the 5.10.xx LTS kernels.  There are some btrfs improvements in newer kernels that haven't been back-ported to the LTS kernel, so I decided to switch over to the stable kernel.  This led me down the rabbit hole of figuring out why my 10Gbps ports were maxing out at 1Gbps.

 

I have bisected the kernel back to the offending commit, and it is a bit of an oddball. It looks to probably be a Threadripper specific issue, due to some timer/clocksource changes.  Reverting this commit: https://lore.kernel.org/all/162437391793.395.1913539101416169640.tip-bot2@tip-bot2/  allows the 10Gbps ports to operate at full speed.  I'm playing around with it this evening, and I'll submit a bug report and/or a patch sometime tomorrow. 

 

I have a few 10Gbps NICs with the same controller as the onboard ports on the X399D8A-2T.  I tested those NICs on two other systems, and those systems did not have the same issue as my Threadripper X399 machine.  The regression occurs with the onboard 10Gbps ports and with the PCIe X550-T2 NIC on the Threadripper machine. 

 

I look forward to see what comes out of the LTS kernel 10 Gbps / X399 'rabbit hole' you described ! I just got the external 1 Gbps (symmetric up / down) installed and (mostly) working in my home office with fixed IP (work related), dynamic IP (the rest) as well as tv-over-internet.

Down the line, symmetric 2.5 Gbps external network might become an option here as well, and I am looking at upgrading my switches and routers to at least 2.5 Gbps if not 10 Gbps. I also want to add a separate home network at 10 Gbps on either MSI X399 Creation or Asus X79 E-WS w/ multiport 10 Gbps add-in cards. For now, decent 10 Gbps value routers etc are still a rare find, though 10 Gbps add-in multi-port cards are a different matter. 

 

Switching gears but somewhat related, speaking of MSI X399 Creation or Asus X79 E-WS, what are your thoughts re. PCIe NVMe add-in cards ? The X399 Creation came with a PCIe 3.0 -based M.2 XPANDER-AERO (4x M.2,fan-cooled, extra PCIe 6 pin) and I am planning to run 4x2 TB NVMe in that in 2x raid 1, and 2x1 TB M.2 NVMe on the mobo in raid 1 as OS (likely Windows Server 2019 though not certain yet). Too complex a plan ? Also, it should be robust.

[tictoc]

For now I just reverted the commit with a patch, and I'm using that patch to build/run stable and mainline kernels for testing.  I also have a few Aquantia 10Gbps NICs, and I'll throw one of those in the server tomorrow to see if it suffers from the same issue as the Intel NICs.

 

Currently I am running 4x 1TB NVMe on an ASUS Hyper m.2 card, in md RAID-0 for a steamcache, squid, and other "victim" data.  The OS is running on 2x 1TB NVMe drives in a btrfs RAID1 array.  I'm not well versed in running something like that on Windows, so I'm not sure how complex and robust it would be.  I know that there were/are? quite a few issues with the fake raid (motherboard/BIOS) on AMD but I've never used it.      

[J7SC_Orion]

5 hours ago, tictoc said:

For now I just reverted the commit with a patch, and I'm using that patch to build/run stable and mainline kernels for testing.  I also have a few Aquantia 10Gbps NICs, and I'll throw one of those in the server tomorrow to see if it suffers from the same issue as the Intel NICs.

 

Currently I am running 4x 1TB NVMe on an ASUS Hyper m.2 card, in md RAID-0 for a steamcache, squid, and other "victim" data.  The OS is running on 2x 1TB NVMe drives in a btrfs RAID1 array.  I'm not well versed in running something like that on Windows, so I'm not sure how complex and robust it would be.  I know that there were/are? quite a few issues with the fake raid (motherboard/BIOS) on AMD but I've never used it.      

  
Thanks ! 

[AllenG]

Sounds like it could be a bug in how the newer kernel calls for pci express lanes to be throttled or cut back (power management related). I've noticed in the past that x550's will work on too little lanes, but they wont do 10g. I've seen the x550's do some odd things, tend to favor the x540's still as those have always been rock solid for me.

[tictoc]

So, I had a few minutes this morning to play around with the commit that I had reverted to solve the networking issue.  The core of the problem is that the tsc clocksource was being marked unstable at boot, which caused the kernel to fall back to hpet, resulting in the abysmal network performance. 

 

Initially I just increased the MAX_SKEW back to 100µs and the problem went away.  I then played with it a bit and found that 70µs was sufficient to keep the kernel from switching to hpet.  Getting ready to file a bug report, and it looks like there are some patches that have been worked on over the last few weeks.  For now I'll just roll with my small patch and see what the final patches look like. https://lkml.org/lkml/2021/11/18/938

 

 

[tictoc]

Small update.

 

Pumps installed, and tubing ran for everything except for pump->CPU and GPU->radiator.   Not sure if I want to keep the drain on the pump inlet or swap it to the outlet.   Either way, there will be some tipping/flippiing of the case in order to get the CPU and GPUs drained for the inevitable GPU swap out.

 

 

 

Looking to pull the machine off my test bench later today.  Then I'll crack open the CPU/GPU blocks and give them a good cleaning.  After that I'll get the system installed into the case, leak test it, and then give it a final clean and flush with Blitz Part 2.  I might actually have this thing up and running in the next couple of days.

[tictoc]

I didn't get this up and runing, but I do have everything except for the wiring done.  I will post some pics when I get home this evening.

 

Part of my goal on downsizing was to offline some data from my main file server, but I didn't get rid of as much data as I thought I would. 

Final main storage pool will be an 8 drive btrfs raid10 with 4x 8TB Seagate EXOS and 4x16TB Seagate EXOS.  I'll update the OP, once I finalize the rest of the storage.

Edited January 26, 2022 by tictoc

[tictoc]

Ready to leak test.  All the difficult to access wiring is done, and I'll slap the drives in after it gets a leak test and Blitz Part 2.

 

This case really isn't big enough for everything that is going in it, but since I've come this far, I'm just going to go ahead with it for now.  I'll probably end up dropping everything in a different case in the not too distant future.

 

Here's a few not so great pics.

 

 

 

Edit: Looking at the pics, and I just noticed that one of the hdd fans is backwards.

Edited February 1, 2022 by tictoc

[tictoc]

The never ending build log has been resurrected.  This is without a doubt the longest I have ever spent on a build without really changing much from the initial config.  

 

Leak test done, and currently bleeding the air out of the system.

 

[tictoc]

After getting the 10 HDDs mounted, I decided to do some temperature testing to set the fan curve for the case fans.  Had a derp moment while putting the front cover back on, and now I'll be installing a new front intake fan.

 

  

It might actually be a good thing, since I ordered a Silverstone AP183 which should push quite a bit more air than the Noctua.

 

I'll be cutting out all the unnecessary mounting points behind te fan which should help to increase airflow and reduce noise.

[Diffident]

That's crazy.  I've never broken blades off of a fan.  

 

Don't worry about fan curves, servers are supposed to sound like jet engines.

[tictoc]

1 hour ago, Diffident said:

That's crazy.  I've never broken blades off of a fan.  

 

Don't worry about fan curves, servers are supposed to sound like jet engines.

 The fan was spinning at 800 rpms when "someone" decided to put the front cover on the case, and might have inserted one of the cover standoffs directly into the path of the fan blades.

 

All the jet engines are going out to pasture, and I will basically end up with nearly the same overall performance and sweet whisper silence. 

[tictoc]

New front intake fan installed, and it moves a bunch more air than the Noctua. 

 

 

Only two SATA SSDs left to mount, but I will need to make one more extension for the SATA power for those drives.

 

Edited April 21, 2022 by tictoc

[tictoc]

Everything is up and running, and it will be getting a two week long stress test with the CPU and GPUs running 24/7, all out, for the BOINC Pentathlon.

 

Setting up the whole software stack for the server will have to wait, so for now it's just running a minimal headless Arch setup for crunching.

 

Edited June 16, 2022 by tictoc

[tictoc]

One more NVMe drive.  

 

 

It took me a minute to get the second drive to show up.  The PCIe bifurcation options are indeed all working, but the labeling in the UEFI is a little misleading.  

 

Build will be complete this weekend.  Swapping out the Vega 64's for a pair of Radeon VII's.  Once that is complete I'll post up some final pics. 

Edited June 16, 2022 by tictoc

[SamsTechStuff]

Forgot about this build, rediscovering cool project is always fun . What are you doing with this now? Definitely makes me wish I went in deeper into my TR build ha.

[tictoc]

There will be some final updates incoming.  This is now all built and mostly tested, and will be put into operation over the next week or two.

[tictoc]

Here is the almost final build. 

 

Missing in these pics is all of the the final HDDs and SATA SSDs, and I am also going to drain the loop to add some QDCs at the GPUs.  I meant to add the QDCs in originally, and then I got in a bit of a rush to get everything up and running for the Pent and spaced it out.  One of the NVMe boot drives is under the GPUs, which cannot be removed without disconnecting the blocks due to the limited clearance between the GPUS and the radiators.

 

 

 

 

[Avacado]

It's a beast for sure. I love that case. 

Edited August 10, 2022 by Avacado