Threadripper Home Server

[tictoc]

It has been about 6 years since I've posted anything resembling a build log online.  That ended up being more of a summary due to time constraints, and that log has pretty much gone the way of the dodo.

 

Spoiler

 

Dualin' 540

Current state of that thread:

 

Final pics:

 

 

 

 

This build will be somewhat of an evolution of that machine which housed my daily driver and my HTPC in one case.

 

For this machine I am planning on retiring most of the gear in my rack (ASRock Rack w/ 2x 2680 v2, X99-E WS w/ 2697 v4, ASRock X570M Pro 4 w/ Ryzen 3 3200G) and consolidating down to a single machine running in a Thermaltake Core X5.

 

I already have most of the parts for this build, so I hope to have it on the bench to start some testing this weekend.

 

Parts list

Case - Thermaltake Core X5

Motherboard - ASRock Rack X399D8A-2T

PSU - EVGA Supernova G2 1000W Silverstone Strider Platimum 1200W

CPU - AMD Threadripper 2970WX

RAM - 4x 8x Micron 16GB DDR4-3200 ECC-UDIMM (eventually 8x 16GB)

GPUs - Radeon 5700XT 2x Vega 64 2x Radeon VII

SSDs - 2x Intel Optane 905P 960GB U.2 NVMe, Intel Optane 905P 480GB U.2 NVMe, ADATA SX8200 Pro 1TB NVMe, SK Hynix P31 1TB NVMe, 4x Inland Premium 1TB NVMe, 2x crucial MX500 2TB SATA

HDDs - 2x HGST Deskstar NAS 4TB, 1x Seagate IronWolf Pro 4TB, 5x 1x Seagate EXOS 8TB, 1x Seagate Ironwolf Pro 8TB, 4x 8x Seagate EXOS X18 16TB, 1x 4TB Toshiba N300 (Hot swap bay for backups)

CD/DVD/BluRay - LG WH14NS40 flashed for 4k UHD

 

Cooling

CPU - Watercool HEATKILLER IV Pro Copper

GPU - EK Vector Radeon VII Copper/Acetal

Radiator - 2x XSPC EX360 2x Watercool HEATKILLER 360-L

Pump - 2x Watercool D5 w/ EK-XRES 100 Reservoir Watercool Industrial Dual D5 Top w/ 100mm Watercool HEATKILLER Tube Reservoir

Fans - Silverstone AP183 (front intake), 6x Arctic P14 PWM (rear exhaust, HDD fans, midplane), 6x Artic P12 PWM (radiator fans) 

 

The first potato pic of this log will be a quick shot of the ASRock board.

 

 

Edited August 3, 2023 by tictoc
Updated Hardware

[axipher]

Looking forward to this project, love the black tubing.

[ENTERPRISE]

Nice ! Will be a solid build. Got nosey and checked out that motherboard, not bad and comes with 2x 10Gbe ports which certainly makes the networking side of things a little more futureproof. Any more GPU's going in that beast for folding ?  

[tictoc]

On 11/4/2020 at 8:21 AM, axipher said:

Looking forward to this project, love the black tubing.

It is the only tube I've used for the last 10 years. For the last 6-7 years it's been EK ZMT, and before that it was just some random EPDM tubing.  Best tubing there is. Zero maintenance, no funky plasticizer, blocks all UV light to help minimize any growth, and can be turned at pretty tight angles without collapsing.  The tubing being matte black is the icing on the cake.

 

On 11/4/2020 at 12:23 PM, ENTERPRISE said:

Nice ! Will be a solid build. Got nosey and checked out that motherboard, not bad and comes with 2x 10Gbe ports which certainly makes the networking side of things a little more futureproof. Any more GPU's going in that beast for folding ?  

The pair of X550's is a nice bonus and frees up a PCIe slot.  Just a few GPUs in this machine, one for the HTPC and one to be used in various VMs. Although I'm sure there will be a bit of F@H run from time to time.

Edited November 7, 2020 by tictoc
typos

[ENTERPRISE]

12 hours ago, tictoc said:

It is the only tube I've used for the last 10 years. For the last 6-7 years it's been EK ZMT, and before that it was just some random EPDM tubing.  Best tubing there is. Zero maintenance, no funky platicizer, blocks all UV light to help minimize any growth, and can it be turned at pretty tight angles without collapsing.  The tubing being matte black is the icing on the cake.

 

The pair of X550's is a nice bonus and frees up a PCIe slot.  Just a few GPUs in this machine, one for the HTPC and one to be used in various VMs. Although I'm sure there will be a bit of F@H run from time to time.

Keeping PCIe slots free with having certain items on mobo instead is a big bonus. A. For airflow or B. You can go crazy and bang in loads of add in cards.

[J7SC_Orion]

That's a nice Threadripper home server build you got there !

 

Re. cooling, I have been using a CPU-only loop with a Heatkiller IV Pro Nickel-Copper for a TR2+  2950X @ 4.275 to 4.325 all-core, along with 2x XSPC RX 360s. Absolutely zero problems for two years now, even with heavy use

Edited November 7, 2020 by J7SC_Orion

[tictoc]

I've been running the Heatkiller block since I first jumped on the Threadripper train with a 2970WX on a Taichi board. Currently using the same block on my main machine which is running a 3960X @ 4.25GHz all-core OC 24/7. 

 

This will be running a very mild OC, since my 2970WX is not a great clocker and hits a pretty steep wall at about 3.75-3.8GHz all-core.  The GPU in the loop (Vega 64 for now) will be undervolted, so the pair of 360s should keep things nice and cool.

 

Getting everything set up on air on my bench to start doing some testing today.  It will be a few weeks before the build actually starts going together, since there is a bunch of testing and validation that I need to do before I migrate everything over to this build.  As part of my down-sizing I am purging a ton of data from storage, since I am a recovering data hoarder.

The build itself is pretty straight forward, but the underlying stack will take a bit to iron out.  I am going to be moving all my data from mdadm/xfs to btrfs, and before I do that there will be a fair amount of testing to do.  Also, I am going to do a bit of poking at bcache, since I haven't played around with that in 4ish years.

Edited July 2, 2021 by tictoc

[ENTERPRISE]

I can also attest to the Heatkiller blocks, fantastic ! Managed to keep my 2990WX from going to mental in the heat department back when I had that system. I remember when I first got it and I was like WOAH heavy chunk of metal haha.

[tictoc]

I forgot the post office closed early on Saturday, so I didn't make it down from my house in time to pick up my Optane drive.  Looks like I'll just be tuning RAM this weekend.

 

Tuning RAM is going to be a real pain on this board.  If the RAM fails to train at a given speed/timings, it won't post, and then I have to go through the following process.

 

Power down -> remove two sticks -> power up -> pray to the memory gods for a successful post -> adjust speed/timings -> reboot with two sticks -> verify post -> power down -> install removed sticks -> power on -> cross fingers and pray for post -> boot into OS -> rinse and repeat.  If after pulling two sticks it still fails to post, then it takes a CMOS reset to get back to a state where the board will post. 

 

Running at 2400 with pretty trash timings right now, hopefully I can just take very small steps and avoid the whole shut down remove RAM reset CMOS shenanigans.

Edited July 2, 2021 by tictoc

[J7SC_Orion]

3 hours ago, ENTERPRISE said:

I can also attest to the Heatkiller blocks, fantastic ! Managed to keep my 2990WX from going to mental in the heat department back when I had that system. I remember when I first got it and I was like WOAH heavy chunk of metal haha.

 

...indeed they are - I paid express delivery from Germany to Canada for that weighty cooling wonder 

   

1 hour ago, tictoc said:

I forgot the post office closed early on Saturday, so I didn't make it down from my house in time to pick up my Optane drive.   Looks like I'll just be tuning RAM this weekend.

 

Tuning RAM is going to be a real pain on this board.  If the RAM fails to train at a given speed/timings, it won't post, and then I have to go through the following process.

 

Power down -> remove two sticks -> power up -> pray to the memory gods for a successful post -> adjust speed/timings -> reboot with two sticks -> verify post -> power down -> install removed sticks -> power on -> cross fingers and pray for post -> boot into OS -> rinse and repeat.  If after pulling two sticks it still fails to post, then it takes a CMOS reset to get back to a state where the board will post. 

 

Running at 2400 with pretty trash timings right now, hopefully I can just take very small steps and avoid the whole shut down remove RAM reset CMOS shenanigans.

 

...RAM tuning on server boards / CMOS is fun (> not).

 

But as you know from your prior 2970WX Taichi, it can pay off big time with Zen...just hoping you don't loose your sanity before then, like when you take your morning toast from the toaster and it feels just like pulling RAM sticks off a server mobo to you

Edited November 7, 2020 by J7SC_Orion

[tictoc]

I'm probably just going to end up pretending that all the Threadripper memory settings don't exist in the BIOS, and just treat it like my C602 board.

 

That board has a nice and easy memory OC. Set speed to 1866MHz, up memory voltage until it's stable, and then just be content that you're running cheap 1333MHz RDIMMs at 1866MHz.

Edited July 2, 2021 by tictoc
typos, fix emoji

[tictoc]

It's alive!!

 

 

Just wrapping up 10 hours of stressapptest on the memory at 2800MHz.

Edited July 2, 2021 by tictoc

[ENTERPRISE]

17 hours ago, tictoc said:

It's alive!!

 

 

Just wrapping up 10 hours of stressapptest on the memory at 2800MHz. 

 

Woop !

[J7SC_Orion]

18 hours ago, tictoc said:

It's alive!!

 

 

Just wrapping up 10 hours of stressapptest on the memory at 2800MHz. 

 

Nice, grats !

[tictoc]

Currently I am testing my RAM at 3200 16-20-16-40.  So far it looks to be stable. 

 

The ICs on these no-name ECC 3200 sticks are Micron rev.E, so now that I've already gone this far, I guess I'll see just how far I can push them while maintaining 100% stability. 

Edited November 14, 2020 by tictoc

[Bastiaan_NL]

Subbed, and good luck with the mem tuning!
I never liked doing that, and I don't think I ever will

[tictoc]

I wouldn't have even started messing with the memory, but originally I couldn't get the board to post with the 3200MHz ECC sticks.  To get it to post the first time, I had to put in a couple of non-ECC sticks.  Once I did that, I was able to power down and swap in a pair of the ECC sticks.  That got me into the UEFI, but it would almost instantly crash because the speed/timings were way off from what the sticks wanted to run at.

 

After the initial brain damage of finding a starting point for the RAM, it has actually been fairly simple to up the speed and tighten the timings.  Two big gotchas for this board/memory are that the board will not even post if TRCD is set below 20, and almost any speed will fail to post or be wildly unstable with Command Rate set at 1T.

[tictoc]

64GB (4x16GB) ended up being 100% stable at 3200|16-20-16-38|1.395v.

 

128GB (8x 16GB) is not stable at the same timings. I have a working (and at least mostly stable) profile for 2933.  Unless it will run at 3200 CL18, which is highly unlikely from my past experience of trying to run 3200 with 128GB on my Taichi board, I'll be running it at 2933.

 

 

Edited July 2, 2021 by tictoc

[ENTERPRISE]

19 hours ago, tictoc said:

64GB (4x16GB) ended up being 100% stable at 3200|16-20-16-38|1.395v.

 

128GB (8x 16GB) is not stable at the same timings. I have a working (and at least mostly stable) profile for 2933.  Unless it will run at 3200 CL18, which is highly unlikely from my past experience of trying to run 3200 with 128GB on my Taichi board, I'll be running it at 2933.  

 

 

For a 128Gb, that is not bad performance mate.  Do you actually use that much or could you go down to 64gb and harness the additional performance in speeds ? Not sure if what you run is memory performance sensitive or just the fact you just need as much RAM as you can get.

[tictoc]

As much as I can get, so even if I have to drop down to something like 2733 or 2800 with cleaned up timings that would be fine.

Edited November 17, 2020 by tictoc

[tictoc]

Not much happening other than continuing some extended testing.

 

 

Right now I'm doing some misc performance/reliability testing on btrfs, and trying to decide what the final storage layout will be.

 

Also, I'll be going with a 200mm Noctua fan in the front panel of the case, which will hopefully move enough air to keep things cool.  Only thing that I might need to address temp wise is the VRMs.  With the goal being silence I'd rather not add a fan mid case to cool the VRMs.  The server will see fairly high load compiling and running test applications, but with no OC, VRM temps should be fine.  If not, then I can always add one of my Heatkiller SW-X 80 DIYs to the loop.  I used a SW-X 60 on my Taichi X399 board, and it cooled the VRMs without a hitch.   

[tictoc]

Nothing too exciting, just making up some new cables for the drives.

 

 

I will be starting to backup/transfer the data on the current server and start migrating the data to the new drive setup this weekend.

Edited July 2, 2021 by tictoc

[Bastiaan_NL]

That looks clean, unlike the mess you would have with the original cables. 
Have fun with the data migration

[tictoc]

10Gb networking makes pretty quick work of transferring data around.  I'll probably spin up a temporary 6-8TB SSD RAID-0 to pull the data off the server, add the old drives to the new storage pool, and then it's just a local rsync to put the data in the new storage pool.  Since I purged a bunch of stuff off my current file server to cold archives, I should be able to get everything moved in 2 batches.  

Edited November 28, 2020 by tictoc

[tictoc]

Just a little update.  It looks like this is going to turn into a very overkill server.  Thinking about swapping out a few things on my workstation, so I might be adding a pair of Radeon VIIs to this machine.  For now just testing with a pair of Vega 64s.

 

Edited July 2, 2021 by tictoc