Threadripper Home Server

[J7SC_Orion]

16 minutes ago, tictoc said:

Just a little update.  It looks like this is going to turn into a very overkill server.  Thinking about swapping out a few thing on my workstation, so I might be adding a pair of Radeon VIIs to this machine.  For now just testing with a pair of Vega 64s.

 

  

overkill = moar = betta 
   

[ENTERPRISE]

On 12/9/2020 at 3:33 AM, J7SC_Orion said:

  

overkill = moar = betta 
   

What GPU's are those again ?

[J7SC_Orion]

6 hours ago, ENTERPRISE said:

What GPU's are those again ?

   

980 Classies ...those four found new pastures (pair each) in recent retro-builds after getting their air-cooler dress back on...

  

  

Edited December 14, 2020 by J7SC_Orion

[ENTERPRISE]

17 hours ago, J7SC_Orion said:

   

980 Classies ...those four found new pastures (pair each) in recent retro-builds after getting their air-cooler dress back on...

  

  

Those were great GPU's back in their day !

[tictoc]

One less than stellar stick of RAM.

 

That stick won't really run at anything over 2666.  Rolled the dice on a new pair of sticks, and we'll see what ICs I get.

 

[axipher]

13 hours ago, tictoc said:

One less than stellar stick of RAM.

 

That stick won't really run at anything over 2666.  Rolled the dice on a new pair of sticks, and we'll see what ICs I get.

 

 

Hopefully you can get that working, have you tried just upping the voltage on the memory then bringing it back down?  Sometimes that can just magically fix a DRAM stick.  I know it's not actually magic, but it has worked for me in the past.

[tictoc]

On 12/19/2020 at 3:06 PM, axipher said:

 

Hopefully you can get that working, have you tried just upping the voltage on the memory then bringing it back down?  Sometimes that can just magically fix a DRAM stick.  I know it's not actually magic, but it has worked for me in the past.

 

I've ran it all the way up and down from 1.25v to 1.6v while testing. That stick just doesn't want to run stable at anything over 2666, even at CL22.  

[axipher]

9 hours ago, tictoc said:

 

I've ran it all the way up and down from 1.25v to 1.6v while testing. That stick just doesn't want to run stable at anything over 2666, even at CL22.  

 

That's a real pain...  The last time I had RAM issues was on my X6-1100T, one bad stick that would not run at stock voltage and unfortunately you don't get per-module voltage settings on pretty much any consumer motherboards.

 

Nowadays, I just take PSU and RAM to be the two most stable parts in a system as they have given me near-zero issues over 15 years of building.

[tictoc]

Two new sticks are set to be delivered on Wednesday, so hopefully I can get it sorted out with the new sticks.  One of the other sticks also showed a single error, but I'm pretty sure that only showed up when trying to get 128GB to run at 3200.

 

I already have mis-matched sticks, so more than likely even if the replacement is good, I will be maxed out at 2866-2933.  Four of the sticks (including the bad one) have some funky Micron Rev H? chips, that I'm pretty sure are some sort of ES ICs.  This late in the life cycle of DDR4 you start to see some funny stuff getting put on modules.  Especially, with something like ECC UDIMMs which are not really made in real quantity.

[tictoc]

I could have saved my self hours of testing, if I would have just replaced the garbage stick sooner.

 

Swapped out the bad stick for one of the new sticks, loaded a 2866 CL16 profile that I had saved, and I am 30 minutes into a stressapptest run with zero errors.  With the old stick in I would start to see errors within 20 seconds of starting a stress test.  

[axipher]

13 minutes ago, tictoc said:

I could have saved my self hours of testing, if I would have just replaced the garbage stick sooner.

 

Swapped out the bad stick for one of the new sticks, loaded a 2866 CL16 profile that I had saved, and I am 30 minutes into a stressapptest run with zero errors.  With the old stick in I would start to see errors within 20 seconds of starting a stress test.  

 

Woohoo

[ENTERPRISE]

5 hours ago, tictoc said:

I could have saved my self hours of testing, if I would have just replaced the garbage stick sooner.

 

Swapped out the bad stick for one of the new sticks, loaded a 2866 CL16 profile that I had saved, and I am 30 minutes into a stressapptest run with zero errors.  With the old stick in I would start to see errors within 20 seconds of starting a stress test.  

 That my friend is always the bloody way lol. We I.T people can be stubborn haha.

[Bastiaan_NL]

Good to hear that it's working now!

[tictoc]

This build is still alive, but it's stuck on my test bench for now while I work out the software stack and make some final decisions on hardware. 

New RAM has been dead stable for 3 weeks running at 2933 CL19.

 

Still not sure which way to go on the GPUs in the server.  I had plans to swap some hardware around, but the lack of GPU availability has kind of slowed progress.

[axipher]

I'm currently looking at storage upgrades for mine, at this point new Threadripper also looking mighty fine...

[tictoc]

I'll probably have to upgrade my bulk media storage sooner rather than later.  I grabbed a few remastered Blu-ray box sets, and that is going to eat up a few TBs of storage.  On a side note the X-Files Blu-rays look really good.  Especially since Season 1 and 2 I had originally ripped from VHS.

Edited July 2, 2021 by tictoc

[J7SC_Orion]

3 hours ago, tictoc said:

I'll probably have to upgrade my bulk media storage sooner rather than later.  I grabbed a few remastered Blu-ray box sets, and that is going to eat up a few TBs of storage.  On a side note the X-Files Blu-rays look really good.  Especially since Season 1 and 2 I had originally ripped from VHS   

 

I ripped my X-Files on S-VHS  - 400 lines of horizontal resolution - WOW.

 

On the other hand, 400 lines of resolution these days are sort of like...

Spoiler

 

   

Edited January 19, 2021 by J7SC_Orion

[tictoc]

I am hoping to get this build kick started this weekend.  I have basically just given up on GPUs for now, so this will probably just be running a pair of Vega 64s until hardware availability frees up.

 

My 42U rack has been stripped of everything but network gear, and I have sold most of the gear that was in the rack.  I am downsizing to a small 12U rack for my network gear (router, 10G switch, 1G PoE switch) and power distribution (PDU and UPS's).  I'll include some pics of that once I get things set up. 

[AllenG]

Solid server there. I'm running a similar setup, aircooled with a 1920x downclocked and locked to 3.2ghz w/ ECC ram @ cl17 2866. Runs my whole home setup, a few virtualized routers, pbx, gameserver, and a nas\dlna with an lsi passed through to it. Whole setup sits at about 100w from the wall that way. The 10G integrated was a massive bonus, especially for the price of the board when i got it. A taichi + x550 would have been almost $480 anyways. Been in service about a year... not one hiccup, love it. Replaced a dual x5650 with it, night and day diff in every aspect. Less power, less heat, boots quicker, and is quite a bit snappier.

 

Only troublesome things i have found are:

1) This board lacks is proper pci-e bifurcation options for using pci-e x16 4x nvme cards.  Maybe if two of us bug them about it at the same time, we can get them to release a new bios with those options available. Im super surprised they overlooked this on a board of this caliber.

 

2) It doesn't play nice when trying to run with Windows 7, 8, 8.1, or 10 LTSB natively though. 7 & 8 will run, but almost every reboot it will re-install all your hardware. 8.1 and 10 LTSB flat out wont boot, they sit spinning forever at bootup. LTSC and newer worked fine, all Linux and Unix I tried worked fine too. All those OS that wont run natively, will run via virtualization just fine though.

 

Interested to hear more out of your further testings.

 

[tictoc]

On 4/6/2021 at 8:53 PM, AllenG said:

Solid server there. I'm running a similar setup, aircooled with a 1920x downclocked and locked to 3.2ghz w/ ECC ram @ cl17 2866. Runs my whole home setup, a few virtualized routers, pbx, gameserver, and a nas\dlna with an lsi passed through to it. Whole setup sits at about 100w from the wall that way. The 10G integrated was a massive bonus, especially for the price of the board when i got it. A taichi + x550 would have been almost $480 anyways. Been in service about a year... not one hiccup, love it. Replaced a dual x5650 with it, night and day diff in every aspect. Less power, less heat, boots quicker, and is quite a bit snappier.

 

Only troublesome things i have found are:

1) This board lacks is proper pci-e bifurcation options for using pci-e x16 4x nvme cards.  Maybe if two of us bug them about it at the same time, we can get them to release a new bios with those options available. Im super surprised they overlooked this on a board of this caliber.

 

2) It doesn't play nice when trying to run with Windows 7, 8, 8.1, or 10 LTSB natively though. 7 & 8 will run, but almost every reboot it will re-install all your hardware. 8.1 and 10 LTSB flat out wont boot, they sit spinning forever at bootup. LTSC and newer worked fine, all Linux and Unix I tried worked fine too. All those OS that wont run natively, will run via virtualization just fine though.

 

Interested to hear more out of your further testings.

 

 

I'll shoot ASRock a message.  I was able to get a beta BIOS for my X399 Taichi with proper PCIe bifurcation a few months before they released the official BIOS with proper bifurcation options (x4x4x4x4; x4x4x8, x8x8, x4x4) for all the slots.

 

Ultimately this machine will be running pretty much everything for my home.

• VM with GPU pass-through for my main TV
• Multiple testing VMs (Debian, Ubuntu, CentOS Stream w/ and w/o GPU pass-through)
• Local mirror and build server for all my Arch boxes
• Build server for my Gentoo and Arch Pi's
• Containers or VMs for: NextCloud, lan-cache, unifi, VPN, matrix, irc, and cameras
• File and backup server running btrfs (48TB-RAID-10 and 8TB-RAID-1)
• Monitoring and Log server - TICK/TIG and possibly Prometheus

Currently I am working through some bugs/regressions on ROCm, but once I get that ironed out, I will hopefully get this machine off the test bench and into it's new home.

 

*EDIT* @AllenG You should send ASRock Rack a support message. http://event.asrockrack.com/tsd.asp   There is a beta BIOS floating around with more bifurcation options.  I haven't tested it yet.

 

Edited April 20, 2021 by tictoc

[tictoc]

Flashed to a beta bios with additional PCIe bifurcation options.  I've yet to test the additional options other than double checking the link speed.  No odd splits like x4x4x8 or lower like I have on some intel platforms, but 4x or 8x NVMe for a smoking fast db server is now possible.

 

Edited July 2, 2021 by tictoc

[Avacado]

5 minutes ago, tictoc said:

Flashed to a beta bios with additional PCIe bifurcation options.  I've yet to test the additional options other than double checking the link speed.  No odd splits like x4x4x8 or lower like I have on some intel platforms, but 4x or 8x NVMe for a smoking fast db server is now possible. 

 

This interests me. 

[AllenG]

Yeah, i ended up getting that beta bios too after e-mailing a few times. Unfortunately in my case, the functions don't seem to work correctly. The closest thing i can get working is dividing slot PCIE 6 into x8 x8... this sees nvme slot 1 & 3 of a 4x nvme aic such as the asrock or asus. x4x4x4x4 does not work, stays in x16 mode.

 

The other slots (PCIE 2/3 and 4/5) are actually already bifurcated by asrock, so i am not surprised i don't get working results out of them. All 4 of those X8 slots actually stem from 2 x16 that has been split into two x8's each to make the 4 slots. Now PCIE 6 on the other hand is a full native X16 slot without any of the bifurcation going on to make it into two slots, so really they could get that one working fully... but i think they just copied some code from the Taichi that is close, but not quite right and don't want to check it further. This last part (PCIE 6) is really the ticket to getting a 4x nvme card running... to be honest having bifurcation options on those other shared slots wouldn't really function correctly anyways from what i can tell as it is always auto detecting for a device in slave slot and changing bifurcation options on its own, every boot. My testing below also shows some oddities on the shared slots, so i'm wondering if they didn't wire things a little non-standard for the slave slots... making giving us a x4 x4 x4 x4 out of the masters of those master/slave ports physically impossible? Not entirely sure, all i know is it definitely didn't work as you'd think it would or should. LOL

 

For those interested, here is what i have found and let Asrock know... but they still haven't said anything back after about 2 weeks. Kinda seems like they want to drop the issue even though its not fully fixed. Kind of a long list of testing and outcomes... but hey, might help.

 

Spoiler

Tools I had available to test:
1 x Asus Hyper M2 x16 card w/ 3x nvme ssd's on slots 1-3
1 x Single M2 x4 NVME Aic w/ 1 nvme ssd.

 

-PCIE 6
x16 works correct
x8 x8 works correct, sees nvme slot 1 & 3 of aic.
x4x4x4x4 does not work, stays in x16 mode

 

-PCIE 2/3
x4x4x4x4 does not work, only one of pcie slot can be used at a time in this config it doesnt seem to care which.
Testing in PCIE2(master x16) sees only nvme slot 1 of aic.
Testing in PCIE3(slave x8) sees only nvme slot 2 of aic. (will not see first x4 in this config, only 2nd x4 of the x8 in slot. Confirmed this by using the additional single x4 nvme aic)

x8x8 mode does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

Auto Switch mode also does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

 

-PCIE 4/5 is the exact same behavior as PCIE 2/3.
x4x4x4x4 does not work, only one of pcie slot can be used at a time in this config it doesnt seem to care which.
Testing in PCIE4(master x16) sees only nvme slot 1 of aic.
Testing in PCIE5(slave x8) sees only nvme slot 2 of aic. (will not see first x4 in this config, only 2nd x4 of the x8 in slot. Confirmed this by using the additional single x4 nvme aic)

x8x8 mode does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

Auto Switch mode also does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

 

-PCIE6 is close to working, since it has no slave slots attached to it getting that one fully working with 4x4 mode would be a big help alone. I can tell dividing up the other two x16 master slots (PCIE2, PCIE4) which have x8 slave slots(PCIE3, PCIE5) might be harder or maybe impossible to do. Those two slot sets definitely took a big step backwards and were better left working the way they had been previously... if it's a trade-off issue. I understand it will have to disable the slave x8 slots in order to run these two master x16's in 4x4 mode.

 

(This was the meat of the correspondence on testing to Asrock)

 

 

 

Edited April 24, 2021 by AllenG

[BWG]

I was looking at the board pics and scratching my head on their choice of colors for an AMD board. Kinda glad to see they were hidden after you installed things. I see you're a fan of monsoon fittings too. I really enjoy their stuff. It's different and easy to work with.

[tictoc]

 

3 hours ago, AllenG said:

Yeah, i ended up getting that beta bios too after e-mailing a few times. Unfortunately in my case, the functions don't seem to work correctly. The closest thing i can get working is dividing slot PCIE 6 into x8 x8... this sees nvme slot 1 & 3 of a 4x nvme aic such as the asrock or asus. x4x4x4x4 does not work, stays in x16 mode.

 

The other slots (PCIE 2/3 and 4/5) are actually already bifurcated by asrock, so i am not surprised i don't get working results out of them. All 4 of those X8 slots actually stem from 2 x16 that has been split into two x8's each to make the 4 slots. Now PCIE 6 on the other hand is a full native X16 slot without any of the bifurcation going on to make it into two slots, so really they could get that one working fully... but i think they just copied some code from the Taichi that is close, but not quite right and don't want to check it further. This last part (PCIE 6) is really the ticket to getting a 4x nvme card running... to be honest having bifurcation options on those other shared slots wouldn't really function correctly anyways from what i can tell as it is always auto detecting for a device in slave slot and changing bifurcation options on its own, every boot. My testing below also shows some oddities on the shared slots, so i'm wondering if they didn't wire things a little non-standard for the slave slots... making giving us a x4 x4 x4 x4 out of the masters of those master/slave ports physically impossible? Not entirely sure, all i know is it definitely didn't work as you'd think it would or should. LOL

 

For those interested, here is what i have found and let Asrock know... but they still haven't said anything back after about 2 weeks. Kinda seems like they want to drop the issue even though its not fully fixed. Kind of a long list of testing and outcomes... but hey, might help.

 

  Reveal hidden contents

Tools I had available to test:
1 x Asus Hyper M2 x16 card w/ 3x nvme ssd's on slots 1-3
1 x Single M2 x4 NVME Aic w/ 1 nvme ssd.

 

-PCIE 6
x16 works correct
x8 x8 works correct, sees nvme slot 1 & 3 of aic.
x4x4x4x4 does not work, stays in x16 mode

 

-PCIE 2/3
x4x4x4x4 does not work, only one of pcie slot can be used at a time in this config it doesnt seem to care which.
Testing in PCIE2(master x16) sees only nvme slot 1 of aic.
Testing in PCIE3(slave x8) sees only nvme slot 2 of aic. (will not see first x4 in this config, only 2nd x4 of the x8 in slot. Confirmed this by using the additional single x4 nvme aic)

x8x8 mode does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

Auto Switch mode also does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

 

-PCIE 4/5 is the exact same behavior as PCIE 2/3.
x4x4x4x4 does not work, only one of pcie slot can be used at a time in this config it doesnt seem to care which.
Testing in PCIE4(master x16) sees only nvme slot 1 of aic.
Testing in PCIE5(slave x8) sees only nvme slot 2 of aic. (will not see first x4 in this config, only 2nd x4 of the x8 in slot. Confirmed this by using the additional single x4 nvme aic)

x8x8 mode does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

Auto Switch mode also does the same things as x4x4x4x4, only one pcie slot can be used at a time in this config. It is no longer dividing the x16 into the two x8 slots properly like it used to. Effectively it has made one slot useless.

 

-PCIE6 is close to working, since it has no slave slots attached to it getting that one fully working with 4x4 mode would be a big help alone. I can tell dividing up the other two x16 master slots (PCIE2, PCIE4) which have x8 slave slots(PCIE3, PCIE5) might be harder or maybe impossible to do. Those two slot sets definitely took a big step backwards and were better left working the way they had been previously... if it's a trade-off issue. I understand it will have to disable the slave x8 slots in order to run these two master x16's in 4x4 mode.

 

(This was the meat of the correspondence on testing to Asrock)

 

 

 

 

The board does have some limits on the number of addressable peripherals.  On the previous BIOS I tested with all the PCIe slots populated and PCIE 2 split into two slots with a passive splitter.  In order for all the devices to be recognized I had to disable onboard audio and the USB 3 header.  I've ran into this issue on other platforms, and depending on what all you need there can be some work-arounds with some things disabled.

 

Your testing seems pretty thorough, but I'll give it a go with an ASRock Ultra Quad M.2 card.

 

1 hour ago, BWG said:

I was looking at the board pics and scratching my head on their choice of colors for an AMD board. Kinda glad to see they were hidden after you installed things. I see you're a fan of monsoon fittings too. I really enjoy their stuff. It's different and easy to work with.

 

Choice of colors is pretty standard for a server motherboard.   I don't think I've ever seen an ASRock Rack board that is anything other than green with white, blue, or black RAM and PCIe slots.