Jump to content

Welcome to ExtremeHW

Welcome to ExtremeHW, register to take part in our community, don't worry this is a simple FREE process that requires minimal information for you to signup.

 

Registered users can: 

  • Start new topics and reply to others.
  • Show off your PC using our Rig Creator feature.
  • Subscribe to topics and forums to get updates.
  • Get your own profile page to customize.
  • Send personal messages to other members.
  • Take advantage of site exclusive features.
  • Upgrade to Premium to unlock additional sites features.
IGNORED

Threadripper Home Server


tictoc
43 Attachments

Recommended Posts

I have been back-and-forth with ASRock, testing a number of beta BIOS.  The same issue still exists with x8x8 working, but x4x4x4x4 just acts like x16.  It is going up the chain to a BIOS engineer, so it seems as if there might be some progress in the near future on the bifurcation issues.

Edited by tictoc
Link to comment
Share on other sites

Folding@Home Staff
730 374
1 hour ago, tictoc said:

I have been back-and-forth with ASRock, testing a number of beta BIOS.  The same issue still exists with x8x8 working, but x4x4x4x4 just acts like x16.  It is going up the chain to a BIOS engineer, so it seems as if there might be some progress in the near future on the bifurcation issues.

 

That's really crap that it's not working, but glad they are on it at least.

 

I got lucky that my Asus Prime X570-Pro supported the 4x4x4x4 mode on the top slot for my 4x NVME M.2 PCIe card without any messing around.

Link to comment
Share on other sites

Reading through this just makes me want to nerd out lol. One thing I will upgrade is my switch to 10G...I am so exciting :p

£3000

Owned

 Share

CPU: AMD Ryzen 9 7950X3D
MOTHERBOARD: MSI Meg Ace X670E
RAM: Corsair Dominator Titanium 64GB (6000MT/s)
GPU: EVGA 3090 FTW Ultra Gaming
SSD/NVME: Corsair MP700 Pro SE Gen 5 4TB
PSU: EVGA Supernova T2 1600Watt
CASE: be quiet Dark Base Pro 900 Rev 2
FANS: Noctua NF-A14 industrialPPC x 6
Full Rig Info

Owned

 Share

CPU: Intel Core i5 8500
RAM: 16GB (2x8GB) Kingston 2666Mhz
SSD/NVME: 256GB Samsung NVMe
NETWORK: HP 561T 10Gbe (Intel X540 T2)
MOTHERBOARD: Proprietry
GPU: Intel UHD Graphics 630
PSU: 90Watt
CASE: HP EliteDesk 800 G4 SFF
Full Rig Info

£3000

Owned

 Share

CPU: 2 x Xeon|E5-2696-V4 (44C/88T)
RAM: 128GB|16 x 8GB - DDR4 2400MHz (2Rx8)
MOTHERBOARD: HP Z840|Intel C612 Chipset
GPU: Nvidia Quadro P2200
HDD: 4x 16TB Toshiba MG08ACA16TE Enterprise
SSD/NVME: Intel 512GB 670p NVMe (Main OS)
SSD/NVME 2: 2x WD RED 1TB NVMe (VM's)
SSD/NVME 3: 2x Seagate FireCuda 1TB SSD's (Apps)
Full Rig Info
Link to comment
Share on other sites

10 hours ago, axipher said:

 

That's really crap that it's not working, but glad they are on it at least.

 

I got lucky that my Asus Prime X570-Pro supported the 4x4x4x4 mode on the top slot for my 4x NVME M.2 PCIe card without any messing around.

Niche board with a niche use case, but every other Threadripper board I've ran worked without a hitch.  I'm not sure why this doesn't work on this board.  I don't care about using NVMe RAID mode or booting the OS off the drives (RAID will be handled by the OS), just looking for what has become a pretty standard option on most server motherboards.

 

Right now I don't really need this to work, but I figured I might as well try to iron out all the possible kinks before I load the hardware in the box.

Link to comment
Share on other sites

On 01/05/2021 at 15:35, Avacado said:

It pleases my OCD

Just realized that if I swap the switch and the patch panel in the rack, I can keep the wires from crossing and not have to label the patch panel 2,1,4,3,6,5...

 

Now we have achieved maximum OCD relief.

rack2.thumb.jpg.c1df7ad16334795585067e21d046a52f.jpg

 

Edited by tictoc
  • Thanks 4
Link to comment
Share on other sites

Beta BIOS #6 looks to be a winner.  In addition to the x4 split now working, there are also x8x4x4 and x4x4x8 splits for PCIE2/3 and PCIE4/5. This should make it possible to split all the slots, including the two slots that are x8 electrical.

 

I haven't fully tested all the options but this works.

bif4.png.599c452a70bb2b595e423a643bbd989f.png

 

Using some old drives to make sure that the final storage configuration will work.

drives.png.be357876296d06ae448429cb0005c2b0.png

 

Drive layout:

  • 6x SATA HDDs on an HBA in PCIe slot 5
  • 4x SATA HDDs on the mini-sas connector (chipset)
  • 2x SATA SSDs (chipset)
  • 1x SATA SSD (Asmedia SATA DOM)
  • 4x NVMe SSD on an ASUS Hyper M.2 X16 v2 in PCIe slot 6
  • 2x M.2 NVMe SSD (onboard slots)
  • 1x U.2 Optane 905p on a PCIe adapter in PCIE slot 3

All of the above is detected and working with 2 GPUs also installed in PCIe slots 2 and 4.  Only empty slot is the top x4 slot which is wired to the chipset.

Edited by tictoc
fixed images
  • Thanks 1
Link to comment
Share on other sites

That is great news they finally got around to fixing it properly. At least they didn't drop your request to fix it like they did for mine, not sure if the tech i was working with just didn't want to deal with it or if my testing notes weren't up to par. Based on your posted tests, that looks good and like everything is finally functioning properly. Hopefully they actually release this bios now.

 

I guess it's time for me to open another ticket to try and get the version you have... the tech i was dealing with hasn't responded in two weeks now.

 

Rack setup is looking good! Good catch on the arrangement to keep cables fanning out proportionally... i see it backwards like that alot in real life actually, funny when you know it was that simple of a change.

 

Link to comment
Share on other sites

Small update on the bifurcation options.  The implementation in the BETA BIOS that I am running seems to be 100% complete.

The slots that are electrically x16 all work in x4x4x4x4 mode, and the x4x4x8 splits also work correctly.

 

This is a huge improvement especially on this platform.  The 2970WX and the 2990WX CPUs have four NUMA nodes, with two NUMA nodes having direct access to memory and certain PCIe slots.  With proper bifurcation on all the slots, there is much more flexibility when running VMs.  It is now much easier to pin high performance VMs to the proper NUMA node, without having to swap devices from slot to slot.

 

The only real limitation now is due to the fact that slots 3 and 5 are only x8 electrically.  At the end of the day it probably doesn't really matter.  With all the bifurcation options working, the use case where this might matter is going to be a niche, of a niche, of a niche, use case, and there are reasonably priced EPYC options available.      

Link to comment
Share on other sites

  • 2 weeks later...

I am actually making progree on the new server now. 

 

I have migrated all the storage off my old 2P server, and hooked everything up to a 3200G/x570 board on one of my benches.   I am currently archiving a bunch of data, since part of the plan on the down sizing was to trim down on some of my data hoarding.  I may end up needing a few more drives, because I'm not sure I want to offline enough stuff to fit in the 24TB media (usable) pool + 4TB (online critical backups)  that I was planning on.  I'm probably going to grab a few more HDDs, which will bring the total up to 10 HDDs in the chasis + 1 in the hot-swap.

 

New Heatkiller radiators arrived last week, and the case is stripped down and cleaned.  Next step will be to plastidip or paint the scratched up acrylic side panel. I'll post some pics once that is done, and then it will be time to actually start putting everything together in the case and get it off my bench. 🙂   

Link to comment
Share on other sites

Folding@Home Staff
7 hours ago, tictoc said:

I am actually making progree on the new server now. 

 

I have migrated all the storage off my old 2P server, and hooked everything up to a 3200G/x570 board on one of my benches.   I am currently archiving a bunch of data, since part of the plan on the down sizing was to trim down on some of my data hoarding.  I may end up needing a few more drives, because I'm not sure I want to offline enough stuff to fit in the 24TB media (usable) pool + 4TB (online critical backups)  that I was planning on.  I'm probably going to grab a few more HDDs, which will bring the total up to 10 HDDs in the chasis + 1 in the hot-swap.

 

New Heatkiller radiators arrived last week, and the case is stripped down and cleaned.  Next step will be to plastidip or paint the scratched up acrylic side panel. I'll post some pics once that is done, and then it will be time to actually start putting everything together in the case and get it off my bench. 🙂   

 

 

I've had great success wet sanding acrylic from 2000, 3000 grit to a da polisher with auto cutting creme, then final polish. Give it a shot if you have the tools.

  • Thanks 1

3.50

Owned

 Share

CPU: 5600x
GPU: EVGA RTX 3090 FTW3 Ultra
GPU 2: EVGA RTX 3080ti FTW3 Ultra
GPU 3: EVGA RTX 3080ti XC3 Hybrid
GPU 4: EVGA RTX 3070ti FTW3 Ultra
GPU 5: MSI RTX 3070 Gaming X Trio
GPU 6: Asus RTX 2080ti ROG STRIX
GPU 7: EVGA RTX 3080ti FTW3 Ultra
Full Rig Info
Link to comment
Share on other sites

On 13/07/2021 at 10:43, BWG said:

 

 

I've had great success wet sanding acrylic from 2000, 3000 grit to a da polisher with auto cutting creme, then final polish. Give it a shot if you have the tools.

That's not a bad idea.  Same method I use to polish my headlights. 🙂

Link to comment
Share on other sites

why people keep giving me ideas how to spend money on something I clearly don't need but I would rather have. haha

 

Seeing this, making me wanna jump start my next project of centralizing my entire home it into one rack and wiring access via cat cable  😃

eh, one can only dream about such a wonderful set up 🤩

Link to comment
Share on other sites

Here I am, just now getting a rack system this last year.  And here you are, going away from a rack into a Threadripper all in one setup.  🤣

Not sure if you like punishment or what, going with water cooling on a server, but it looks absolutely amazing so far.  Them cable management pictures......man oh man. 🤤

Link to comment
Share on other sites

17 hours ago, pioneerisloud said:

Here I am, just now getting a rack system this last year.  And here you are, going away from a rack into a Threadripper all in one setup.  🤣

Not sure if you like punishment or what, going with water cooling on a server, but it looks absolutely amazing so far.  Them cable management pictures......man oh man. 🤤

 

Most of my gear that runs 24/7 is water-cooled.  As long as you follow some best practices like EPDM tubing, distilled+Mayhems biocide and inhibitor, dual pumps in serial, there is not really anything to worry about or any additional maintenance.

 

I haven't touched my workstation in 16 months other than to top off the fluid one time, and it runs 24/7 @ about 1900-2000W. 😀

  • Thanks 2
Link to comment
Share on other sites

22 minutes ago, tictoc said:

 

Most of my gear that runs 24/7 is water-cooled.  As long as you follow some best practices like EPDM tubing, distilled+Mayhems biocide and inhibitor, dual pumps in serial, there is not really anything to worry about or any additional maintenance.

 

I haven't touched my workstation in 16 months other than to top off the fluid one time, and it runs 24/7 @ about 1900-2000W. 😀

That's why I like forums like this man, learn something new everyday. 😄  Never thought to run dual pumps in serial.  I had only ever learned of simple loops, and that's all I ran before.  Pretty sure I was following good practices still, my last loop did last like 2 years without draining, and sold the whole rig functional.  But yeah.

I just assumed that air was simpler to maintain on something that's on 24/7 365.  I wasn't thinking leaks or anything silly like that.  Was thinking punishment as in its more parts that could potentially fail and lead to downtime of a "server", that's all.  I guess dual pumps in serial kind of negates that point, since you'd have a backup pump already running so at least things still move.  And the biocide + inhibiter I'm assuming would stop things growing better than a killcoil would.  Neat stuff!

Edited by pioneerisloud
Link to comment
Share on other sites

40 minutes ago, pioneerisloud said:

That's why I like forums like this man, learn something new everyday. 😄  Never thought to run dual pumps in serial.  I had only ever learned of simple loops, and that's all I ran before.  Pretty sure I was following good practices still, my last loop did last like 2 years without draining, and sold the whole rig functional.  But yeah.

I just assumed that air was simpler to maintain on something that's on 24/7 365.  I wasn't thinking leaks or anything silly like that.  Was thinking punishment as in its more parts that could potentially fail and lead to downtime of a "server", that's all.  I guess dual pumps in serial kind of negates that point, since you'd have a backup pump already running so at least things still move.  And the biocide + inhibiter I'm assuming would stop things growing better than a killcoil would.  Neat stuff!

Even with only one pump, you can survive a pump failure if you have enough rad space and high rpm fans.  I've woken up to a dead pump with the system still chugging along at 100% load.  I've also had an almost complete blockage, but kept the system going for days before I cleaned it out since we were in the middle of the Pentathlon.

Link to comment
Share on other sites

  • 2 months later...

The last few months have been very busy, so this project has stalled. 

 

Back at it again, and I will post some better pics, but here's a few quick ones of the current progress.

Radiators are in, half of the drive brackets are in, and all of the fans are installed. 

 

fansRads1.thumb.jpg.3848a0da31575e35a7fba292125cc50f.jpg

fansRads2.thumb.jpg.551351128934ebb5c417e3a2a42e0076.jpg

fansRads3.thumb.jpg.bac36dc0efadd35e2b9506f9c6aa1fd4.jpg

 

 

 

  • Thanks 1
  • Respect 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...

Important Information

This Website may place and access certain Cookies on your computer. ExtremeHW uses Cookies to improve your experience of using the Website and to improve our range of products and services. ExtremeHW has carefully chosen these Cookies and has taken steps to ensure that your privacy is protected and respected at all times. All Cookies used by this Website are used in accordance with current UK and EU Cookie Law. For more information please see our Privacy Policy