Build Log: Spotswood Custom GPU Bench

[BWG]

The bench has a free power button now.

[tictoc]

On 11/07/2021 at 09:08, Bastiaan_NL said:

@tictoc you scare me with stories like that.. Currently looking for the max combined PCIe powerdraw on my x299 sage and black 30cm PCIe 3.0 x16 powered riser cables...

And I already know the EVGA 3090's are really power hungry on the PCIe side, drawing much more than the common maximum of 75W through the slot. (I've seen up to 88W already)

 

You should be fine with that board.  The Sage is designed as a compute/rendering/ML workstation, so it is designed to run several power hungry GPUs.  It has auxillary PCIe power, and it sounds like you also have powered risers.  I regulary run 4 Radeon VIIs, each pulling >380W, on my Threadripper workstation without a hitch.

[Bastiaan_NL]

22 hours ago, BWG said:

The bench has a free power button now.

 

That's a lot better

 

 

10 minutes ago, tictoc said:

 

You should be fine with that board.  The Sage is designed as a compute/rendering/ML workstation, so it is designed to run several power hungry GPUs.  It has auxillary PCIe power, and it sounds like you also have powered risers.  I regulary run 4 Radeon VIIs, each pulling >380W, on my Threadripper workstation without a hitch.

Thanks, I use all the power plugs on the board (24+8+8+6) too. The risers are not powered, but the cables are not hot or warmer than any other part of the system (if I can find black 30cm powered ones I will buy them for sure). 

It's not easy to find any specific information about the combined powerdraw through the slots, I was hoping the manual would provide that information.  210W was no problem through the slot, that was the max yesterday

[tictoc]

6-pin PCIe can safely provide a mimimum additional 150W, so unless you get really crazy and run 14 GPUs on that board, you should be good.

Edited July 13, 2021 by tictoc
typos

[BWG]

1 hour ago, tictoc said:

The 6-pin can safely provide a mimimum additional 150W, so unless you get really crazy and run 14 GPUs on that board, you should be good.

 

Yeah, I was running 2 per 6 pin, but that's like the limit, and with 4 PSU's over there, I just decided to do 1 per lol. I'm getting ready to start hooking into the 3 m.2's on the 2nd tier though, so I probably will try returning to 2 per 6 pin. Now I know you were talking to Bastiaan, but I still benefit from this advice.

Edited July 12, 2021 by BWG

[BWG]

Got 1 gpu on the way to go in my desktop, and need 1 more for the bench.

 

Here's the reconfig results. Top tier needs cable managed better, but both psus were swapped out for 1300w ones.

 

 

[Bastiaan_NL]

Those cables need some work indeed! But it looks good to see a card in (almost)every slot. 
I'm thinking about getting a lot of sleeved single PCIe cables instead of those double cables where you need to hide that little part.. 
And they are way more flexible so better to route, bundle and hide if needed. Downside is the cost of all those cables... 

[Avacado]

8 minutes ago, Bastiaan_NL said:

Those cables need some work indeed! But it looks good to see a card in (almost)every slot. 
I'm thinking about getting a lot of sleeved single PCIe cables instead of those double cables where you need to hide that little part.. 
And they are way more flexible so better to route, bundle and hide if needed. Downside is the cost of all those cables... 

Indeed, but so costly just to hide away a 6 pin. What I REALLY want is round PCI-e extension cables, or PSU cable style ones. 

Edited July 15, 2021 by Avacado

[tictoc]

If you don't mind pulling the pins out of the connectors, you can get large sizes of braided cable sleeving and combine multiple PCIe cables into a single run.  You might be able to expand the sleeving enough to get it to fit over the connector without removing it.  Then it would just be one nice and neat budle running up to the GPUs, with each connector exiting the sleeving right at the GPU.  I built most of the wiring harness for my old Jeep like that, and it looks really clean.

 

https://www.cabletiesandmore.com/general-purpose-braided-sleeving

 

[Bastiaan_NL]

A custom wire loom would look good on yours too @BWG  

 

20 hours ago, Avacado said:

Indeed, but so costly just to hide away a 6 pin. What I REALLY want is round PCI-e extension cables, or PSU cable style ones. 

Really costly.. The Corsair Premium sleeved cables are €26 for a pair over here, and I need at least 5 pairs (not even in stock).. And 65cm might not be enough.

 

19 hours ago, tictoc said:

If you don't mind pulling the pins out of the connectors, you can get large sizes of braided cable sleeving and combine multiple PCIe cables into a single run.  You might be able to expand the sleeving enough to get it to fit over the connector without removing it.  Then it would just be one nice and neat budle running up to the GPUs, with each connector exiting the sleeving right at the GPU.  I built most of the wiring harness for my old Jeep like that, and it looks really clean.

 

https://www.cabletiesandmore.com/general-purpose-braided-sleeving

 

I don't mind pulling the pins, and I have a lot of left over cable sleeves from my casemodding days. So I might try to make my own PCIe harness. The only problem is that I like to switch cards around, and having 3x8 at each of the 6 locations might be a bit too much.

[BWG]

Couple things done to this in order to manage the cables a bit better, but there's still some things I'm wanting to do to it before I call it finalized.

 

 

Edited July 25, 2021 by BWG

[tictoc]

That is looking great, and so is your F@H production.  Awesome to see all that hardware running a volunteer compute project.

Edited July 25, 2021 by tictoc

[BWG]

And now it's all blue. I'm going to order some fans now. I don't run it on fah a lot, but I try to fire it up weekly, or at least the good ppd stuff.

[Diffident]

I think @BWG is currently folding on as many GPU's as I've owned in my entire life.  I just counted, I've owned 13 GPU's starting with the Sierra Screamin 3D that I bought the week it launched in 1996.

[Bastiaan_NL]

Looks great @BWG!
A lot cleaner and organized

 

 

@Diffident I remember 29 cards I've owned, of which 21 different models. 

This is what I remember:

8600gt 9800gt gtx275x2 gts250x2 gts450x3 gtx460sx2 gtx480x2 gtx580 980ti m40 2080ti 2080s 1660ti 1650s 3060 3070ti 3090x2 5700xtx2 hd3850 hd4850 hd6870

[BWG]

These look promising. I'm getting ready to put them in and replace the 3.0 ones. 

 

 

 

Confirmed legit pci-e 4.0. Nice! Pretty inexpensive too.

 

 

Edited November 16, 2021 by BWG

[BWG]

On 21/05/2021 at 15:46, Avacado said:

Your going to need 30million + PPD to beat us in the comps.

 

I fixed the images in the OP, and then saw this which was funny.  

 

Edit: Might as well update the picture. 

 

Edited November 16, 2021 by BWG

[EHW Ai]

It really is amazing what YOU can do with extruded alu and Spotswood does a great job of help making it happen!

[PCSarge]

now if only he made a rack for dual xeon intel server boards

[ENTERPRISE]

3 hours ago, PCSarge said:

now if only he made a rack for dual xeon intel server boards

Perhaps message them ? You never know what might come of it.

[Avacado]

3 hours ago, PCSarge said:

now if only he made a rack for dual xeon intel server boards

Money and patience make all things possible. TBH, I would involve Gregg as he probably has the best relationship with him.  

[firedfly]

3 hours ago, PCSarge said:

now if only he made a rack for dual xeon intel server boards

 

He may have already made a case for a board like that.  If not, he can customize a case based on measurements you send him.  My main FAH rig is in one of his cases that I customized a bit and I'm quite happy with the result.  Definitely send him a message.

[EHW Ai]

10 hours ago, PCSarge said:

now if only he made a rack for dual xeon intel server boards

 

It wouldn't hurt to ask him directly.

 

FWIW, the setup I use is based somewhat on his designs, but I made it myself studying the images he has on his website. To keep construction simpler, I used a motherboard plate from a Mountain Mods motherboard tray which fits ATX/EATX form factors. While Mountain Mods doesn't carry a motherboard tray suitable for what amounts to larger rack sized equipment, I imagine a plate/motherboard tray that would suit your needs could be a diy project if you can/are willing to invest the effort (and some moolah; and I don't feel it would get that expensive if you don't allow it too). 

 

Just some sound advice from the old guy.

 

Edited April 1, 2022 by iamjanco