Workstation-Desktop Hybrid "Orca"

[tictoc]

All of the m.2 slots come off the CPU. Chipset is LAN, SATA, wifi, bluetooth, and USB. If you can run lstopo either in Linux or WSL, you can see which of your PCIe slots are connected to the numa nodes that have direct access to memory. That can be more of a bottleneck than x8 or x16. Depending on the workload, the increased latency of PCIe->CPU->Infinty Fabric->Memory is noticeable even outside of benchmarks.

 

*Edit* I can check my x399 Taichi later today, since I think they are the same. I misremembered how x399 is split up. All of the PCIe slots can have direct access to memory, but you can check/lock your processes to run on cores with direct memory access. On Linux you can use numactl to bind processes to certain numa nodes, and on Windows I think you can use something like Process Lasso.

Edited May 21, 2020 by tictoc

[ENTERPRISE]

 

I would go top slot for the M.2 card, 3rd slot for primary GPU and 4th slot for secondary GPU. You lose a little bandwidth on the 2nd GPU, but there's no way around that if you want full bandwidth for the M.2 card. I'm not sure if SLI will work with a x8 and a x16 card, but with the above layout you can still use the x1 slot and you can drop a nic in the 2nd slot.

 

 

Good question regarding SLI running at different slot rates. While I have no evidence of this nor been in the situation, I would imagine either the system will match the PCIe slot speeds (So dropping one from 16x to 8x). Or it has no bearing at all on performance as the majority of SLI data transfer happens over the SLI link bus. All theory, never tested it.

[J7SC_Orion]

 

Good question regarding SLI running at different slot rates. While I have no evidence of this nor been in the situation, I would imagine either the system will match the PCIe slot speeds (So dropping one from 16x to 8x). Or it has no bearing at all on performance as the majority of SLI data transfer happens over the SLI link bus. All theory, never tested it.

 

 

Yeah, I would prefer to keep my two 2080 Tis at 16x each, though theoretically, with the TR X399, there should be enough lanes. In any case, if that doesn't work right (ie performance loss on GPU NVLink), I just plug the Aero Xpander 4x M.2 into another MSI X399 mobo I have that carries a single GPU - or perhaps even in an older Asus X99-E-WS (Intel), if I can get the drivers going. That should also actually work (w/ the right drivers), given that this is 'just another' PCIe spec card...!?

 

 

[ENTERPRISE]

 

 

Yeah, I would prefer to keep my two 2080 Tis at 16x each, though theoretically, with the TR X399, there should be enough lanes. In any case, if that doesn't work right (ie performance loss on GPU NVLink), I just plug the Aero Xpander 4x M.2 into another MSI X399 mobo I have that carries a single GPU - or perhaps even in an older Asus X99-E-WS (Intel), if I can get the drivers going. That should also actually work (w/ the right drivers), given that this is 'just another' PCIe spec card...!?

 

 

Yeah this would work in any system as it is just an expansion card, I would imagine you do not even need specific drivers as windows will see this as a standard expansion device or will just install the drives and not that it comes from the same PCIe address.

[tictoc]

Yeah, I would prefer to keep my two 2080 Tis at 16x each, though theoretically, with the TR X399, there should be enough lanes. In any case, if that doesn't work right (ie performance loss on GPU NVLink), I just plug the Aero Xpander 4x M.2 into another MSI X399 mobo I have that carries a single GPU - or perhaps even in an older Asus X99-E-WS (Intel), if I can get the drivers going. That should also actually work (w/ the right drivers), given that this is 'just another' PCIe spec card...!?

 

In order for the 4x M.2 card to work, the motherboard will need to support PCIe bifurcation on a x16 slot. There is no PLX chip onboard the expander card, so you will need to be able to set the slot to x4x4x4x4 in the BIOS. ASRock has very good bifurcation support in their BIOS and ASUS has it enabled on many of their high end boards.

 

[J7SC_Orion]

Thanks guys...I have one MSI X399 Pro Carbon as my sole X399 "reserve", plus my soho office has both an Asus X99-E-WS and Asus X79-E WS humming away at daily tasks. PLX is onboard w/ at least the Asus E WS mobos

[tictoc]

PLX on the motherboard won't necessarily help. I know that there were some modded x79 BIOS floating around that had PCIe bifurcation added. I'm not sure if ASUS ever enabled it in any of the official BIOS. PCIe bifurcation is common on server boards, but is really only common on X299, X399, and trx40 consumer boards. ASRock is really the best at this, and I was able to get a modded BIOS direct from ASRock support to allow me to run 4 GPUs plus an HBA and a NIC without using ridiculously expensive active PCIe splitters. About 2 months after ASRock sent me the modded BIOS, they added the various bifurcation configurations to the official BIOS.

 

I'm pretty sure that the X99-E-WS doesn't support PCIe bifurcation, and if you use the passive M.2 expander card in that board, only the first NVMe drive will show up.

Edited May 21, 2020 by tictoc

[J7SC_Orion]

...well, a project for a rainy weekend ;-) In any case, the MSI X399 Creation, with which the Aoero 4-slot M.2 Xpander card came, obviously supports it, at least if I'm willing to give up some lanes of the secondary GPU. On the other hand, the Asus X99-E WS (pic / brief description below) comes not only with 4x PCIe 3.0 x16, but also with native M.2 NVME onboard...besides, unlike the MSI X399 Creation build (and the X79-E WS also in use), it currently doesn't have SLI GPUs installed anyway.

 

On a related note, before I take the plunge, I should confirm that new 2TB M.2 NVME PCIe 4.0 drives will work in the (PCIe 3.0) MSI Aero Xpander, albeit with the drives in PCIe 3.0 mode only

[J7SC_Orion]

...Finally had some time to play around with additional DDR4 speed settings (went to 32GB from 64GB for the tests, Aida pre-set in the bios, and 1.38v for the Samsung B-die for all settings). This TR2 2950X does have a very good IMC, so it will run quad-channel DDR4 3600 without drama, but the slightly looser primary timings negated any gains over the standard DDR4 3466 daily settings - basically, it's a tie. Also, this mobo / bios just doesn't seem to like DDDR4 3533...at all...ever...

 

Here are the Aida 'read' results for ll three settings

 

 

...and some Aida64 'memory copy' results for 3466 and 3600:

 

Edited June 22, 2020 by J7SC_Orion

[ENTERPRISE]

...Finally had some time to play around with additional DDR4 speed settings (went to 32GB from 64GB for the tests, Aida pre-set in the bios, and 1.38v for the Samsung B-die for all settings). This TR2 2950X does have a very good IMC, so it will run quad-channel DDR4 3600 without drama, but the slightly looser primary timings negated any gains over the standard DDR4 3466 daily settings - basically, it's a tie. Also, this mobo / bios just doesn't seem to like DDDR4 3533...at all...ever...

 

Here are the Aida 'read' results for ll three settings

 

 

...and some Aida64 'memory copy' results for 3466 and 3600:

 

What RAM do you have per chance ? I found that with Zen + getting 3600Mhz was possible with my GSkill Trident Z but it was not truly stable. I had to back it down to 3466Mhz as well. I just think you hit the IMC ceiling with anything above 3466Mhz. As we all know though, lower clock with tighter timings will beat out higher clocks so it is not a huge loss. 3200Mhz was the sweet spot for Zen+ so any higher is just a bonus IMHO.

[J7SC_Orion]

 

What RAM do you have per chance ? I found that with Zen + getting 3600Mhz was possible with my GSkill Trident Z but it was not truly stable. I had to back it down to 3466Mhz as well. I just think you hit the IMC ceiling with anything above 3466Mhz. As we all know though, lower clock with tighter timings will beat out higher clocks so it is not a huge loss. 3200Mhz was the sweet spot for Zen+ so any higher is just a bonus IMHO.

 

 

RAM is 'native' 3866 GSkill TridentZ GTZR for 'Z370'...I have 4 sets of that Sammy-B model which wonder between machines, depending on the specific work tasks. It was not even on the QVL list for the mobo, but that turned out to be a good thing...not using XMP but did it manually with lots and lots of testing, 3466 end result was awfully close to Ryzen Dram Calculator 'fast'settings. I can get error-free 3600 (so far) but only at 16-16-16 and looser secondaries etc...3466 at 14-14-14 is just a better, faster setting overall, as you suggest re. significantly tighter timings with only 133 MHz loss . Weirdly. 3533 is WORSE than 3600 - I guess the bios writers didn't really finalize that particular setting ?!

 

 

 

[ENTERPRISE]

Nice kit indeed, same as mine except mine was native at 3200Mhz (I did not go higher simply as I knew Zen+ had a ceiling that can easily be reached by overclocking). That being said I am able to OC this memory to 3733Mhz on the Zen 2 platform with the 3950X. Fantastic kits.

 

It certainly improves performance by messing with the timings...sometimes a little more time consuming but all do able.

[486]

I should have joined the shiny ram cool club too.. but eh

 

Hey @ENTERPRISE is your 3950x capable of 1900IF? I see you have it at 1866 right now assuming 1:1. I was running 1866 but latest bios for my board forced me down to 1833 (on my 3700x)

 

 

[ENTERPRISE]

I should have joined the shiny ram cool club too.. but eh

 

Hey @ENTERPRISE is your 3950x capable of 1900IF? I see you have it at 1866 right now assuming 1:1. I was running 1866 but latest bios for my board forced me down to 1833 (on my 3700x)

 

 

Yes, I was able to run 1900IF with my RAM OC. However I did hit some instabilities, but that was not necessarily down to me hitting an IF ceiling. My RAM native speed is 3200Mhz and I have pushed it a fair way. So I may have hit the ceiling to my RAM OC as opposed to the IF. What I can say is that the 1900IF on the 3950X is certainly do able as it will boot into windows and work for the majority of tasks, but I was doing RAM tests specifically and I was getting errors here and there, as such I had to wind my IF and my RAM OC a step back to achieve 100% stability.

[J7SC_Orion]

...time for some additional workstation builds...these will be based on older workstation mobos as they won't do 'front-line' database serving, just host some basic development work.

 

Re. cooling fans for the rads, there are the Sunon genuine server fans but they are loud - may be I stick with the GentleTyphoon AP29s for now...

 

 

[ENTERPRISE]

...time for some additional workstation builds...these will be based on older workstation mobos as they won't do 'front-line' database serving, just host some basic development work.

 

Re. cooling fans for the rads, there are the Sunon genuine server fans but they are loud - may be I stick with the GentleTyphoon AP29s for now...

 

 

Will still be a very capable machine, Oh look more Gentle Typhoon fans, I have a load of these new still haha.

[J7SC_Orion]

...wow - time flies ! Haven't built a machine and/or updated this thread since July ! 

 

Did some basic maintenance on the workstation yesterday, such as checking fluids for both loops (5x 360/55 rads...). Fluids look like new even after 22 months...all told, I only had to top up with 20 ml over that whole period...

  

 

 

...debating if / when to upgrade...but so far, no real need via new use cases...and this 'work'station has even learned how to fly

  

 

[Bastiaan_NL]

Time does fly, just like that 'work'station 

What sort of fluid do you use? 

[J7SC_Orion]

19 minutes ago, Bastiaan_NL said:

Time does fly, just like that 'work'station 

What sort of fluid do you use? 

 

1/3rd distilled water, 2/3rds Thermaltake '1000' - not sexy, but I have very positive experience with it over multiple builds and many years...rated for multi-metal use as well