HP Z840 36C/72T Workstation Build

[Storm-Chaser]

I re-applied the liquid metal paste and it's better but still not perfect, I think the pump might be failing at this point. I have another one.

 

 

Edited April 17, 2022 by Storm-Chaser

[Storm-Chaser]

Just confirmed the Z840 will support PCIe Gen 3 NVMe M.2 SSD drives, which is good because I have a spare Samsung 970 PRO to swap in. 

 

More details to follow. Have to buy the adapter first. 

[Storm-Chaser]

Ive already figured out my next CPU upgrade path. But it's still probably a couple months out. 

 

E5 2698v4 20C/40T 2.2GHz base / 3.6GHz boost

 

TDP: 135W

L3 Cache: 50MB

x2

So that will be 40 cores vs 36 cores (as it stands now.)

Also going from 22nm to 14nm and from 145W TDP to 135 W TDP. The added benefit is that V4 processors support 2400MHz memory, and since I have 2400MHz modules installed, it will allow me to run them at the full speed (as opposed to just 2133MHz for v3 cpus)

 

Also installing an additional 360mm radiator in the bottom part of the case. Not entirely sure how to direct air flow over the radiator but there are two 80mm fans that flow from front to back, so I think I can get that sorted out easily enough, especially if I just remove the pcie slot covers from the back of the case, so the air from the radiators just flows front to back, in line with the direction of airflow through the case. The radiator will have three Noctua NF-12 fans which are totally awesome, very quiet, (except for the color) and hold up really well. 

 

 

[HeyItsChris]

Holy crap those are some insane numbers on AIDA64 CPU Cache Memory benchmark!  100GB/s+ memory bandwidth & more than 5TB/s L1 Cache Performance, wow!

 

PS - Your temperatures are amazing for those powerful processor(s).

Edited April 19, 2022 by HeyItsChris

[Storm-Chaser]

5 hours ago, HeyItsChris said:

Holy crap those are some insane numbers on AIDA64 CPU Cache Memory benchmark!  100GB/s+ memory bandwidth & more than 5TB/s L1 Cache Performance, wow!

 

PS - Your temperatures are amazing for those powerful processor(s).

Here are the numbers after final tweaking - memory tweaking in BIOS, CPU undervolting, NB set to 3000 MHz using throttlestop. Not bad. Only number I am not happy with is latency, but that's the best I can do until I get the V4 chips next month. It says quad channel in CPUz but it's actually octal channel, because all sixteen DIMM slots are populated. (64GB total)

 

 

 

[tictoc]

Actually it's still quad channel.  Only Epyc, Threadripper Pro, and newer Xeon Gold/Platinum CPUs have 8-channel memory controllers.

 

Anything that is crossing the QPI to a far NUMA node, is going to suffer a hit in latency and performance.  No idea how AIDA works, but in my experience, Windows generally tends to be less NUMA aware, so if the OS scheduler is not pinning processes to local memory then there will be a drop in performance.

Edited April 19, 2022 by tictoc

[Storm-Chaser]

1 hour ago, tictoc said:

Actually it's still quad channel.  Only Epyc, Threadripper Pro, and newer Xeon Gold/Platinum CPUs have 8-channel memory controllers.

 

Anything that is crossing the QPI to a far NUMA node, is going to suffer a hit in latency and performance.  No idea how AIDA works, but in my experience, Windows generally tends to be less NUMA aware, so if the OS scheduler is not pinning processes to local memory then there will be a drop in performance.

I cannot agree with this. It's directly in the HP documentation that the rig supports octal channel memory. And, if you add up the read and write speeds AIDA64 is reporting, you will find that it does in fact use all eight channels to reveal true performance data. (As in my memory architecture is 8 channel effective)

 

 

 

 

@tictoc Does this sound right to you? 

Edited April 19, 2022 by Storm-Chaser

[Storm-Chaser]

8 hours ago, HeyItsChris said:

PS - Your temperatures are amazing for those powerful processor(s).

Liquid metal is awesome sauce.

 

I will never ever go back to conventional thermal paste again. 

Hard to believe a tiny 80mm x 80mm radiator can deliver the performance like this, but it's real. Makes you wonder about weak AIOs that are twice the size of this one coming short. I just put a 2' classic blue light in the chassis. I will get some pics of that tonight, it's looking really good but not visible except from the front. And hard to believe I get these temps AT IDLE. They very rarely kick about 20%. So the rig is super quiet most of the time. But I tell you what, if you want to jack up the fans this thing is basically insane server level airflow if you want it. With that, I could probably run a 100% load across all CPUs at 4.5GHz or better. In a perfect world. But many people think this rig is loud and that is simply NOT the case. With proper tuning, you can have near silent machine that steamrolls current tech. 

 

 

 

[Storm-Chaser]

1 hour ago, tictoc said:

so if the OS scheduler is not pinning processes to local memory then there will be a drop in performance.

HP recommends the exact setup I've dropped in. It's actually listed as one of the "best" scenarios for your memory configuration / architecture for the z840. 

[Storm-Chaser]

2 hours ago, tictoc said:

Actually it's still quad channel

I think CPUz only sees one CPU at a time (as in it assess a single CPU only even if you have a multiprocessor rig when analyzing your systems memory architecture. This means it will only report quad channel because it can only see one memory controller at a time. Since each memory controller has four active channels, it is 8 channel effective architecture as reported accurately in AIDA64. 

[tictoc]

Octal rank is different than octal channel.  Octal rank refers to the number of ranks per dimm.  LRDIMMs generally come in quad rank and octal rank, which allows increased capacity per dimm. The increased capacity of LRDIMMs trades maximum capacity for performance. 

In regards to whether or not it's octa channel, the platform itself (2 quad channel CPUs) technically has 8 channels, but each CPU can only address 4 channels locally.  While the QPI is a fast path (8GT/s on Xeon v3) it's no where near as fast as local memory access. Generally speaking taking the performance hit is worth it due to the doubling of addressable memory.

 

Just saw your reply and that is all correct.

 

Specifically for your system the setup is good.  The memory you have is single rank, so if you wanted to increase performance, you could look at getting some dual rank dimms which would bump up the memory throughput.  For maximum performance anything above dual rank will generally be slower, and is only worth it if you actually need the additional capacity.  

Edited April 20, 2022 by tictoc

[Storm-Chaser]

1 hour ago, tictoc said:

Octal rank is different than octal channel.  Octal rank refers to the number of ranks per dimm.  LRDIMMs generally come in quad rank and octal rank, which allows increased capacity per dimm. The increased capacity of LRDIMMs trades maximum capacity for performance. 

In regards to whether or not it's octa channel, the platform itself (2 quad channel CPUs) technically has 8 channels, but each CPU can only address 4 channels locally.  While the QPI is a fast path (8GT/s on Xeon v3) it's no where near as fast as local memory access. Generally speaking taking the performance hit is worth it due to the doubling of addressable memory.

 

Just saw your reply and that is all correct.

 

Specifically for your system the setup is good.  The memory you have is single rank, so if you wanted to increase performance, you could look at getting some dual rank dimms which would bump up the memory throughput.  For maximum performance anything above dual rank will generally be slower, and is only worth it if you actually need the additional capacity.  

Okay! More performance is left on the table! Thats exactly why I started down this road, I would really like to fully load this thing for maximum performance. Thank you for all this information. This project is starting to get fun. I have to sell my single rank, and I can probably get $300 for it, plus I have some DDR4 C19 4133 kits I can sell which should net me a total of $600 to spend on dual rank memory, next month. I totally dismantled my 9600KF rig. So that whole deal is being retired, I'm going to sell some of the parts off and probably repurpose the 360mm radiator for use in the z840 (If I really need more cooling headroom).

 

Cleaned up my computer area, added a second video card to the Z840, so I could add the third monitor and dismantled the 9600KF rig (it was time). Selling most of it off so if anyone wants anything let me know. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Edited April 20, 2022 by Storm-Chaser

[Storm-Chaser]

9600KF RIP (this was long overdue)

 

 

 

Edited April 20, 2022 by Storm-Chaser

[Storm-Chaser]

8 hours ago, tictoc said:

your system the setup is good.  The memory you have is single rank, so if you wanted to increase performance, you could look at getting some dual rank dimms which would bump up the memory throughput. 

What am I looking at for an impact on performance with dual rank memory? I'm seeing 3-5% average improvement from a bit of research. In terms of bandwidth. Do two single rank modules work as like one dual rank module?

[Storm-Chaser]

Little video so you can get an idea of sound - with the fans turned all the way up. Forgive my potato phone.

 

 

 

[Avacado]

Looking good. In the event you ever do any GPU benching on that rig. Try to only run 1 1080p monitor. 

[Storm-Chaser]

1 hour ago, Avacado said:

Looking good. In the event you ever do any GPU benching on that rig. Try to only run 1 1080p monitor. 

It's all really just for fun no real requirements for this rig other than to get to know the platform. I have a 5700XT gaming X can throw in there if I get bored and want to game, but the problem is it may interfere with my 240mm or 360mm supplemental radiator (If I can get all core turbo hack to work)

 

Where is your 512T machine? I'm patiently waiting....

 

 

[Avacado]

7 minutes ago, Storm-Chaser said:

It's all really just for fun no real requirements for this rig other than to get to know the platform. I have a 5700XT gaming X can throw in there if I get bored and want to game, but the problem is it may interfere with my 240mm or 360mm supplemental radiator (If I can get all core turbo hack to work)

 

Where is your 512T machine? I'm patiently waiting....

 

 

Received the 2133 ECC ram. I really couldn't justify spending enough to fill all 16 slots when 10 DIMMs cost $300. Board is in route from Cali, would love to get it before the weekend. The CPUs should be here today. The coolers are another story, coming from China. Could only find 2U coolers and i'm sure they aren't going to be enough, but I didn't want to water cool it. I could have bought some Noctua variants for the LGA 3647, but they were like $200 a piece. 

 

I was not willing to spend more than a grand for a pet project.

Edited April 20, 2022 by Avacado

[Storm-Chaser]

4 minutes ago, Avacado said:

Received the 2133 ECC ram. I really couldn't justify spending enough to fill all 16 slots when 10 DIMMs cost $300. Board is in route from Cali, would love to get it before the weekend. The CPUs should be here today. The coolers are another story, coming from China. Could only find 2U coolers and i'm sure they aren't going to be enough, but I didn't want to water cool it. I could have bought some Noctua variants for the LGA 3647, but they were like $200 a piece. 

 

I was not willing to spend more than a grand for a pet project.

Okay cool. 

As you may know the 9600KF is no mas. Thank god. I never liked that rig. But it served it's purpose and performed great for me for overclocking. But this is my main machine now so I feel I can invest some money into seeing as how it more or less will give me decent performance levels for years to come. Once I sell off most of the 9600KF rig, I will more or less be in pretty good standing having spent only about $500 total (out of pocket) on the new rig after all is said and done. 

[tictoc]

6 hours ago, Storm-Chaser said:

What am I looking at for an impact on performance with dual rank memory? I'm seeing 3-5% average improvement from a bit of research. In terms of bandwidth. Do two single rank modules work as like one dual rank module?

 

That is correct.  Two single rank dimms per channel is effectively equal to one dual rank dimm per channel.  With dual ranked dimms you would effectively be running quad rank if you populate all 8 slots, but generally speaking depending on the workload you will see a bump in performance.  

 

I never really tuned my 2P v3 system, because I moved over to Threadripper shortly after acquiring it.  The best performance on my Xeon v2 system, was using dual ranked dimms, but only installing 4 dimms per socket.  My ASRock Rack board allowed me to bump the memory speed over the default for my sticks, so I was able to run 1333MHz sticks at 1866MHz.  The board wasn't stable at 1866MHz with all 16 slots populated.  Running in quad channel with 4 dimms per socket at the higher speed was the most performant setup, and I only moved away from that when I started to need more than 64GB of memory.    

Edited April 20, 2022 by tictoc

[Avacado]

On 20/04/2022 at 08:35, Storm-Chaser said:

Okay cool. 

As you may know the 9600KF is no mas. Thank god. I never liked that rig. But it served it's purpose and performed great for me for overclocking. But this is my main machine now so I feel I can invest some money into seeing as how it more or less will give me decent performance levels for years to come. Once I sell off most of the 9600KF rig, I will more or less be in pretty good standing having spent only about $500 total (out of pocket) on the new rig after all is said and done. 

Kind of freaking out right now. I just read this 

LGA 3647 - Wikipedia

EN.WIKIPEDIA.ORG

Apparently there are 2 versions of the P socket. But I have no way of knowing if the board that I bought is compatible with the Phi chips. Knowing my luck it wont be. I will find out tonight, but i'm kind of pissed that it's not easy to see. 

X11DAi-N | Motherboards | Products | Super Micro Computer, Inc.

WWW.SUPERMICRO.COM

 

[tictoc]

2 hours ago, Avacado said:

Kind of freaking out right now. I just read this 

LGA 3647 - Wikipedia

EN.WIKIPEDIA.ORG

Apparently there are 2 versions of the P socket. But I have no way of knowing if the board that I bought is compatible with the Phi chips. Knowing my luck it wont be. I will find out tonight, but i'm kind of pissed that it's not easy to see. 

X11DAi-N | Motherboards | Products | Super Micro Computer, Inc.

WWW.SUPERMICRO.COM

 

 

The board you linked (X11DAi-N) is socket P0 for Xeon Scalable-SP 1st and 2nd gen CPUs.

x11dai-n_quickRef.pdf

 

This is the only ATX form factor Supermicro board that I am aware of that supports Knight Landing. https://www.supermicro.com/products/motherboard/Xeon_Phi/K1SPE.cfm

 

There might be others, but I've never seen a working one in the wild other than the boards from the Supermicro and ASRock Rack developer workstations.  I think Intel originally planned on Knight's Landing/Mill being compatible with all 3647 boards, but scrapped that plan when they EOL'd the Xeon Phi.  I am pretty sure (but not positive) that you need a Xeon Phi specific board.

[Avacado]

1 minute ago, tictoc said:

 

The board you linked (X11DAi-N) is socket P0 for Xeon Scalable-SP 1st and 2nd gen CPUs.

x11dai-n_quickRef.pdf 1.65 MB · 1 download

 

This is the only ATX form factor Supermicro board that I am aware of that supports Knight Landing. https://www.supermicro.com/products/motherboard/Xeon_Phi/K1SPE.cfm

 

There might be others, but I've never seen a working one in the wild other than the boards from the Supermicro and ASRock Rack developer workstations.  I think Intel originally planned on Knight's Landing/Mill being compatible with all 3647 boards, but scrapped that plan when they EOL'd the Xeon Phi.  I am pretty sure (but not positive) that you need a Xeon Phi specific board.

I know it. Found that out today. Thank you for the research. Looks like all this stuff is going back to sender. I was expecting to run dual socket. Sad face

[Storm-Chaser]

1 minute ago, Avacado said:

I know it. Found that out today. Thank you for the research. Looks like all this stuff is going back to sender. I was expecting to run dual socket. Sad face

 I think as a collective forum we should be able to find a solution here without you having to return everything. Lets just see if we can find you some sort of workaround...

[Avacado]

6 minutes ago, Storm-Chaser said:

 I think as a collective forum we should be able to find a solution here without you having to return everything. Lets just see if we can find you some sort of workaround...

There really isn't one. I had literally no idea that a socket could have sub variants. Given i'm at work and piecing together research last minute on a hunch. I have never encountered that before