Tight Fit! Standalone ESXi server

[Laithan]

I figured I would start off the server thread with my latest ESXi server that I built for myself running ESXi v6.5.

 

I performed a secure erase and updated the firmware on the drives and all other devices where possible. This was a fun build and ended up being a bit of a puzzle since everything was so tight and had a specific install order. There is a pretty even split between new and used parts but overall the cost was extremely low in comparison to purchasing a server with similar specs. The pictures are at various stages during completion, testing and some datastore imaging. The 4TB drives and controller fans were the last to be installed. The architecture is IVY Bridge-E, 22nm / DDR3.

 

Specs:

20 Cores/40 threads @ 3.6Ghz turbo

192GB DDR3 1600Mhz ECC memory

4TB raw SSD storage, 12TB raw spinners

Dual 10GB NIC + USB 3.0

 

Details:

KTH-PL316/16G KINGSTON 1600Mhz ECC 16GBx8

KTH-PL38/8G KINGSTON 1600Mhz ECC 8GBx8

(192GB Total)

 

4U E-ATX Rackmount Server Chassis IPC E-430

Lenovo Thinkstation D30 1120W GOLD Rated dual EPS power supply

 

Intel S2600CO C600 chipset dual processor motherboard (basically X79 but with dual sockets)

Intel® Xeon® Processor E5-2680v2 x 2 (10-core, 20 threads @ 3.60Ghz turbo x 2, 40 PCI-e 3.0 lanes)

Noctua NH-U9DXi4 90mm SSO2 CPU Cooler x 2 (PUSH/PULL)

 

LSI MegaRAID 9266-8i w/BBU + Fastpath (SSDs)

LSI MegaRAID 9260-8i w/BBU + Fastpath (Spinners)

AVC Ball bearing cooling fan for LSI 9266-8i

AVC Ball bearing cooling fan for LSI 9260-8i

 

ICY DOCK ToughArmor MB994SK-1B Full Metal 4 Bay 2.5" SAS/SATA HDD/SSD Cage x 2

Samsung 840 PRO 512GB SSD x 8

Seagate ST4000NM0033 4TB 7200RPM SATA 6Gb/s x 3

 

Intel X550-T2 10Gx2 Ethernet Server Adapter

USB 3.0 PCI-e adapter

 

NOCTUA NF-A8 80mm PWM x2 exhaust fans

NOCTUA NF-B9 93mm PWM x4 CPU fans

NOCTUA NF-F12 120mm PWM Industrial Intake fan

 

 

----

 

I'll post some pics here and also a video for the extra geeky LMAO

 

Here is my ultra nerdy thrown together quick video version @schuck6566 inspired me to make lol!

It is even in 4K - (Nerd glasses and labcoat required)

https://youtu.be/GXP3DhyWrEg

 

 

HI-RES PICTURES

 

 

Edited January 17, 2020 by Laithan

[schuck6566]

I figured I would start off the server thread with my latest ESXi server that I built for myself running free ESXi v6.5.

 

I performed a secure erase and updated the firmware on the drives and all other devices where possible. This was a fun build and ended up being a bit of a puzzle since everything was so tight and had a specific install order. There is a pretty even split between new and used parts but overall the cost was extremely low in comparison to purchasing a server with similar specs. The pictures are at various stages during completion, testing and some datastore imaging. The 4TB drives and controller fans were the last to be installed. The architecture is IVY Bridge-E, 22nm / DDR3.

 

Specs:

20 Cores/40 threads @ 3.6Ghz turbo

192GB DDR3 1600Mhz ECC memory

4TB raw SSD storage, 12TB raw spinners

Dual 10GB NIC + USB 3.0

 

Details:

KTH-PL316/16G KINGSTON 1600Mhz ECC 16GBx8

KTH-PL38/8G KINGSTON 1600Mhz ECC 8GBx8

(192GB Total)

 

4U E-ATX Rackmount Server Chassis IPC E-430

Lenovo Thinkstation D30 1120W GOLD Rated dual EPS power supply

 

Intel S2600CO C600 chipset dual processor motherboard (basically X79 but with dual sockets)

Intel® Xeon® Processor E5-2680v2 x 2 (10-core, 20 threads @ 3.60Ghz turbo x 2, 40 PCI-e 3.0 lanes)

Noctua NH-U9DXi4 90mm SSO2 CPU Cooler x 2 (PUSH/PULL)

 

LSI MegaRAID 9266-8i w/BBU + Fastpath (SSDs)

LSI MegaRAID 9260-8i w/BBU + Fastpath (Spinners)

AVC Ball bearing cooling fan for LSI 9266-8i

AVC Ball bearing cooling fan for LSI 9260-8i

 

ICY DOCK ToughArmor MB994SK-1B Full Metal 4 Bay 2.5" SAS/SATA HDD/SSD Cage x 2

Samsung 840 PRO 512GB SSD x 8

Seagate ST4000NM0033 4TB 7200RPM SATA 6Gb/s x 3

 

Intel X550-T2 10Gx2 Ethernet Server Adapter

USB 3.0 PCI-e adapter

 

NOCTUA NF-A8 80mm PWM x2 exhaust fans

NOCTUA NF-B9 93mm PWM x4 CPU fans

NOCTUA NF-F12 120mm PWM Industrial Intake fan

 

 

----

 

NOTE

As I am writing this post I have been made aware of a 5 picture picture limit. I had uploaded approximately 30 pictures. At this time I am going to post this message but remove all pics until I have a chance to sort this out. I will return with an update shorty, thank you

 

Hey bro, make a slide show vid of your pics, post on you youtube and post the video on here. :eat_popcom: you build vids are always fun to watch.(like your xeon build!)

[Laithan]

Doing just that! Will be fun I shall return.

 

(but imo it would be worth considering increasing the 5 picture limit for the build forums)

[ENTERPRISE]

The limit is 75 for attachments across the board, I am still adjusting these permissions for all usergroups (Takes a while lol) but I have adjusted this one immediately to accommodate.

 

Thanks,

E

Edited January 6, 2020 by ENTERPRISE

[Laithan]

The limit is 75 for attachments across the board, I am still adjusting these permissions for all usergroups (Takes a while lol) but I have adjusted this one immediately to accommodate.

 

Thanks,

E

 

 

<salutes> Thanks!!

 

I was able to update the original post above

Edited January 8, 2020 by Laithan

[schuck6566]

Dang! That's just sick! :thumbs_thumbsupup:

[ENTERPRISE]

Just sat and watched the video, love server builds. Out of interest @Laithan , what do you use that baby for ?

[Laithan]

Just sat and watched the video, love server builds. Out of interest @Laithan , what do you use that baby for ?

 

Thanks for all the help getting the pics straightened out. They are now behind the spoiler in the original post.

 

I've got a bunch of VMs spun up for various things (some are combined).

* Domain Controller (Active Directory)

* Internal DNS

* WSUS

* File Storage/Print

* Pi-Hole

* Steam cache

* Plex Media

* Nessus

* VAS

* Legacy XP (special peripheral support)

* Win10 testing (collecting dust lol)

* Windows 7 P2V (JIC my pc is down)

* Syslog

* Security

* vCenter (testing with VMUG advantage)

* other tests and unfinished projects..

 

 

GoPro 4K/60 footage gobbles up terabytes quick so that sorta started me on the idea of building a 2nd server. My next adventure is to get vMotion working between 2 standalone hosts. I do have shared storage (NFS) but it is slow (1GB vs 10GB) so I prefer to keep the VMs running locally until maybe one day when I can upgrade my QNAP NAS to one that supports 10GB.

 

 

Edited January 17, 2020 by Laithan

[ENTERPRISE]

Just sat and watched the video, love server builds. Out of interest @Laithan , what do you use that baby for ?

 

Thanks for all the help getting the pics straightened out. They are now behind the spoiler in the original post.

 

I've got a bunch of VMs spun up for various things (some are combined).

* Domain Controller (Active Directory)

* Internal DNS

* WSUS

* File Storage/Print

* Pi-Hole

* Steam cache

* Plex Media

* Nessus

* VAS

* Legacy XP (special peripheral support)

* Win10 testing (collecting dust lol)

* Windows 7 P2V (JIC my pc is down)

* Syslog

* Security

* vCenter (testing with VMUG advantage)

* other tests and unfinished projects..

 

 

GoPro 4K/60 footage gobbles up terabytes quick so that sorta started me on the idea of building a 2nd server. My next adventure is to get vMotion working between 2 standalone hosts. I do have shared storage (NFS) but it is slow (1GB vs 10GB) so I prefer to keep the VMs running locally until maybe one day when I can upgrade my QNAP NAS to one that supports 10GB.

 

 

I actually wanted to build a server at one point for WSUS, but then I downsized and had no need to have a WSUS server any longer. I did use my current server as a DC but again after down scaling, there was no need to use a server to enforce particular policies so I went back to standalone management.

 

Now Steam Cache is something I may find useful considering I have a steam library, I did look at it very briefly but I did not look into the ins and outs of setting it up, may have to revisit that.

 

Glad you were able to get the pictures sorted, looks sweet. Glad you use Noctua, I do not use anyone else to be honest. As it happened I got a 24 Port PoE Switch the other day (Managed) and it has 3x40mm fans (little overkill IMHO) but they are so loud, which made complete sense after I removed them to find they are Delta fans...so good fans but way to loud for the office ( thought I do know this one is meant for a rack lol). I purchased 3x Noctua 40mm fans which come with low noise adapters which should help nicely. I was considering only getting one or two, but I can only assume that if you are using a good amount of PoE that the PSU gets toasty...so why risk it over a few quid lol.

Edited January 17, 2020 by ENTERPRISE

[Laithan]

 

I actually wanted to build a server at one point for WSUS, but then I downsized and had no need to have a WSUS server any longer. I did use my current server as a DC but again after down scaling, there was no need to use a server to enforce particular policies so I went back to standalone management.

 

Now Steam Cache is something I may find useful considering I have a steam library, I did look at it very briefly but I did not look into the ins and outs of setting it up, may have to revisit that.

 

Glad you were able to get the pictures sorted, looks sweet. Glad you use Noctua, I do not use anyone else to be honest. As it happened I got a 24 Port PoE Switch the other day (Managed) and it has 3x40mm fans (little overkill IMHO) but they are so loud, which made complete sense after I removed them to find they are Delta fans...so good fans but way to loud for the office ( thought I do know this one is meant for a rack lol). I purchased 3x Noctua 40mm fans which come with low noise adapters which should help nicely. I was considering only getting one or two, but I can only assume that if you are using a good amount of PoE that the PSU gets toasty...so why risk it over a few quid lol.

 

Yes those Delta fans I think could be used as spare parts for Jet engines

 

 

 

[ENTERPRISE]

Haha your telling me, these are the new guys.

 

[Laithan]

I never bother with the LNA as they are quiet enough already (especially when you compare to Delta lol). I even Dyed my Nocuta fans black in my PC lol (I have 8 Nocuta fans, all Dyed black). Having said that, I do think the "Beige" color looks pretty good in a server case especially with chrome grill grates.

[ENTERPRISE]

I never bother with the LNA as they are quiet enough already (especially when you compare to Delta lol). I even Dyed my Nocuta fans black in my PC lol (I have 8 Nocuta fans, all Dyed black). Having said that, I do think the "Beige" color looks pretty good in a server case especially with chrome grill grates.

 

Yeah, I have yet to try them without any LNA, I will see how load they are at ''Stock'' first. Haha yeah I have some black Noctua's but I bought them that way but I was tempted to dye some earlier in the year. The funny thing is that the beige and brown colour is such a Noctua thing...you see a fan with those colours and you know what it is lol.

[Laithan]

And at least they aren't RGB lol

(OMG I am sooooo sick of RGB... kill it already)

[schuck6566]

And at least they aren't RGB lol

(OMG I am sooooo sick of RGB... kill it already)

 

before U knock RGB fans, when they are free from the contest on here, you can't really complain... https://www.in-win.com/en/fans/aurora <--- what I won. Edit: If you're gonna be saving 4k/60fps video, I found the drives for you... lol.

 

Edited January 19, 2020 by ENTERPRISE

[Laithan]

Actually those are different, I do like those GTZ!

 

Those drives! <drool> Yeah I'm sure before I know it I'll be needing those. Mr. Wallet strictly objects right now. lol

 

 

[ENTERPRISE]

Certainly nice storage drives but with them being SMR, the speed would take a hit I believe.

[Laithan]

Certainly nice storage drives but with them being SMR, the speed would take a hit I believe.

 

I had not really looked into SMR drives until now. It seems they are very different and although they look the same, they should not be used in the same way. They are best for write once, read many times scenarios and it seems that using these drives in a "standard workload" where there are frequent writes could result in errors...even crashing the O/S from what I read.

 

Yikes, cannot simply look at the capacity/price anymore, now you have to research underlying technology also. Some good info here.

[ENTERPRISE]

Certainly nice storage drives but with them being SMR, the speed would take a hit I believe.

 

I had not really looked into SMR drives until now. It seems they are very different and although they look the same, they should not be used in the same way. They are best for write once, read many times scenarios and it seems that using these drives in a "standard workload" where there are frequent writes could result in errors...even crashing the O/S from what I read.

 

Yikes, cannot simply look at the capacity/price anymore, now you have to research underlying technology also. Some good info here.

 

Lots of people actually buy external hdd's as they are cheaper in order to simply rip out the OEM SMR Drives. For simple storage, these are fine but I hear people use these in NAS enclosures for example and I cannot think of anything worse personally. NAS's are generally 24/7 beasts and is something you can end up doing lots of reads and writes too, so I am not sure why people thing they are suitable for that use scenario.

 

Anytime I buy a new HDD, I scrutinize it a fair it, especially if they are for use in my NAS, but I only either use NAS drives or surveillance drives just due to their endurance.

[Laithan]

Worth a watch re: CMR vs SMR. I personally plan to stay clear of these regardless of the use case.

 

[ENTERPRISE]

I have always know to stay well clear from SMR drives. I will not use them for anything as the performance is just painful, even for portable uses. The fact that WD is still going to use them for one of the RED lines for NAS use is just upsetting. Never use SMR drives in a NAS unless you enjoy poor performance.