Dedicated PLEX media server - Need Plex Gurus

[Laithan]

 

I've been thinking of home surveillance options and would be interested if you went down this route in seeing what you do.

 

I will continue to update this thread with my testing

 

 

 

Oh nice....but now this gets me thinking...Stop giving me ideas that involve money haha.

 

ROFL

No better time to have a robust security system

 

Just be glad you don't have the "overkill" bug

[Laithan]

All done!

Final build pics

 

(all done with the hardware anyway, I still have heat testing, overclocking, Plex performance and likely some BlueIris testing.)

 

 

The new shorter SFF-8087 cables were perfect! All that clutter gone! The 60mm Delta fan was also perfect. I am using PWM on all but one fan (it supports PWM but I just have it maxxed).

 

 

 

 

The Delta fans were a good choice because there is a lot of heat to remove from this case but they are doing well. The power is averaging just above 400W @ idle (slight overclock) so when I am finished overclocking it will likely be more (if the cooler is strong enough).

 

PS. For the curious, no the HD bay fan doesn't keep the controllers cool. The LSI 9271's need direct cooling but they are nice and cool now

Edited August 9, 2020 by Laithan

[Laithan]

Here are some quick benchmarks on the storage.

 

C: = LSI 9271-8i / (3) SAMSUNG 850 PRO RAID 5

D: = LSI 9271-8i / (10) SEAGATE 7200 RPM RAID 5 + (1) SAMSUNG 850 PRO CACHECADE 2.0

 

Benchmarks performed with a test file size both smaller and larger than the controller cache size. This will show maximum performance within both of these scenarios.

 

 

The HDDs when not cached should have performed better methinks... I wonder if the overall performance is limited to the speed of the SSD in the CacheCade.... Hmmm.. The performance looks about what I would expect from a single fast SSD.

Edited August 9, 2020 by Laithan

[J7SC_Orion]

Wow, congrats !

That X79-E WS starts to look smaller with all those other components in there - good thing it as 72 instead the usual 40 PCIe lanes for that gen. BTW, do you know what the full power draw at the wall is when under full storage i/o load ?

[Laithan]

Wow, congrats !

That X79-E WS starts to look smaller with all those other components in there - good thing it as 72 instead the usual 40 PCIe lanes for that gen. BTW, do you know what the full power draw at the wall is when under full storage i/o load ?

 

I'm going to further investigate a few things related to storage and while I am at it I will test this and report back.

 

I decided to step back and leave everything stock at this time. I want to see how it performs before I go through the trouble of overclocking and dealing with additional heat. I already have an idea of the power with and without the overclock and a broad estimate would be roughly 50W-100W additional with the overclock applied.

 

In other news, I'm having an issue installing ASUS AI Suite III for Windows 10 x64... Apparently the latest updates keep breaking it. I found the newest (??) .60 version and it also refused to install. This is a problem because I was depending on using that application to manage my fan curves. I tried SpeedFan and it doesn't detect 2 of my fans . I just found FanExpert 4 standalone install so fingers crossed. If anyone has ever worked with PWM fans you'll know right away why a controller of some sort is needed, the constant reving up and down would drive anyone crazy

 

[Laithan]

Some info:

 

(1) The reason AU Suite III would not install is because they release these apps programmed for specific motherboard sensors. You MUST install an older one that supports the sensors in your board. In my case I needed to install an older ASUS AI Suite II which did not work well so I ended up with Argus monitoring software instead and it works great so far.

 

(2) @J7SC_OrionJ7SC_Orion With stock clocks I'm @ roughly 310W at the desktop idle.. When I fired up I/O meter and some other background copy jobs the most I saw was around 350W. If the CPU/GPU was being pushed at the same time it would be higher. I am curious how much power draw I'll see with Plex transcoding using both the cpu and gpu for different things.

 

I will experiement with EPU also if it doesn't give me too much trouble.

Edited August 10, 2020 by Laithan

[J7SC_Orion]

...I'm trying to figure out if you mean Asus AI Suite 3, if so, I ran into similar issues over the years...best to carry it forward over different CPU, chipset and the 'same' Win install, then update the latter. Just don't try uninstalling AI Suite 3 w/o first taking nerve-calming tonics

 

...looking forward to see wattage with Plex transcoding, plus all the components and peripherals you are running. Also, I have some foggy memories re. EPU vs TPU during the first go-around when my Asus X79-E WS was new...all I remember is that I decided to leave it on TPU, given preference for oc'ing goals.

[ENTERPRISE]

Fantastic build completion ! Nice work, looks like a great rig and I would say that your HDD array + HDD/SSD config looks to be performing at about expected speeds so not much to worry about there. As for EPU, disable that puppy as that relates to power savings I do believe. If power saving is a goal then leave it as it is, but if you are looking at maximum performance I tend to disable it.

[Laithan]

 

​

​

​

​

The HDDs when not cached should have performed better methinks... I wonder if the overall performance is limited to the speed of the SSD in the CacheCade.... Hmmm.. The performance looks about what I would expect from a single fast SSD.

 

Quoting myself like a champ

 

My spidey senses were correct. The performance was WAY lower than it should have been. It was indeed was bottlenecked by the single SSD. I wiped out the array and this time left the SSD out and did not enable cachecade. Notice the performance difference from the old scores above and now. In addition, since I now had an extra SSD installed I made an Acronis image of the O/S array, wiped that out also and added the 4th SSD. I got a little boost in performance there also (array was still initializing).

 

Note: Cache size below refers to CONTROLLER cache.

 

 

 

The downside of cachecade 2.0 (that I've now discovered):

(1) If the sequential performance of the SSD(s) being used is less than the total performance of the HDD array, then the SSD will become the bottleneck.

 

(2) If you lose the cache'd drive then you cannot access the rest of the array (I also did not know this). This means that a RAID 1 mirror would be a minimum for redundancy but also in order to exceed the performance of (10) 7,200rpm enterprise class drives (256MB cache each) I would need to use at least 4 or more likely 5 SSDs in RAID 5.

 

Note: Notice the (10) drives actually write faster than the (4) SSDs? Only with sequential of course..

 

Note2: I also realize that I may have not even been able to take full advantage of cachecade 2.0 since I'm not running a database, I did it because "I could" and to live up to my middle-name (overkill) lol.

 

Note3: You can now see why I am such a big fan of these LSI/AVAGO 9271-8i controllers. (10) HDD drives @ 150MB/s each = 1.5GB/s and I'm reaching that maxium threshold on the writes. This is excellent efficiency and usage of those drives either solid state or spinning. (4) SSD drives @ 550MB/s each = 2.2GB/s and again we can see the performance reaching just below that mark. If you are curious yourself, just don't forget the active cooling for them

 

Edited August 12, 2020 by Laithan

[Laithan]

Wow, congrats ! ​

That X79-E WS starts to look smaller with all those other components in there - good thing it as 72 instead the usual 40 PCIe lanes for that gen. BTW, do you know what the full power draw at the wall is when under full storage i/o load ?

 

Thank you, I am really happy with how it came out. I was able to get the fan situation sorted last night also by using a certain version of ASUS AI Suite II that I found. I have fan curves configured for all fans now .

 

I also did some burn-in testing. While running IOMeter on the HHD array I fired up the AIDA64 Burn-in test to stress the CPU/MEMORY/CACHE and I was able to pull about 450W from the wall. This test does NOT use the GPU so it could increase theoretically. I doubt transcoding is going to use a ton of power but I would say the server is going to use between 300W and 400W under typical "real world" load. I will certainly report actual transcoding speeds when I get to that point (last step).

 

 

Fantastic build completion ! Nice work, looks like a great rig. As for EPU, disable that puppy as that relates to power savings I do believe. If power saving is a goal then leave it as it is, but if you are looking at maximum performance I tend to disable it.

 

Thank you very much. Yeah power savings is kind of one of my goals since this will be running 24x7... I donate enough to Mr. Electric already lol.. I am not expecting magic but it appears to be a feature specific to ASUS that is "above and beyond" your typical power savings. I am assuming that would mean same performance, less power but I haven't really looked into EPU that much yet other than a thought of it possibly saving me money. I'm going to hold off until everything is fully up and running before I do any testing as I might need some real world tasks to stress things. If you're saying that what it really does is reduce performance in favor of power savings then yeah, I don't want any of that

[J7SC_Orion]

...Glad you got your custom fan-curve and SSD cache issues settled. As to the power consumption, I'm jelly...my main workstation can easily peak past 1070W (2x 2080 Ti @ 380W max EACH on stock bios, 1x oc'ed TR, various peripherals). 450W + - sounds like a dream

[Laithan]

The CPU is pretty old at this point but it is worth mentioning that the highest core/thread count consumer CPU on the X79 platform was 6 cores/12 threads. All of the unlocked processors were either 4/8 or 6/12. There is a fun fact here at the end.

 

The next "sorta" secret on this platform is that you can use Xeons which are unlocked. The E5-1650v2 and E5-1660v2 are actually unlocked processors and overclock extremely well. A lot of people knew this but the average person didn't.

 

Then we have a very special CPU that very few knew about (at the time).. the E5-1680v2. This is the ONLY 8 core/16 thread UNLOCKED processor on the entire platform (there was also a 10-core server CPU but it wasn't unlocked). There is also a very large 25MB L3 cache. This is the highest amount of cache per core on the entire platform if I am not mistaken. This processor was never sold on retail shelves. This processor was not available for OEMs. This processor was never sold in any Windows based workstations. This processor wasn't listed in the promotional material for the E5-16xxV2 series. Intel was getting ready to promote X99 where they would introduce desktop/workstation 8 core/16 threads (also with 25MB cache, so this CPU is basically an early Haswell). The ONLY way to obtain this E5-1680V2 processor when it launched was through a purchase of a 2013 Intel based MAC PRO. It is clear that Intel was sort of "hiding" this CPU as they wanted everyone to focus on the upcoming Haswell.

Edited August 13, 2020 by Laithan

[Laithan]

...Glad you got your custom fan-curve and SSD cache issues settled. As to the power consumption, I'm jelly...my main workstation can easily peak past 1070W (2x 2080 Ti @ 380W max EACH on stock bios, 1x oc'ed TR, various peripherals). 450W + - sounds like a dream

 

lol Well done! That's the way to do it!

That is a monster system!

<drool>

 

I'd love to build a TR system one day as I've never built or used one yet... How do you like it? When it comes time to refresh my ESXi server I suspect that's the platform I'll end up on and by then the prices should be more reasonable. Are any of the TR CPUs unlocked?

 

 

 

 

[J7SC_Orion]

 

lol Well done! That's the way to do it!

That is a monster system!

<drool>

 

I'd love to build a TR system one day as I've never built or used one yet... How do you like it? When it comes time to refresh my ESXi server I suspect that's the platform I'll end up on and by then the prices should be more reasonable. Are any of the TR CPUs unlocked?

 

 

...all TR CPUs are unlocked, afaik (don't know for sure about the new 3995X TR Pro line = 64c/128t, 8 channel Ram). However, with latest TR gens in particular, heavy cooling and 'PBO' setting usually gets you better every-day results than old-school multiplier / FSB all-core oc

 

 

[ENTERPRISE]

 

Thank you, I am really happy with how it came out. I was able to get the fan situation sorted last night also by using a certain version of ASUS AI Suite II that I found. I have fan curves configured for all fans now .

 

I also did some burn-in testing. While running IOMeter on the HHD array I fired up the AIDA64 Burn-in test to stress the CPU/MEMORY/CACHE and I was able to pull about 450W from the wall. This test does NOT use the GPU so it could increase theoretically. I doubt transcoding is going to use a ton of power but I would say the server is going to use between 300W and 400W under typical "real world" load. I will certainly report actual transcoding speeds when I get to that point (last step).

 

 

 

 

Thank you very much. Yeah power savings is kind of one of my goals since this will be running 24x7... I donate enough to Mr. Electric already lol.. I am not expecting magic but it appears to be a feature specific to ASUS that is "above and beyond" your typical power savings. I am assuming that would mean same performance, less power but I haven't really looked into EPU that much yet other than a thought of it possibly saving me money. I'm going to hold off until everything is fully up and running before I do any testing as I might need some real world tasks to stress things. If you're saying that what it really does is reduce performance in favor of power savings then yeah, I don't want any of that

 

That is fair then, leave that baby enabled !

 

The CPU is pretty old at this point but it is worth mentioning that the highest core/thread count consumer CPU on the X79 platform was 6 cores/12 threads. All of the unlocked processors were either 4/8 or 6/12. There is a fun fact here at the end.

 

The next "sorta" secret on this platform is that you can use Xeons which are unlocked. The E5-1650v2 and E5-1660v2 are actually unlocked processors and overclock extremely well. A lot of people knew this but the average person didn't.

 

Then we have a very special CPU that very few knew about (at the time).. the E5-1680v2. This is the ONLY 8 core/16 thread UNLOCKED processor on the entire platform (there was also a 10-core server CPU but it wasn't unlocked). There is also a very large 25MB L3 cache. This is the highest amount of cache per core on the entire platform if I am not mistaken. This processor was never sold on retail shelves. This processor was not available for OEMs. This processor was never sold in any Windows based workstations. This processor wasn't listed in the promotional material for the E5-16xxV2 series. Intel was getting ready to promote X99 where they would introduce desktop/workstation 8 core/16 threads (also with 25MB cache, so this CPU is basically an early Haswell). The ONLY way to obtain this E5-1680V2 processor when it launched was through a purchase of a 2013 Intel based MAC PRO. It is clear that Intel was sort of "hiding" this CPU as they wanted everyone to focus on the upcoming Haswell.

 

That is interesting, I did not know they did any unlocked Xeons during that period of time, glad to see you got lucky and manage to have an unlocked one at your disposal.

 

 

...all TR CPUs are unlocked, afaik (don't know for sure about the new 3995X TR Pro line = 64c/128t, 8 channel Ram). However, with latest TR gens in particular, heavy cooling and 'PBO' setting usually gets you better every-day results than old-school multiplier / FSB all-core oc

 

 

PBO does some wonders, it is a very good system assuming you have a great WC setup. That being said, aside from PBO, you can also get the best out of your CPU by per CCD overclocking. I have OC'ed better than PBO and better than trying for an all core OC by overclocking each CCD and bringing each one to its maximum. That is how I got 4525/4500/4500/4350 on my 3950X. An all core OC at 4.5+ was not possible on my chip so I decided I would get the best out of each CCD and leave it their. This also helps with keeping single core performance at its highest and not having to sacrifice that just because the other CCD's could not keep up. It does take a little more time than an all core OC or just leaving it to PBO. However once you get there, it is rewarding and kind of a fun process.

 

 

 

[Laithan]

 

...all TR CPUs are unlocked, afaik (don't know for sure about the new 3995X TR Pro line = 64c/128t, 8 channel Ram). However, with latest TR gens in particular, heavy cooling and 'PBO' setting usually gets you better every-day results than old-school multiplier / FSB all-core oc

 

 

You have to love that! AMD is doing great things

 

 

 

 

 

It does take a little more time than an all core OC or just leaving it to PBO. However once you get there, it is rewarding and kind of a fun process.

 

 

Exactly, I love that. I never actually heard of PBO until you mentioned it. Quite interesting indeed.

 

 

[Laithan]

I've got the RAMDISK installed (ASUS Ramdisk 2.03.00). We'll see how reliable it is but so far so good.

 

I've got 32GB total which the Plex server definitely won't need so why not.. It will save writes on my SSD array and the performance speaks for itself. This is only on 1600Mhz ram and nothing is overclocked yet so this is just baseline speed.

 

 

 

 

I was also testing network performance and I'm incredibly happy to see much better transfer rates vs running in a virtual machine. The bottleneck is the slowest drive involved, not the wire The screenshot below is from Windows 7 x64 connected to the Plex server via administrative share. SMB is a bit older on 7 although it doesn't really appear to be slowing down much if at all. Will have to test Windows 10 -> Windows 10.

 

 

 

I've pre-staged all currentl data (less than 3TB) and I'm pretty much ready to turn it into a Plex server now.

[ENTERPRISE]

I've got the RAMDISK installed (ASUS Ramdisk 2.03.00). We'll see how reliable it is but so far so good.

 

I've got 32GB total which the Plex server definitely won't need so why not.. It will save writes on my SSD array and the performance speaks for itself. This is only on 1600Mhz ram and nothing is overclocked yet so this is just baseline speed.

 

 

 

 

I was also testing network performance and I'm incredibly happy to see much better transfer rates vs running in a virtual machine. The bottleneck is the slowest drive involved, not the wire The screenshot below is from Windows 7 x64 connected to the Plex server via administrative share. SMB is a bit older on 7 although it doesn't really appear to be slowing down much if at all. Will have to test Windows 10 -> Windows 10.

 

 

 

I've pre-staged all currentl data (less than 3TB) and I'm pretty much ready to turn it into a Plex server now.

 

Nice, looking good.

 

Another program that may be of interest to you is Primo Cache. I use it for writing my data to my RAM first when it comes to files etc and then it will do a deferred write to my SSD's. The nice thing is that aside from the speed, as it defers the writes back to the SSD in bulk, it helps reduce the wear on your drives which you would get from multiple writes. I have used it for a good while now, super reliable and never caused me any instability.

[Laithan]

 

Nice, looking good.

 

Another program that may be of interest to you is Primo Cache. I use it for writing my data to my RAM first when it comes to files etc and then it will do a deferred write to my SSD's. The nice thing is that aside from the speed, as it defers the writes back to the SSD in bulk, it helps reduce the wear on your drives which you would get from multiple writes. I have used it for a good while now, super reliable and never caused me any instability.

 

Thank you

 

I will definitely take a look that looks very impressive. The server is on battery backup so I would be OK with trying it. I actually saw that ASUS offers something like that also call ed ASUS RamCache. Initially I confused it for the ASUS RamDisk because I didn't know that it existed. It actually sounds very much like Cachecache except taking it to the next level using system RAM and SSDs instead of SSDs and HDDs. I'd have to look and see if it is recommended for use with HDDs. I think at this point I'm just going to try everything as-is first and see how things perform. I am a little worried about the CPU since it does not have Quicksync but I'm feeling confident that the Turing NVENC/NVDEC will do the job. Fingers crossed!

 

[ENTERPRISE]

 

Thank you

 

I will definitely take a look that looks very impressive. The server is on battery backup so I would be OK with trying it. I actually saw that ASUS offers something like that also call ed ASUS RamCache. Initially I confused it for the ASUS RamDisk because I didn't know that it existed. It actually sounds very much like Cachecache except taking it to the next level using system RAM and SSDs instead of SSDs and HDDs. I'd have to look and see if it is recommended for use with HDDs. I think at this point I'm just going to try everything as-is first and see how things perform. I am a little worried about the CPU since it does not have Quicksync but I'm feeling confident that the Turing NVENC/NVDEC will do the job. Fingers crossed!

 

Yeah do not blame you for just using it as is, you can always look into it later if you run into issues, but for testing and assessing performance over time, you might as well leave it as is. By the way, do not forget to add this in Rig Creator, we are interested in the full specs of this beast.

[J7SC_Orion]

 

Thank you

 

I will definitely take a look that looks very impressive. The server is on battery backup so I would be OK with trying it. I actually saw that ASUS offers something like that also call ed ASUS RamCache. Initially I confused it for the ASUS RamDisk because I didn't know that it existed. It actually sounds very much like Cachecache except taking it to the next level using system RAM and SSDs instead of SSDs and HDDs. I'd have to look and see if it is recommended for use with HDDs. I think at this point I'm just going to try everything as-is first and see how things perform. I am a little worried about the CPU since it does not have Quicksync but I'm feeling confident that the Turing NVENC/NVDEC will do the job. Fingers crossed!

 

 

With a sophisticated build and all its components like this, there will be lots and lots of options to fine tune for months to coem. BTW, I have used and just love Asus RAM disk, one reason why I tend to run larger amounts of RAM in my builds. Asus RamDisk even works on some non-Asus boards if they are of similar vintage. That said, RamDisk obviously might not be such a good idea on a Plex media server, but it can make mincemeat out of the fastest nvme drives out there.

[Laithan]

 

Yeah do not blame you for just using it as is, you can always look into it later if you run into issues, but for testing and assessing performance over time, you might as well leave it as is. By the way, do not forget to add this in Rig Creator, we are interested in the full specs of this beast.

 

Thanks! I do keep forgetting to to that

 

I did get the Plex media server installed and configured and did some preliminary testing. I still have some tweaking to do as @J7SC_Orion said below. I noticed a little difficulty with 4K transcoding. I think this is the same issue just about everyone has and the only bottleneck that I am seeing is with a single core on the CPU.

 

I assume this is the AUDIO transcoding that is causing me issues... (have to look into workarounds perhaps)

 

 

 

 

 

 

With a sophisticated build and all its components like this, there will be lots and lots of options to fine tune for months to coem. BTW, I have used and just love Asus RAM disk, one reason why I tend to run larger amounts of RAM in my builds. Asus RamDisk even works on some non-Asus boards if they are of similar vintage. That said, RamDisk obviously might not be such a good idea on a Plex media server, but it can make mincemeat out of the fastest nvme drives out there.

 

 

Agree. This is my 2nd build during the pandemic (built a PC for my son also) so it was a good project and something to help keep me busy for a while. Thankfully in this build I didn't go in completely blind, the only component I didn't have experience with was the SAS expander and cachecade 2.0. Turns out it is just plug N play so I didn't even have to configure anything. I've been testing things and it has really gone very smooth for the most part. I ripped a 4K Ultra movie last night and everything is working as expected there. The last issue is the one above, if I am dealing with single core performance then I may need to dive into overclocking sooner than expected.

Edited August 15, 2020 by Laithan

[axipher]

Yes, audio transcoding is still a single core task. Being at 100% will happen on 1080p or 4k at various numbers of audio channels, it will just be able to transcode longer buffers and keep processing audio. It's normal and doesn't necessarily lock that core fully to harm other tasks on that thread.

 

I can't see a 3+ GHz core having issues with the audio processing unless the source is like 7.2 or Atmos or something and trying to transcode it to 2.1 or just stereo.

 

You could try and see if you can change the audio options on the playback client, but hopefully its not a hardware issue in that newer CPU's have some instructions that somehow speed that portion up.

[Laithan]

Yes, audio transcoding is still a single core task. Being at 100% will happen on 1080p or 4k at various numbers of audio channels, it will just be able to transcode longer buffers and keep processing audio. It's normal and doesn't necessarily lock that core fully to harm other tasks on that thread.

 

I can't see a 3+ GHz core having issues with the audio processing unless the source is like 7.2 or Atmos or something and trying to transcode it to 2.1 or just stereo.

 

 

Well, ya it was lol . Source was a 4K UHD movie.

 

I had multiple transcodes from 4k to 1080p running at the same time and I really didn't have any actual playback troubles.. I think I was alarmed about the core being maxxed out more than anything. My laptop was choking a bit because it only has 2 real cores (4 threads) but that wasn't the server's fault...

 

I also ran into some bandwidth issues with my TVs that are 4K capable. Plex wants to direct play them and it is too much for wireless. I managed to get it to transcode to 1080p on my TCL but the Samsung doesn't want to play.. again not the server's fault so overall I think I may actually be OK. It is good to know others see that high single core workload.

[Laithan]

I was starting to get alarmed because the entire RAMDISK was filling up..

 

 

but after some reading it appears that Plex will basically fill up whatever size you provide it and it will perform cleanup automatically as the drive gets down to 100MB free..

 

We shall see

 

EDIT: I didn't seem to have any problems... when I stopped the movie it cleaned it all up. I read that at least 4GB was recommended for each simultanoues transcode.

Edited August 15, 2020 by Laithan