Best RAID Card For NVMe ?

[ENTERPRISE]

Hey Chaps, 

 

So I wondered if any of you had any recommendations as to a good RAID card for NVMe drives. I did a little searching, most of the add-in cards for NVMe support RAID, but do not setup the RAID independently from the motherboard etc, which ultimately in my mind is better should you have any motherboard issues, at least your array is not tits up ! I did take a look and found this boy: HighPoint M.2 NVMe RAID Controller via PCI-Express 4.0 x16, but while I am sure it is awesome and certainly would fulfill my needs....it is whack expensive lol. 

 

Any ideas ? 

 

Thanks,

E

[KyadCK]

Software RAID, or use AMD or Intel's built in tools. No motherboard is capable of NVMe RAID, they just enable you to use RST/VROC or AMD's RAID tools. Even the Highpoint drive is just a PLX chip and software;

 

https://www.anandtech.com/show/16247/highpoint-updates-nvme-raid-cards-for-pcie-40-up-to-8-m2-ssds

Quote

HighPoint's NVMe RAID is implemented as software RAID bundled with adapter cards featuring Broadcom/PLX PCIe switches. HighPoint provides RAID drivers and management utilities for Windows, macOS and Linux. Competing software NVMe RAID solutions like Intel RST or VROC achieve boot support by bundling a UEFI driver in with the rest of the motherboard's firmware.

 

NVMe drives are already RAID cards. They have LPDDR4 Cache, they have multi-core controllers, and they have stacks of Flash NAND (SSDs) on data buses. They can already push >4GB/s. There is no bus that will give you a performance benefit over PCI-e 4.0 that you can use on modern desktop hardware, and adding another step in the chain will hurt latency. There is simply no controller short of an actual FPGA, ASIC, or true CPU that can keep up.

 

What is your end goal? If you are attempting to protect data, then get a automated networked backup solution. Unless you absolutely need higher sequential read/writes, you will see no benefit from RAID, better to just pony up for next gen Optane or whatever the fastest PCI-e 4.0 NAND SSD is at the time.

Edited December 30, 2020 by KyadCK

[ENTERPRISE]

10 hours ago, KyadCK said:

Software RAID, or use AMD or Intel's built in tools. No motherboard is capable of NVMe RAID, they just enable you to use RST/VROC or AMD's RAID tools. Even the Highpoint drive is just a PLX chip and software;

 

https://www.anandtech.com/show/16247/highpoint-updates-nvme-raid-cards-for-pcie-40-up-to-8-m2-ssds

 

NVMe drives are already RAID cards. They have LPDDR4 Cache, they have multi-core controllers, and they have stacks of Flash NAND (SSDs) on data buses. They can already push >4GB/s. There is no bus that will give you a performance benefit over PCI-e 4.0 that you can use on modern desktop hardware, and adding another step in the chain will hurt latency. There is simply no controller short of an actual FPGA, ASIC, or true CPU that can keep up.

 

What is your end goal? If you are attempting to protect data, then get a automated networked backup solution. Unless you absolutely need higher sequential read/writes, you will see no benefit from RAID, better to just pony up for next gen Optane or whatever the fastest PCI-e 4.0 NAND SSD is at the time.

Hey bud,

 

Thanks for the response and info. The purpose of this is not so much day to day gains or for backup reasons. This is more of a study project/review for balls to the wall speed in RAID 0. I understand the use cases are niche.

 

I would need to do some research on the add in cards benefit I mentioned over a standard expansion add in card.

[KyadCK]

It's not going to go as cleanly as you might expect, but you are obviously welcome to try. You're probably going to want to reach out to Level1Techs about it if you plan to go over a few drives.

 

https://forum.level1techs.com/t/fixing-slow-nvme-raid-performance-on-epyc/151909

 

Spoiler

 

 

 

 

 

There is no benefit to add in cards because they do not perform any function. It is that simple, there are no RAID cards for NVMe, only software, and at these speeds the file systems themselves break if the CPU doesn't first.

 

What a time to be alive eh?

Edited December 30, 2020 by KyadCK

[ENTERPRISE]

3 hours ago, KyadCK said:

It's not going to go as cleanly as you might expect, but you are obviously welcome to try. You're probably going to want to reach out to Level1Techs about it if you plan to go over a few drives.

 

https://forum.level1techs.com/t/fixing-slow-nvme-raid-performance-on-epyc/151909

 

  Reveal hidden contents

 

 

 

 

 

There is no benefit to add in cards because they do not perform any function. It is that simple, there are no RAID cards for NVMe, only software, and at these speeds the file systems themselves break if the CPU doesn't first.

 

What a time to be alive eh?

Thanks for the great info then. In that case will stick with onboard slots for testing (Saves cash), though an add in card may make life easier in the future as a future WC GPU will block one of my onboard m.2 slots making it a pain to swap out.

 

Not having played with NVMe raid, Is it yet possible to create a bootable array ? So you can have windows boot from it ? 

 

 

[KyadCK]

AMD: 

https://www.amd.com/system/files/TechDocs/AMD NVMe_SATA RAID Quick Start Guide for Windows OS.pdf

 

Intel: 

https://www.intel.com/content/dam/support/us/en/documents/server-products/server-boards/Intel-VROC-Quick-Configuration-Guide.pdf

https://www.intel.com/content/www/us/en/support/products/122484/memory-and-storage/ssd-software/intel-virtual-raid-on-cpu-intel-vroc.html

 

Its not much different from how you would make a bootable motherboard RAID SATA array. You'll need to set up the array in BIOS, add the driver during install, all that fun stuff.

[ENTERPRISE]

1 hour ago, KyadCK said:

AMD: 

https://www.amd.com/system/files/TechDocs/AMD NVMe_SATA RAID Quick Start Guide for Windows OS.pdf

 

Intel: 

https://www.intel.com/content/dam/support/us/en/documents/server-products/server-boards/Intel-VROC-Quick-Configuration-Guide.pdf

https://www.intel.com/content/www/us/en/support/products/122484/memory-and-storage/ssd-software/intel-virtual-raid-on-cpu-intel-vroc.html

 

Its not much different from how you would make a bootable motherboard RAID SATA array. You'll need to set up the array in BIOS, add the driver during install, all that fun stuff.

 

Ahh well that is simple then. Have had RAID arrays before and seems no different now.

[J7SC_Orion]

On 12/30/2020 at 2:50 AM, ENTERPRISE said:

Hey bud,

 

Thanks for the response and info. The purpose of this is not so much day to day gains or for backup reasons. This is more of a study project/review for balls to the wall speed in RAID 0. I understand the use cases are niche.

 

I would need to do some research on the add in cards benefit I mentioned over a standard expansion add in card.

  

We priced out an alternate solution at the office the other day re. fast backups: U.2 SSDs ..as motherboard vendors rarely offer those onboard, there is a little M.2 > U.2 adapter (older MSI video below). The question is whether that costs U.2 speed, and also whether it is able to work with PCIe 4.0...

 

Intel's new PCIe 4.0 U.2 Optane drives are insane performers, and unfortunately priced accordingly, but there some good used PCIe 3.0 Optane drives to be had...more on those > here

 

 

 

[ENTERPRISE]

28 minutes ago, J7SC_Orion said:

  

We priced out an alternate solution at the office the other day re. fast backups: U.2 SSDs ..as motherboard vendors rarely offer those onboard, there is a little M.2 > U.2 adapter (older MSI video below). The question is whether that costs U.2 speed, and also whether it is able to work with PCIe 4.0...

 

Intel's new PCIe 4.0 U.2 Optane drives are insane performers, and unfortunately priced accordingly, but there some good used PCIe 3.0 Optane drives to be had...more on those > here

 

 

 

Optane would of course be an option for RAW speed but my project really only calls for NVMe in RAID. However will look into those U.2 Adapters just out of interest.

 

 

[J7SC_Orion]

@ENTERPRISE  @Laithan ...a bit off-topic, though this would hold a heck of a lot of nvme raid cards, and apparently also has 2x U.2 onboard...

 

From the 'pressing-your-nose-on-the-Porsche-showroom-window' file:  Gigabyte's upcoming TRX80 Threadripper Pro board...up to 2 TB 8-channel RAM (!), 7x PCIe 4 x 16, dual 10 GbE Lan - and it's black/silver shiny ! 

 

source

 

Edited January 1, 2021 by J7SC_Orion

[ENTERPRISE]

14 minutes ago, J7SC_Orion said:

@ENTERPRISE  @Laithan ...a bit off-topic, though this would hold a heck of a lot of nvme raid cards, and apparently also has 2x U.2 onboard...

 

From the 'pressing-your-nose-on-the-Porsche-showroom-window' file:  Gigabyte's upcoming TRX80 Threadripper Pro board...up to 2 TB 8-channel RAM (!), 8x PCIe 4 x 16, dual 10 GbE Lan - and it's black/silver shiny ! 

 

source

 

Ha, That is one nice looking board. If I were on that platform, that would certainly be a choice !

[Laithan]

On 1/1/2021 at 2:12 PM, J7SC_Orion said:

@ENTERPRISE  @Laithan ...a bit off-topic, though this would hold a heck of a lot of nvme raid cards, and apparently also has 2x U.2 onboard...

 

From the 'pressing-your-nose-on-the-Porsche-showroom-window' file:  Gigabyte's upcoming TRX80 Threadripper Pro board...up to 2 TB 8-channel RAM (!), 7x PCIe 4 x 16, dual 10 GbE Lan - and it's black/silver shiny ! 

 

source

 

 

Now that's my kind of motherboard!!

 

With regard to NVMe RAID, I think the current answers are only true... for now...

 

Like they say "If you build it, they will come"... I think that it is only a matter of time but there are a couple of barriers that only time can really address.

 

For one, I think with desktop board the # of PCI-e lanes is extremely limited (for NVMe) creating a bottleneck fairly quickly. PCI-e 4.0 can help with this however it might not be until NVMe is supported on PCI-e 5.0 where this can really shine.

 

The other issue is the "controller".. since NVMe interfaces directly with PCI-e natively, a "controller" isn't really what is needed in a traditional sense. NVMe RAID would essentially have to be an add-in PCI-e device or emulator, in hardware that gives each NVMe dedicate PCI-e bandwidth and presents those drives to the PCI-e interface on the motherboard and "seen" as a single drive. I think this is something we will in the next few years possibly but we are just not there yet IMO... but I could be totally wrong, we shall have to wait and see.

[J7SC_Orion]

..one of those new WRX80 TR Pro workstation monsters (which come with 2x U.2) would be nice with 2x Intel's new P5800X Optane (PCIe 4.0) in RAID  ...one probably could even add M.2 to U.2 adapters and then two more P5800X before ever even using one of the PCIe 4.0 x 16 slots .

 

Sounds like a great plan ...until one hits the checkout / cashier 

  

 

Edited January 30, 2021 by J7SC_Orion

[ENTERPRISE]

FYI, I ended up buying the HighPoint M.2 NVMe RAID Controller via PCI-Express 4.0 x16, not so much for playing with RAID (not that it is needed to do RAID) but because it also allows me to swap out NVMe drives more easily than compared to when they are on my motherboard. Will also be reviewing this add in card so keep your eyes peeled for that

[Laithan]

2 hours ago, ENTERPRISE said:

FYI, I ended up buying the HighPoint M.2 NVMe RAID Controller via PCI-Express 4.0 x16, not so much for playing with RAID (not that it is needed to do RAID) but because it also allows me to swap out NVMe drives more easily than compared to when they are on my motherboard. Will also be reviewing this add in card so keep your eyes peeled for that

Are you able to set that up in RAID 0? Did you have to use Windows DM for the RAID? What kind of performance are you getting with ATTO? I guess I should wait for the full review

 

Edited January 30, 2021 by Laithan

[ENTERPRISE]

4 hours ago, Laithan said:

Are you able to set that up in RAID 0? Did you have to use Windows DM for the RAID? What kind of performance are you getting with ATTO? I guess I should wait for the full review

 

Haha. Yeah these will be in the review. I have not got the card in hand yet. Should be arriving soon. Then I need to get a pair of NVMe drives for RAID 0 Testing.

 

[Laithan]

I had heard of these a while back but forgot about them.. The Linus just posted a new video featuring the. The "HoneyBadger". Athough it doesn't use the term RAID, this appears to be a harware based solution that combines multiple NVMe drives together which would essentially be similar in functionality to RAID.

 

This thing is extremely impressive... 24GB/s 

https://www.liqid.com/products/composable-storage/element-lqd4500-pcie-aic-ssd

Edited January 31, 2021 by Laithan

[ENTERPRISE]

2 hours ago, Laithan said:

I had heard of these a while back but forgot about them.. The Linus just posted a new video featuring the. The "HoneyBadger". Athough it doesn't use the term RAID, this appears to be a harware based solution that combines multiple NVMe drives together which would essentially be similar in functionality to RAID.

 

This thing is extremely impressive... 24GB/s 

https://www.liqid.com/products/composable-storage/element-lqd4500-pcie-aic-ssd

That seems like a fairly similar device to the one I am getting to be honest, will be interesting. Either way they are impressive solutions for maximum throughput, much of which the majority of users would never need, or notice when doing your big standard tasks, aside from in system file transfers however, those will be super hot.

[gugugusg]

On 31/01/2021 at 23:06, ENTERPRISE said:

That seems like a fairly similar device to the one I am getting to be honest, will be interesting. Either way they are impressive solutions for maximum throughput, much of which the majority of users would never need, or notice when doing your big standard tasks, aside from in system file transfers however, those will be super hot.

can i see the review please?

[ENTERPRISE]

5 hours ago, gugugusg said:

can i see the review please?

Unfortunately the review was not undertaken as the product was not performing as required. I spent a good deal of time with Highpoint to resolve the issue of the poor RAID speeds, but at that time was unable to get it to work as I needed it to. 

 

As I purchased the item myself, I was only comitting to a review if it was viable, but as it wasn't I had to move to the next solution (on-board RAID) and abandoned the review.

[UltraMega]

NVME RAID 0 would be pretty crazy haha. 

 

Ultimately I think it would be really niche because there are a lot of other dedicated PCIe options if you need something faster than a typically NVME solid state drive, but it would be really cool. 

Edited October 23, 2021 by UltraMega

[ENTERPRISE]

1 hour ago, UltraMega said:

NVME RAID 0 would be pretty crazy haha. 

 

Ultimately I think it would be really niche because there are a lot of other dedicated PCIe options if you need something faster than a typically NVME solid state drive, but it would be really cool. 

Very niche. Honestly it is not required as non RAID 0 performance is just fine. I did it just because I could and actually wanted the convenience of combined drives. Like lots of things, on paper its awesome but in real world, much of the time you wouldn't notice.

[Laithan]

Did you end up buying the High Point SSD7103 Bootable 4X M.2 NVMe RAID Controller or was it a different model?