The Great Tesla M40 Adventure "Projeckt T3sla"

[Avacado]

Need your help @tictoc. I have the rig running, it was plagued by BSODs and ity seemed to be better after a BIOS update. It does not see the 5 Teslas, I don't see an option for above 4g decoding in BIOS. 

 

*Edit. Found 4g by disabling CSM support. Still, no joy on getting the system to see any of the 5 Teslas attached. I have exhausted my knowledge on this one. 

Edited July 26, 2021 by Avacado

[tictoc]

I know that option is in there somewhere, but I don't remember for sure where it is.  I think it's in the "Boot" section of the UEFI. The last time I ran it I had multiple GPUs running plus at least one GPU on a m.2->PCIe card. 

[Avacado]

6 minutes ago, tictoc said:

I know that option is in there somewhere, but I don't remember for sure where it is.  I think it's in the "Boot" section of the UEFI. The last time I ran it I had multiple GPUs running plus at least one GPU on a m.2->PCIe card. 

Yea. Im running all 4 slots with 2x 4x M.2 risers. I can't figure this one out. 

[tictoc]

I know that I had some shenanigans when I tried to run headless with that board.  Are you able to boot into the OS?  If not you might try putting a GPU with display out in the primary GPU slot.  Also, make sure your boot drive is GPT formatted and not MBR, since you have to disable CSM in order to enable Above 4G decoding.

[Avacado]

1 minute ago, tictoc said:

I know that I had some shenanigans when I tried to run headless with that board.  Are you able to boot into the OS?  If not you might try putting a GPU with display out in the primary GPU slot.  Also, make sure your boot drive is GPT formatted and not MBR, since you have to disable CSM in order to enable Above 4G decoding.

I can load into OS fine. It sees the GT710, just cant figure out how to get it to recognize the Teslas. 

[tictoc]

I am probably not going to be much help on Windows. 

 

Do the GPUs show up in device manager at all?  Only suggestion I would have is to just add GPUs one by one to see if it is an issue with that many GPUs.

 

Also, you could disable HD Audio and some USB ports in the BIOS, to check if it is an allocation issue.

Edited July 26, 2021 by tictoc

[Avacado]

5 minutes ago, tictoc said:

I am probably not going to be much help on Windows. 

 

Do the GPUs show up in device manager at all?  Only suggestion I would have is to just add GPUs one by one to see if it is an issue with that many GPUs.

 

Also, you could disable HD Audio and some USB ports in the BIOS, to check if it is an allocation issue.

Will try that. I don't have the bandwidth for anymore failure tonight. I had tried just one of them, no joy. Nothing seen in device manager with 5 or 1. I will disable some ports and HD tomorrow, thanks for the tip.

Edited July 26, 2021 by Avacado

[teslaM40tester]

Hello to everyone,

 

I found this forum while searching for info about the M40. A short introduction to me: have been fiddling with computers for decades now, first machine was a VIC 20, in my youth I modded my ATARI STs to 16Mhz, I know how to solder or read out an EEPROM, so I´d guess I am on an enthusiast level

 

I got my hands on 6 TESLA M40 with 12GB. Managed to get it running in windows, did some CUDA tests, in WDMM Mode some DX12 calculations, OpenCL, Vulkan and right now I am working on rerouting the Output through an RX550 in order to use it for casual gaming, maybe for the kids....

 

I was hoping you guys could help me with a certain issue:

 

I´d like to get into training my own models (ESRGAN would be my starting point, seems well documented) - therefore I want to put as many of the M40 in one machine as possible. So far I only found bigger Threadripper boards which offer 4 or more full 16x PCIE Slots and have enough PCIE Lanes to fully support the M40s... But: These are quite expensive - and actually, I don´t think I need full bandwith. I´d probaly get along with 4x speed. But I´ve read several mentionings that the TESLAS only work on 16x Slots.

 

Some older XEON Boards seem to have seperate PCIE Splitters on Board, enabling real 4 x PCIe16x Slots. Some people report flashing a QUadro Bios on the Tesla, opening up the possibility to let them run on 1x, 4x, 8x slots...

 

How coul I manage to get at least 4 M40s running on a motherboard (which I have to buy then) without spending too much on board and CPU?

 

I even have a small Riser-Thingy for mining (which I don´t really do, just out of interest, I fiddled with it) which has 4 PCIE16X slots, but only at 1x Speed .... Would be convinient, if this could work.

 

Any thoughts?

 

Thanx

[Avacado]

2 hours ago, teslaM40tester said:

Hello to everyone,

 

I found this forum while searching for info about the M40. A short introduction to me: have been fiddling with computers for decades now, first machine was a VIC 20, in my youth I modded my ATARI STs to 16Mhz, I know how to solder or read out an EEPROM, so I´d guess I am on an enthusiast level

 

I got my hands on 6 TESLA M40 with 12GB. Managed to get it running in windows, did some CUDA tests, in WDMM Mode some DX12 calculations, OpenCL, Vulkan and right now I am working on rerouting the Output through an RX550 in order to use it for casual gaming, maybe for the kids....

 

I was hoping you guys could help me with a certain issue:

 

I´d like to get into training my own models (ESRGAN would be my starting point, seems well documented) - therefore I want to put as many of the M40 in one machine as possible. So far I only found bigger Threadripper boards which offer 4 or more full 16x PCIE Slots and have enough PCIE Lanes to fully support the M40s... But: These are quite expensive - and actually, I don´t think I need full bandwith. I´d probaly get along with 4x speed. But I´ve read several mentionings that the TESLAS only work on 16x Slots.

 

Some older XEON Boards seem to have seperate PCIE Splitters on Board, enabling real 4 x PCIe16x Slots. Some people report flashing a QUadro Bios on the Tesla, opening up the possibility to let them run on 1x, 4x, 8x slots...

 

How coul I manage to get at least 4 M40s running on a motherboard (which I have to buy then) without spending too much on board and CPU?

 

I even have a small Riser-Thingy for mining (which I don´t really do, just out of interest, I fiddled with it) which has 4 PCIE16X slots, but only at 1x Speed .... Would be convinient, if this could work.

 

Any thoughts?

 

Thanx

Welcome to the forums! The short answer to your question is yes, Teslas can run less than 16x. You can make use of your M.2. Slots with M.2. To PCIe 4x risers which are affordable and can be found on eBay. PCIe lanes will indeed be a challenge and as you noted, either Xeon or TR are your only real choices. You are going to have to bite the bullet if you go TR. The CPUs are reasonable but the x399 boards are crazy expensive.

 

Past using them for folding at home, all of the ways you intend on using them are Chinese to me. 

Edited August 19, 2021 by Avacado

[ENTERPRISE]

On 19/07/2021 at 14:24, Joe__Bishop said:

DId it burn through PCB?

It looks like just cap gave its life, should be able to solder a new one

Thus said, it looks like it is directly off power delivery lane

 

So I would check that power cable of psu and taste psu as well for just in case

Smell and lick taste are the best

 

I swear, I thought it was an actual product of forum, I would totally buy as sticky notepad for my home

It is a good product idea Lol

[teslaM40tester]

On 19/08/2021 at 13:10, Avacado said:

Welcome to the forums! The short answer to your question is yes, Teslas can run less than 16x. You can make use of your M.2. Slots with M.2. To PCIe 4x risers which are affordable and can be found on eBay. PCIe lanes will indeed be a challenge and as you noted, either Xeon or TR are your only real choices. You are going to have to bite the bullet if you go TR. The CPUs are reasonable but the x399 boards are crazy expensive.

 

Past using them for folding at home, all of the ways you intend on using them are Chinese to me. 

Thank you, that helps a lot. So I will test the Teslas on a slot with less than 16x and see if it works. If it does, I will investigate if the slower speed on the slot makes a difference... For Calculations, I might even revert to some SLI Bridge, which probably will even out the lower bandwith of the port. This way I might get along with a "normal Board"

[Avacado]

I was not able to make this project successful and it is now closed. All Tesla M40's/X399 boards have been sold on ebay. If anyone like @teslaM40tester has any questions moving forward, I will be more than happy to share what I have learned working with them. 

Edited August 24, 2021 by Avacado

[teslaM40tester]

Just a short reply, since I had a few Minutes to test somenthing out:

 

Managed to get one M40 running on a "miningriser" - which is a 1xPCIE -> 4 x 16x Slot Adapter with a little splitter on it. Seems the Splitter reduces the slots to PCIE2 Gen, so basically, the M40 ran at 1 x GEN2 PCIE

 

So that confirms that it can run on older Gens and not full lane supported Slots - which is a very good thing.

 

Of course every workload that is dependend on a high bandwith on the PCIE slot might suffer that way - heavy calculations on the card where only a small result from the calculaton is delieverd over the bus are no problem (like mining, FAH, boinc, 3D, rendering, etc...)

 

So just for fun, I did some tests with two M40 in these slots:

 

Blender rendering is fun, alsmost double the speed of one card, even on low pcie speeds

octane bench: 2 x M40 -> faster than one  Tesla P100, 1080TI etc.... Almopst in the ballpark of small RTX cards - not too bad...

 

Since I know now that older and smaller PCIE slots are not problem, I will take a look at how I can easily but all 6 of them to use, maybe for a Blender crunchy box...

 

 

[Avacado]

58 minutes ago, teslaM40tester said:

Just a short reply, since I had a few Minutes to test somenthing out:

 

Managed to get one M40 running on a "miningriser" - which is a 1xPCIE -> 4 x 16x Slot Adapter with a little splitter on it. Seems the Splitter reduces the slots to PCIE2 Gen, so basically, the M40 ran at 1 x GEN2 PCIE

 

So that confirms that it can run on older Gens and not full lane supported Slots - which is a very good thing.

 

Of course every workload that is dependend on a high bandwith on the PCIE slot might suffer that way - heavy calculations on the card where only a small result from the calculaton is delieverd over the bus are no problem (like mining, FAH, boinc, 3D, rendering, etc...)

 

So just for fun, I did some tests with two M40 in these slots:

 

Blender rendering is fun, alsmost double the speed of one card, even on low pcie speeds

octane bench: 2 x M40 -> faster than one  Tesla P100, 1080TI etc.... Almopst in the ballpark of small RTX cards - not too bad...

 

Since I know now that older and smaller PCIE slots are not problem, I will take a look at how I can easily but all 6 of them to use, maybe for a Blender crunchy box...

 

 

I am glad it will work with your application, I stand by what I said, if you really want to run 6+ of them, utilize the M.2. Slots @4x. with risers. Any newer gen card is backwards compatible with previous gen speeds. I.E. Gen 3 card can run Gen 1/2 depending on PCI-e lane bifurcation and capability of the motherboard. 

 

Also, I don't know what chassis you will be running them in, but they have terrible thermal performance if using standard air passthrough in the passive style. I recommend the blower style fans w/ 3D printed mounts and wouldn't you know it, I have a surplus. If you are interested, let me know and i'll send a few your way.  

Edited August 26, 2021 by Avacado

[damric]

Ok I remember reading this build log back in 2021. I just read through it again

 

Questions:

 

1. Do these Teslas work with Windows 10 Pro or do I need Enterprise edition? 

 

2. Can I use any card or iGPU for display output with these or is it finicky?

 

3: You said overclocking was possible in Afterburner? Did you the Maxwell BIOS editor thing? I'm wondering if voltage could be changed in there like on the Maxwell Quadros.

 

4. Is $70 for the 24GB card a good deal? It's local and the guy even has the power adapters.

[Avacado]

1. Yes, they will work with win 10 Pro.

 

2. In my experience on windows based platforms they are very finicky. I try to put them on a board where the only driver installed is a tesla one. An iGPU board would be the best solution, second preference would be a quadro for display out, they tend to be less conflicting than with GeForce drivers. 

 

3. Overclocking is only minimally possible, there is no default core OC or MEM OC in AB. All the sliders are disabled by default. You can however vBIOS mod them with Maxwell BIOS Tweaker. You will have to play with it and see what will work for you.

 

4. $70 Is a good deal for the card, yes. Assuming it is functional. You will thermal throttle if you try to load the card 100% while on passive cooling. I think I still have an extra 3D printed shroud with fan for the M40's. They are loud AF, but you will not be able to fold without REALLY good active cooling, I promise. 

[Cerberus]

make sure to have a recent board, something that can do above 4g encode.

[Avacado]

6 minutes ago, Cerberus said:

make sure to have a recent board, something that can do above 4g encode.

Yes, 100% if you plan to have more than one. 

[damric]

18 minutes ago, Avacado said:

1. Yes, they will work with win 10 Pro.

 

2. In my experience on windows based platforms they are very finicky. I try to put them on a board where the only driver installed is a tesla one. An iGPU board would be the best solution, second preference would be a quadro for display out, they tend to be less conflicting than with GeForce drivers. 

 

3. Overclocking is only minimally possible, there is no default core OC or MEM OC in AB. All the sliders are disabled by default. You can however vBIOS mod them with Maxwell BIOS Tweaker. You will have to play with it and see what will work for you.

 

4. $70 Is a good deal for the card, yes. Assuming it is functional. You will thermal throttle if you try to load the card 100% while on passive cooling. I think I still have an extra 3D printed shroud with fan for the M40's. They are loud AF, but you will not be able to fold without REALLY good active cooling, I promise. 

Ok I'm probably going to get it. It seems like something fun to mess around with and maybe put under water. I'll hit you up for the accessories.

[ENTERPRISE]

2 hours ago, damric said:

Ok I'm probably going to get it. It seems like something fun to mess around with and maybe put under water. I'll hit you up for the accessories.

As @Avacadosays, these boys run HOT HOT HOT. They need a good amount of active cooling to remain at stock speeds when at 100% loads, otherwise you throttle.