Radeon Instinct MI25 16GB HBM2

[damric]

 

 

There's a couple guys selling a whole bunch of these old server cards on ebay. They are listed as buy-it-now for like $70-80, but one guy took my offer of $60 shipped. Not too bad since I sold an actual Vega 64 for $200 a couple months back. These server cards only have Linux drivers, but can be used with Windows if you flash it to WX9100, which also gives you output on the mini-display port. Thanks to a few guys at techpowerup for this idea.

 

 

These cards have no fans, meant for high rpm server chassis fans to blow through. I saw some 3D printed adapters, but I happen to have a cheap universal water block I'm mounting. Going to need a little bit of metal cutting to get it to fit!

 

Had to cut out that tab on the base plate. Also that heat pipe was just laying there, not soldered or anything so I coated the backside with MX-4.

Cheap ebay GPU block fit the Vega die exactly. Looks like the tubes will have enough room.

I wanted to keep the base plate because it keeps the VRMs cooled. I reused the stock thermal pads since they were in good condition. Base plate also connects the back plate.

Looks good enough like this but I wonder if the shroud cover will also fit back on?

Yes just had to trim the aluminum. I need to sand down the edges and put some black polish on there.

So this is how it will look. Going to get it running this week on one of my folding benches.

[SamsTechStuff]

It's a bummer about the non existent mxgpu driver for homelabbers but very cool to see that these can used for [email protected] How does it perform?

[damric]

30 minutes ago, SamsTechStuff said:

It's a bummer about the non existent mxgpu driver for homelabbers but very cool to see that these can used for [email protected] How does it perform?

In theory it will be the same as Vega 64, so 1-2 million per day +/- how good the WUs are.

[damric]

Filled the loop back up and letting the pump run overnight to check leaks and collect air bubbles.

That's a small cut I had to make on the shroud for the tubes

I like the look of the shroud, but if VRM thermals are bad then it's coming off and I'll strap a fan onto the base plate.

It's only a 240mm cheap aluminum radiator. I imagine it will be fine, but we'll find out tomorrow when I run it hard.

 

@neurotix wasn't joking when he said these used to be extremely expensive. I just looked it up and the MSRP was over $16,000. What was Raja thinking? I got it for $60!

Edited May 14 by damric

[neurotix]

Thanks for the pics and wow, I knew they were a few thousand dollars but didn't realize they were originally $16k

 

Reminds me of my used Nissan 300zx I used to have and that it was $32k in 1990. Was cool to own something that was originally that expensive.

 

Best of luck on your watercooling exploits, may they be leak free. It'll be interesting to see how much ppd that thing puts out.

[damric]

30 minutes ago, neurotix said:

Thanks for the pics and wow, I knew they were a few thousand dollars but didn't realize they were originally $16k

 

Reminds me of my used Nissan 300zx I used to have and that it was $32k in 1990. Was cool to own something that was originally that expensive.

 

Best of luck on your watercooling exploits, may they be leak free. It'll be interesting to see how much ppd that thing puts out.

I spent all morning messing around with it. Since it has no Windows drivers, only Linux, and the one mini-display output is disabled by default, I had two options:

 

1) Flash it to WX9100. This allows Windows drivers and enables the mini-dp. But there's no clock tuning in the driver like there was for Vega 64 or 56.

OR

2) Leave the firmware as MI25 and install a linux distro. Yeah I'm no good at that, and this would also mean no benchmarking, but at least it would fold at stock config.

 

I went with option #1. So now it's folding away happily and says 660K PPD on project 18717, but that's at the default clock/power config which seems to be running around 1200-1250MHz.

 

I know it can do more. I've got to figure out how to unlock clock adjustments. I think I might be able to flash the VEGA 64 FE (16GB) BIOS to it and be able to tune it, but I'll probably lose the mini-dp. Not a big deal since I have it installed with 5600G with integrated graphics, but I don't think that will allow me to run benchmarks on it. I'll have to play around and see.

 

BTW the coolant temp is fine with just the weak 240mm aluminum radiator. 34C while folding. GPU temp 44C, hot spot 70C.

Edited May 15 by damric

[neurotix]

19 minutes ago, damric said:

I spent all morning messing around with it. Since it has no Windows drivers, only Linux, and the one mini-display output is disabled by default, I had two options:

 

1) Flash it to WX9100. This allows Windows drivers and enables the mini-dp. But there's no clock tuning in the driver like there was for Vega 64 or 56.

OR

2) Leave the firmware as MI25 and install a linux distro. Yeah I'm no good at that, and this would also mean no benchmarking, but at least it would fold at stock config.

 

I went with option #1. So now it's folding away happily and says 660K PPD on project 18717, but that's at the default clock/power config which seems to be running around 1200-1250MHz.

 

I know it can do more. I've got to figure out how to unlock clock adjustments. I think I might be able to flash the VEGA 64 FE (16GB) BIOS to it and be able to tune it, but I'll probably lose the mini-dp. Not a big deal since I have it installed with 5600G with integrated graphics, but I don't think that will allow me to run benchmarks on it. I'll have to play around and see.

 

BTW the coolant temp is fine with just the weak 240mm aluminum radiator. 34C while folding. GPU temp 44C, hot spot 70C.

 

Linux won't help you, don't bother. There are drivers for it but AMD GPU support in Linux sucks. Afaik they don't even have a proper control panel, and overclocking is not available.

 

On the Nvidia side, with the official Nvidia drivers and an app called GreenWithEnvy, you get both a proper control panel and a very slick overclocking utility that can do a lot, unfortunately it lacks voltage control though.

[damric]

Well I discovered a nifty program called overdriventool, and it's allowing me to tweak clocks, power, and voltage, so it's all good now.

[neurotix]

Never heard of it, but congrats!

[neurotix]

Also, what are the stock clocks and what is your OC now? Does the HBM oc at all?

 

I never had Vega but I did have Crossfire Sapphire R9 Fury Nitro for a while. The HBM wouldn't budge and I think they didn't OC past 1150MHz. You probably remember the advertisement and slides for the Fury X with Koduri saying "they're an overclockers dream" before they came out, lol. They were great cards though and ran every game I had at Ultra except for Rise of the Tomb Raider because 4K textures took more than 4GB ram and the Fury's only had 4GB HBM.

 

Vega can clock much higher than my old Fury's, like at least 1800MHz or so right?

 

Let me know the stock clocks and what you've gotten the thing to now.

[damric]

55 minutes ago, neurotix said:

Also, what are the stock clocks and what is your OC now? Does the HBM oc at all?

 

I never had Vega but I did have Crossfire Sapphire R9 Fury Nitro for a while. The HBM wouldn't budge and I think they didn't OC past 1150MHz. You probably remember the advertisement and slides for the Fury X with Koduri saying "they're an overclockers dream" before they came out, lol. They were great cards though and ran every game I had at Ultra except for Rise of the Tomb Raider because 4K textures took more than 4GB ram and the Fury's only had 4GB HBM.

 

Vega can clock much higher than my old Fury's, like at least 1800MHz or so right?

 

Let me know the stock clocks and what you've gotten the thing to now.

Stock clock max boost is 1500MHz. When folding or doing anything serious it hits power limit and is more like 1200-1300MHz. I can exceed power limits as high as I want with registry tweaks but then the card just eats power and runs hot as hell to boost higher. Yeah I had a Vega 64 running steady at 1800MHz but with extremely cold water like 15C and a 450W power draw

 

It's got a high ASIC quality 94% so I'm going to experiment undervolting and see where that goes. I probably need to add active cooling to the back of the card too sice there's some bare VRMs there.

[neurotix]

2 minutes ago, damric said:

Stock clock max boost is 1500MHz. When folding or doing anything serious it hits power limit and is more like 1200-1300MHz. I can exceed power limits as high as I want with registry tweaks but then the card just eats power and runs hot as hell to boost higher. Yeah I had a Vega 64 running steady at 1800MHz but with extremely cold water like 15C and a 450W power draw

 

It's got a high ASIC quality 94% so I'm going to experiment undervolting and see where that goes. I probably need to add active cooling to the back of the card too sice there's some bare VRMs there.

Good luck. Yeah, not worth it to have that high of power draw for just like 50k more PPD.

 

Lol @ the Vega 64 450w power draw. I have my RTX 4090 oc'ed now and putting up about 22m-24m points daily and in my OC tool it shows wattage and the highest it gets is around 350w. Using the stock "gaming" bios this time, so a 450w limit but it never comes close to it.

 

I wonder, with the upgrades to your main rig and the full watercooling on it, as well as your test benches with weird pro cards and cheap watercooling components, if all that money you spent could have gotten you a 4090 instead by now that would dwarf everything you have running currently and at far less power consumption? Devil's Advocate: get a 4090

[damric]

1 minute ago, neurotix said:

Good luck. Yeah, not worth it to have that high of power draw for just like 50k more PPD.

 

Lol @ the Vega 64 450w power draw. I have my RTX 4090 oc'ed now and putting up about 22m-24m points daily and in my OC tool it shows wattage and the highest it gets is around 350w. Using the stock "gaming" bios this time, so a 450w limit but it never comes close to it.

 

I wonder, with the upgrades to your main rig and the full watercooling on it, as well as your test benches with weird pro cards and cheap watercooling components, if all that money you spent could have gotten you a 4090 instead by now that would dwarf everything you have running currently and at far less power consumption? Devil's Advocate: get a 4090

What fun is that?

[neurotix]

17 minutes ago, damric said:

What fun is that?

True enough. I'm glad you enjoy your ghetto watercooling and benches and since I can't do it myself (but want to) I am honestly envious of your ability. You did a really great job modding this card the way you did, especially since it keeps the stock look of it. Good job bud.

[damric]

7 minutes ago, neurotix said:

True enough. I'm glad you enjoy your ghetto watercooling and benches and since I can't do it myself (but want to) I am honestly envious of your ability. You did a really great job modding this card the way you did, especially since it keeps the stock look of it. Good job bud.

Yes thank you. It does look way better than those ugly Quadros. 

 

My next project is a reference HD 6850 for the HWBOT cheap chips comp next month. I'll begin testing that one this week, but it arrived pretty dirty so I'll need to clean it up and repaste.

[neurotix]

9 minutes ago, damric said:

Yes thank you. It does look way better than those ugly Quadros. 

 

My next project is a reference HD 6850 for the HWBOT cheap chips comp next month. I'll begin testing that one this week, but it arrived pretty dirty so I'll need to clean it up and repaste.

I ran a 6870 back in the day and folded on it in the team comp for a year straight and got a badge for it.

 

Don't expect more than 1000MHz core clock (if that) and 1250MHz memory. Unless you put it under chilled water or something.

[damric]

Getting some good PPD after a bit of tuning:

 

[damric]

This card allows only very minimal tuning and follows a rather strict power envelope curve. But that said, I was able to somewhat tune clocks and power limits using overdriventool, and I was able to apply offset voltages using an old version of Sapphire Trixx.

 

I tuned a [email protected] profile for efficiency, and at 100W I am still pulling over 800K on project 18717. So 80% of the performance at half the power is good to me. I might be able to drop a few more millivolts, but that's going to need some long term testing.

 

I stuck a fan on the backside of the card over the VRM and socket area, so that might help some when I tune for HWBOT next week.