Radeon Instinct MI25 16GB HBM2

[damric]

 

 

There's a couple guys selling a whole bunch of these old server cards on ebay. They are listed as buy-it-now for like $70-80, but one guy took my offer of $60 shipped. Not too bad since I sold an actual Vega 64 for $200 a couple months back. These server cards only have Linux drivers, but can be used with Windows if you flash it to WX9100, which also gives you output on the mini-display port. Thanks to a few guys at techpowerup for this idea.

 

 

These cards have no fans, meant for high rpm server chassis fans to blow through. I saw some 3D printed adapters, but I happen to have a cheap universal water block I'm mounting. Going to need a little bit of metal cutting to get it to fit!

 

Had to cut out that tab on the base plate. Also that heat pipe was just laying there, not soldered or anything so I coated the backside with MX-4.

Cheap ebay GPU block fit the Vega die exactly. Looks like the tubes will have enough room.

I wanted to keep the base plate because it keeps the VRMs cooled. I reused the stock thermal pads since they were in good condition. Base plate also connects the back plate.

Looks good enough like this but I wonder if the shroud cover will also fit back on?

Yes just had to trim the aluminum. I need to sand down the edges and put some black polish on there.

So this is how it will look. Going to get it running this week on one of my folding benches.

[SamsTechStuff]

It's a bummer about the non existent mxgpu driver for homelabbers but very cool to see that these can used for F@H. How does it perform?

[damric]

30 minutes ago, SamsTechStuff said:

It's a bummer about the non existent mxgpu driver for homelabbers but very cool to see that these can used for F@H. How does it perform?

In theory it will be the same as Vega 64, so 1-2 million per day +/- how good the WUs are.

[damric]

Filled the loop back up and letting the pump run overnight to check leaks and collect air bubbles.

That's a small cut I had to make on the shroud for the tubes

I like the look of the shroud, but if VRM thermals are bad then it's coming off and I'll strap a fan onto the base plate.

It's only a 240mm cheap aluminum radiator. I imagine it will be fine, but we'll find out tomorrow when I run it hard.

 

@neurotix wasn't joking when he said these used to be extremely expensive. I just looked it up and the MSRP was over $16,000. What was Raja thinking? I got it for $60!

Edited May 14, 2023 by damric

[neurotix]

Thanks for the pics and wow, I knew they were a few thousand dollars but didn't realize they were originally $16k

 

Reminds me of my used Nissan 300zx I used to have and that it was $32k in 1990. Was cool to own something that was originally that expensive.

 

Best of luck on your watercooling exploits, may they be leak free. It'll be interesting to see how much ppd that thing puts out.

[damric]

30 minutes ago, neurotix said:

Thanks for the pics and wow, I knew they were a few thousand dollars but didn't realize they were originally $16k

 

Reminds me of my used Nissan 300zx I used to have and that it was $32k in 1990. Was cool to own something that was originally that expensive.

 

Best of luck on your watercooling exploits, may they be leak free. It'll be interesting to see how much ppd that thing puts out.

I spent all morning messing around with it. Since it has no Windows drivers, only Linux, and the one mini-display output is disabled by default, I had two options:

 

1) Flash it to WX9100. This allows Windows drivers and enables the mini-dp. But there's no clock tuning in the driver like there was for Vega 64 or 56.

OR

2) Leave the firmware as MI25 and install a linux distro. Yeah I'm no good at that, and this would also mean no benchmarking, but at least it would fold at stock config.

 

I went with option #1. So now it's folding away happily and says 660K PPD on project 18717, but that's at the default clock/power config which seems to be running around 1200-1250MHz.

 

I know it can do more. I've got to figure out how to unlock clock adjustments. I think I might be able to flash the VEGA 64 FE (16GB) BIOS to it and be able to tune it, but I'll probably lose the mini-dp. Not a big deal since I have it installed with 5600G with integrated graphics, but I don't think that will allow me to run benchmarks on it. I'll have to play around and see.

 

BTW the coolant temp is fine with just the weak 240mm aluminum radiator. 34C while folding. GPU temp 44C, hot spot 70C.

Edited May 15, 2023 by damric

[neurotix]

19 minutes ago, damric said:

I spent all morning messing around with it. Since it has no Windows drivers, only Linux, and the one mini-display output is disabled by default, I had two options:

 

1) Flash it to WX9100. This allows Windows drivers and enables the mini-dp. But there's no clock tuning in the driver like there was for Vega 64 or 56.

OR

2) Leave the firmware as MI25 and install a linux distro. Yeah I'm no good at that, and this would also mean no benchmarking, but at least it would fold at stock config.

 

I went with option #1. So now it's folding away happily and says 660K PPD on project 18717, but that's at the default clock/power config which seems to be running around 1200-1250MHz.

 

I know it can do more. I've got to figure out how to unlock clock adjustments. I think I might be able to flash the VEGA 64 FE (16GB) BIOS to it and be able to tune it, but I'll probably lose the mini-dp. Not a big deal since I have it installed with 5600G with integrated graphics, but I don't think that will allow me to run benchmarks on it. I'll have to play around and see.

 

BTW the coolant temp is fine with just the weak 240mm aluminum radiator. 34C while folding. GPU temp 44C, hot spot 70C.

 

Linux won't help you, don't bother. There are drivers for it but AMD GPU support in Linux sucks. Afaik they don't even have a proper control panel, and overclocking is not available.

 

On the Nvidia side, with the official Nvidia drivers and an app called GreenWithEnvy, you get both a proper control panel and a very slick overclocking utility that can do a lot, unfortunately it lacks voltage control though.

[damric]

Well I discovered a nifty program called overdriventool, and it's allowing me to tweak clocks, power, and voltage, so it's all good now.

[neurotix]

Never heard of it, but congrats!

[neurotix]

Also, what are the stock clocks and what is your OC now? Does the HBM oc at all?

 

I never had Vega but I did have Crossfire Sapphire R9 Fury Nitro for a while. The HBM wouldn't budge and I think they didn't OC past 1150MHz. You probably remember the advertisement and slides for the Fury X with Koduri saying "they're an overclockers dream" before they came out, lol. They were great cards though and ran every game I had at Ultra except for Rise of the Tomb Raider because 4K textures took more than 4GB ram and the Fury's only had 4GB HBM.

 

Vega can clock much higher than my old Fury's, like at least 1800MHz or so right?

 

Let me know the stock clocks and what you've gotten the thing to now.

[damric]

55 minutes ago, neurotix said:

Also, what are the stock clocks and what is your OC now? Does the HBM oc at all?

 

I never had Vega but I did have Crossfire Sapphire R9 Fury Nitro for a while. The HBM wouldn't budge and I think they didn't OC past 1150MHz. You probably remember the advertisement and slides for the Fury X with Koduri saying "they're an overclockers dream" before they came out, lol. They were great cards though and ran every game I had at Ultra except for Rise of the Tomb Raider because 4K textures took more than 4GB ram and the Fury's only had 4GB HBM.

 

Vega can clock much higher than my old Fury's, like at least 1800MHz or so right?

 

Let me know the stock clocks and what you've gotten the thing to now.

Stock clock max boost is 1500MHz. When folding or doing anything serious it hits power limit and is more like 1200-1300MHz. I can exceed power limits as high as I want with registry tweaks but then the card just eats power and runs hot as hell to boost higher. Yeah I had a Vega 64 running steady at 1800MHz but with extremely cold water like 15C and a 450W power draw

 

It's got a high ASIC quality 94% so I'm going to experiment undervolting and see where that goes. I probably need to add active cooling to the back of the card too sice there's some bare VRMs there.

[neurotix]

2 minutes ago, damric said:

Stock clock max boost is 1500MHz. When folding or doing anything serious it hits power limit and is more like 1200-1300MHz. I can exceed power limits as high as I want with registry tweaks but then the card just eats power and runs hot as hell to boost higher. Yeah I had a Vega 64 running steady at 1800MHz but with extremely cold water like 15C and a 450W power draw

 

It's got a high ASIC quality 94% so I'm going to experiment undervolting and see where that goes. I probably need to add active cooling to the back of the card too sice there's some bare VRMs there.

Good luck. Yeah, not worth it to have that high of power draw for just like 50k more PPD.

 

Lol @ the Vega 64 450w power draw. I have my RTX 4090 oc'ed now and putting up about 22m-24m points daily and in my OC tool it shows wattage and the highest it gets is around 350w. Using the stock "gaming" bios this time, so a 450w limit but it never comes close to it.

 

I wonder, with the upgrades to your main rig and the full watercooling on it, as well as your test benches with weird pro cards and cheap watercooling components, if all that money you spent could have gotten you a 4090 instead by now that would dwarf everything you have running currently and at far less power consumption? Devil's Advocate: get a 4090

[damric]

1 minute ago, neurotix said:

Good luck. Yeah, not worth it to have that high of power draw for just like 50k more PPD.

 

Lol @ the Vega 64 450w power draw. I have my RTX 4090 oc'ed now and putting up about 22m-24m points daily and in my OC tool it shows wattage and the highest it gets is around 350w. Using the stock "gaming" bios this time, so a 450w limit but it never comes close to it.

 

I wonder, with the upgrades to your main rig and the full watercooling on it, as well as your test benches with weird pro cards and cheap watercooling components, if all that money you spent could have gotten you a 4090 instead by now that would dwarf everything you have running currently and at far less power consumption? Devil's Advocate: get a 4090

What fun is that?

[neurotix]

17 minutes ago, damric said:

What fun is that?

True enough. I'm glad you enjoy your ghetto watercooling and benches and since I can't do it myself (but want to) I am honestly envious of your ability. You did a really great job modding this card the way you did, especially since it keeps the stock look of it. Good job bud.

[damric]

7 minutes ago, neurotix said:

True enough. I'm glad you enjoy your ghetto watercooling and benches and since I can't do it myself (but want to) I am honestly envious of your ability. You did a really great job modding this card the way you did, especially since it keeps the stock look of it. Good job bud.

Yes thank you. It does look way better than those ugly Quadros. 

 

My next project is a reference HD 6850 for the HWBOT cheap chips comp next month. I'll begin testing that one this week, but it arrived pretty dirty so I'll need to clean it up and repaste.

[neurotix]

9 minutes ago, damric said:

Yes thank you. It does look way better than those ugly Quadros. 

 

My next project is a reference HD 6850 for the HWBOT cheap chips comp next month. I'll begin testing that one this week, but it arrived pretty dirty so I'll need to clean it up and repaste.

I ran a 6870 back in the day and folded on it in the team comp for a year straight and got a badge for it.

 

Don't expect more than 1000MHz core clock (if that) and 1250MHz memory. Unless you put it under chilled water or something.

[damric]

Getting some good PPD after a bit of tuning:

 

[damric]

This card allows only very minimal tuning and follows a rather strict power envelope curve. But that said, I was able to somewhat tune clocks and power limits using overdriventool, and I was able to apply offset voltages using an old version of Sapphire Trixx.

 

I tuned a folding@home profile for efficiency, and at 100W I am still pulling over 800K on project 18717. So 80% of the performance at half the power is good to me. I might be able to drop a few more millivolts, but that's going to need some long term testing.

 

I stuck a fan on the backside of the card over the VRM and socket area, so that might help some when I tune for HWBOT next week.

[Lifted]

Good, Sir! 
You are the first man that had anything more or less conventional for cooling!

 

1. Can you please tell me what is the actual spacing for this block/screws (in inches if possible or mm)?
I have some cooling for gpus such as Accelero Xtreme IV (by Arctic), but it doesn't fit every gpu, in fact 3070 was already past their support, but MI25 is older, so I am keeping my fingers crossed. Cooler has adjustable screw holes, so I hope it can accommodate. Smallest one is 45mm x 45mm spacing:

Accelero Xtreme IV | High-End VGA Cooler with Backside Cooler | ARCTIC

WWW.ARCTIC.DE

---

High-efficiency cooling with a patented back side cooler which keeps the board including all sensitive components around the GPU on low temperature.

2. Did you flash the BIOS from Linux, or did you use the clip programmer (by clamping the bios chips)?

I want this GPU, for the same exact reason that you mentioned. I am building a workstation PC, and refuse to buy gaming GPU, when I have RTX 4090 in a separate machine. Would be nice to have a strong GPU such as this for older games, and for rendering what I need to render. 

Thank you!

[damric]

I'll take a look at it today and measure. Another option is 3D printed fan adapter for the stock cooler.

 

I am terrible at linux so I flashed it in Windows 10. I used a system that had video out through a 5600G APU graphics.

[Lifted]

8 hours ago, damric said:

I'll take a look at it today and measure. Another option is 3D printed fan adapter for the stock cooler.

 

I am terrible at linux so I flashed it in Windows 10. I used a system that had video out through a 5600G APU graphics.

Please do

This is REALLY good news, because I have 2 more GPUs I can easily use for those purposes. I did flash like that before, and also was able to revive a bad bios flash from 2nd gpu. 

Do you think it is wise to flash Vega 64 BIOS on it?
 

Forgot to mention: I did see that fan shroud mod, but it will be blowing air like crazy, pc will be sitting next to me, so that's scenario is not the best. I did consider it though, before searching forums and found you   

 

Do you think gentle blowing air over VRMs through GPU fans of Accelero Extreme IV is enough to keep VRM cool enough? Heatsink is no doubt enough.

I just hope the contact is good if I keep original VRM heatsink on MI25, but just add my core cooler. Similar how you added waterblock over it. It gives me hope! LOL

I used to cool my overclocked RTX 3070 with it, and it was hitting 65 C max. While fans barely spinning. Backside of GPU may not be cooled by Accelero's beastly aluminum heatsink, but I can use stock backplate of the MI25 Instinct.

 

Did you have any oddities with this GPU after flashing? 
I would like to have a decent gaming card on top of workstation build. 
If you played any games, was the card solid after that BIOS mode?

I know this was not your intended use when you have built the system, but worth asking.

I looked into running NVIDIA P40 for those purposes as well, and it seems like P40 is comparable on "paper", but what do you think is better for rendering and gaming at the same time?

 

I will still need to buy some parts, such as 6 pin to 8 pin connector

I find these posts REALLY cool, because it gives life to components that were never meant to be what they are. 
 

[LabRat]

11 hours ago, Lifted said:

1. Can you please tell me what is the actual spacing for this block/screws (in inches if possible or mm)?

 

Fury, Vega, and VII's GPU+HBM Heatsink mounting pitch is 64mm x 64mm.
s far as I can research, that 64x64mm square mount is 'unique' to AMD's HBM-GCN/NCU cards (not sure about BC-160/V520/V540 HBM-Navi, though).

[Lifted]

6 minutes ago, LabRat said:

 

Fury, Vega, and VII's GPU+HBM Heatsink mounting pitch is 64mm x 64mm.
s far as I can research, that 64x64mm square mount is 'unique' to AMD's HBM-GCN/NCU cards (not sure about BC-160/V520/V540 HBM-Navi, though).

Very nice! Thank you 

If Damric can please confirm this, before I pull a purchase (since he has it in physical possession). 

That way we can establish it for everyone in Google search who is on the quest like me  

[LabRat]

6 minutes ago, Lifted said:

Very nice! Thank you 

If Damric can please confirm this, before I pull a purchase (since he has it in physical possession). 

That way we can establish it for everyone in Google search who is on the quest like me  

Good idea.

I did manage to non-physically verify that is the mount via both Aliexpress listings for coolers, and old Reddit Posts.

I too, have been on a quest to try and find an affordable air cooler that'll fit my MI25 -> WX9100.

I've almost bought 3 'for parts' Vega/Fury 'partner' cards that came w/ better coolers. Sadly, many completely delete the frontplate and have too big of a coldplate to just mount-on unmodified.
I want to keep the frontplate for VRM and board-component cooling, and I don't want to have to dismount it (potentially, ruining thermal pads).

 

So far, the 'best' option is just 'eating it' and buying a Morpheus II Core off Amazon. Unfortunately, that costs more than the card did...

Note: The Morpheus series will not mount onto Vega 56/64 cards while the frontplate is still installed (at least, not without A LOT of cutting). However! MI25's frontplate is MUCH SHORTER than Vega56/64's. At most, I see a teensy bit of material towards the Bracket needing filed-away. (which, could be CAREFULLY done w/o dismounting the plate)

Edited August 27, 2023 by LabRat

[Lifted]

7 minutes ago, LabRat said:

Good idea.

I did manage to non-physically verify that is the mount via both Aliexpress listings for coolers, and old Reddit Posts.

I too, have been on a quest to try and find an affordable air cooler that'll fit my MI25 -> WX9100.

I've almost bought 3 'for parts' Vega/Fury 'partner' cards that came w/ better coolers. Sadly, many completely delete the frontplate and have too big of a coldplate to just mount-on unmodified.
I want to keep the frontplate for VRM and board-component cooling, and I don't want to have to dismount it (potentially, ruining thermal pads).

 

So far, the 'best' option is just 'eating it' and buying a Morpheus II Core off Amazon. Unfortunately, that costs more than the card did...

So, you are on the quest too! HAHA!  
Awesome

I just confirmed by checking that Accelero cooler is 60mm at the largest hole position from screw to screw. So, it is not compatible with Vega 64, if MI25 matches the hole spacing it will be incompatible too. After we confirm ;- )

You are correct on the fact that Morpheus II Core does fit Vega 64, and indeed it is more expensive than MI25, if you can find one. Since they are discontinued, as far as I know.

I remember looking at it years ago when I was looking to cool some parts.

Edited August 27, 2023 by Lifted