etf rat's nest

[firedfly]

This is the team thread for rat's nest.

 

Current Team Roster

@firedfly (captain) - NVIDIA | Quadro P620

 

@pioneerisloud - AMD | RX 580

 

@Cerberus - Wildcard | GTX 980

Edited January 10, 2022 by firedfly

[BWG]

Go Team!

[Bastiaan_NL]

Welcome Rats!

[pio]

Woot woot!!!

Finally got my box to stay online AND folding leaving an RDP connection open to it off my server.  So should be smooth sailing from here.

[tictoc]

Does Windows RDP still switch the GPU driver and kill whatever is running?  Some other RDP solutions like TeamViewer and Chrome Remote Desktop should work fine on a Windows box.

[firedfly]

22 minutes ago, tictoc said:

Does Windows RDP still switch the GPU driver and kill whatever is running?  Some other RDP solutions like TeamViewer and Chrome Remote Desktop should work fine on a Windows box.

 

It does something so the current WU will finish but the next won't start after you disconnect RDP.  I found that as long as an RDP session is active, folding will work fine.  I can't reliably keep an RDP connection from my desktop to one of my Windows folding systems.  I've worked around this by using a third system that can keep a reliable connection.  The third system RDPs into the folding system.  I can then RDP into the third system and connect/disconnect as needed.  As long as the third system keeps the RDP session active, then folding will continue.  I believe this is the same solution that @pioneerisloud is using.

 

@BWG mentioned that using a dummy plug in one of the graphics ports might work around the issue as well (I don't have a monitor connected to the folding system).

[pio]

31 minutes ago, firedfly said:

 

It does something so the current WU will finish but the next won't start after you disconnect RDP.  I found that as long as an RDP session is active, folding will work fine.  I can't reliably keep an RDP connection from my desktop to one of my Windows folding systems.  I've worked around this by using a third system that can keep a reliable connection.  The third system RDPs into the folding system.  I can then RDP into the third system and connect/disconnect as needed.  As long as the third system keeps the RDP session active, then folding will continue.  I believe this is the same solution that @pioneerisloud is using.

 

@BWG mentioned that using a dummy plug in one of the graphics ports might work around the issue as well (I don't have a monitor connected to the folding system).

Correct it is.  I have a VM going on my rack server system (that was running already anyway).  I RDP into that.  That VM is already RDP'ed into my folding system, and I just leave that RDP session open 24/7.  Seems its been working fine since I set it up that way.  TY for the suggestion by the way.

[tictoc]

Since WSL and SSH are now a thing on Windows, you could just use that to monitor and control the folding client, if you don't want to mess around with RDP sessions.  Quick example of how I do that on Linux.  Would just need to use the right paths for Windows.

 

Monitor the log refreshing every 30 seconds:

$ watch -n30 "tail -n25 /opt/fah/log.txt"

 

Pause/resume/finish all the slots (make sure that FAHClient is in your path):

$ FAHClient --send-pause
$ FAHClient --send-unpause
$ FAHClient --send-finish

 

To do the same for individual slots, just append the slot number (00, 01, 02, etc) to the end of the command

$ FAHClient --send-pause/unpause/finish 01

 

I alias the above commands to easy to remeber things like pfold, rfold, fstat, etc.

[firedfly]

1 hour ago, tictoc said:

Since WSL and SSH are now a thing on Windows, you could just use that to monitor and control the folding client, if you don't want to mess around with RDP sessions.

 

Have you used this method to fix the hang after rdp disconnects?  I wouldn't expect this to help, but I could be wrong.

 

Either way, those commands are useful to have.

[tictoc]

No idea on the RPD hangs.  Just a slightly OT post of another option to control and monitor remote clients. 

[mmcdonald12]

Luke got me tuned up.  Should be about 12 hours per work unit now.  

 

[Bastiaan_NL]

My 2080Ti system is a headless system too. I run RDP from my main system to start everything and I can shutdown the main or whatever I want to do with it and the 2080Ti system will keep running without an issue. No dummy, only a power cord and a wifi antenna is needed. I might be lucky that it's working so good, cause usually it simply doesn't work. 

 

That sounds like almost double the performance @mmcdonald12, or did Luke pause the client while tuning the GPU?

[firedfly]

Quote

FEAR MY VEGA 8!!!

 

 

12 minutes ago, mmcdonald12 said:

Luke got me tuned up.  Should be about 12 hours per work unit now. 

Unless I'm missing something, that 12 hours that is shown is to complete the last 39.84% of the current work unit.  That would put the total time for this work unit at about 30.5 hours.  Still seems like an improvement over what you had before!

[damric]

3 minutes ago, Bastiaan_NL said:

That sounds like almost double the performance @mmcdonald12, or did Luke pause the client while tuning the GPU?

 

We had paused it while rough tuning, and I wish I would have paused it while fine-tuning. She lost about 6 hours of her progress after a 3rd crash when finding the iGPU frequency ceiling, but everything seems rock solid now.

 

This was a machine I built last year for the kids doing schoolwork at home and Michele saw the prizes, and the rules, and we had a discussion and she decided to try and do this. She was on a single stick of DDR4-2400 and stock 1250MHz iGPU frequency and I saw that she wouldn't even make the WU deadline.

 

Now she is running dual channel DDR4-3400CL16 and iGPU at 1725MHz. 1750 wasn't crashing but 1775 was, so that gives her a little bit of margin. RAM at 3466 was crashing. It's Micron E so maybe I can tighten up or add more voltage but I don't want her to have any more down time while I tweak that. It seems like she's getting almost 3-4x folding performance now. I think it doubled just from going dual channel.

 

I dialed back CPU core speed and voltage to 2600MHz 0.975v since she's only running on a wraith prism. Temps are good 45C. Her folding rig is sitting right next to mine.

 

I think it's awesome that she's on a team with @pioneerisloud since he was one of my mentors over a decade ago.

[Bastiaan_NL]

Really interesting @damric, and good to hear it seems rock solid. 
Fingers crossed that it'll remain stable!
And the best thing is to finish a unit before tweaking or changing anything. That might be a bit hard with a system that takes a long time to finish a unit, but if you don't want bad work units it's the best thing to do.

 

Edited August 3, 2021 by Bastiaan_NL

[pio]

Oh how the tides turn, you guys are all MY mentors now.   I feel like one of them old farts half the time, and I'm not even that old yet lol.

Glad to hear we've got everyone up and running solid now.  We may not have the strongest variety of equipment folding, but we're going to give it our alls dammit!  Is there an old fart emoji? 

I STILL haven't even looked at the prize list BWG was trying to persuade me in with.  I'm not in it to win it, I'm in it to help E and the site.  If we win prizes in the process, well, that'll be pretty cool.  New hardware is always welcomed around here.

[Avacado]

4 hours ago, mmcdonald12 said:

Luke got me tuned up.  Should be about 12 hours per work unit now.  

 

Mad, but also not mad at the same time about your notepad trash talk.

Edited August 3, 2021 by Avacado

[BWG]

You can leave the memory on the GPU stock or underclock it @damric

[damric]

33 minutes ago, BWG said:

You can leave the memory on the GPU stock or underclock it @damric

Yeah I know memory bandwidth isn't needed much for folding on graphics cards, but her iGPU actually seemed like it was severely bottlenecked by the single channel RAM at 2400MT/s and her performance kept climbing as I started bumping the DDR. Maybe because the infinity fabric connecting it all together, who knows.

[BWG]

1 minute ago, damric said:

Yeah I know memory bandwidth isn't needed much for folding on graphics cards, but her iGPU actually seemed like it was severely bottlenecked by the single channel RAM at 2400MT/s and her performance kept climbing as I started bumping the DDR. Maybe because the infinity fabric connecting it all together, who knows.

 

Oh I meant the GPU Memory. 

[damric]

1 minute ago, BWG said:

 

Oh I meant the GPU Memory. 

Yeah it's an APU, so integrated graphics. Probably the Infinity Fabric making the Ryzen 3200G direct traffic better to the iGPU side. 

[firedfly]

I'm trying to live up to the team name.  It will just get worse when my third card arrives later this week

 

[Bastiaan_NL]

Firedwood tech benches, cards not included...  

[mmcdonald12]

Hello all,  

 

I am currently having issues with my current GPU. It is having some trouble so you want to switch over to GT 730 in the morning as soon as damric gets a chance to configure it.

@BWG@firedfly

[BWG]

1 hour ago, mmcdonald12 said:

Hello all,  

 

I am currently having issues with my current GPU. It is having some trouble so you want to switch over to GT 730 in the morning as soon as damric gets a chance to configure it.

@BWG@firedfly

 

Go ahead, I can fix the points and multiplier in the morning. I just said the same thing to avacado.