Long running tasks

Message boards : Number crunching : Long running tasks

To post messages, you must log in.

AuthorMessage
Ruud van der Kroef

Send message
Joined: 29 May 22
Posts: 4
Credit: 64,793,443
RAC: 0
   
Message 1627 - Posted: 3 Nov 2022, 9:51:17 UTC

In the (recent) past I had tasks, that ran from 4 to 12 hours, also depending on the type of CPU.
Now I have a number of tasks that will run for a week or more. Is this new?
And will the amount of credit be adjusted accordingly?

Regards, Ruud
ID: 1627 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Allen Paschke

Send message
Joined: 27 Jan 18
Posts: 23
Credit: 13,556,172
RAC: 41,573
   
Message 1628 - Posted: 3 Nov 2022, 15:25:45 UTC - in response to Message 1627.  

I am having the same problem. CPU tasks typically take 6 - 10 hours to complete, using 3 threads

Starting this morning:
- I canceled one task that was less than 1 % complete after 2+ hours
- I canceled a second task that was 5% complete after 6 hours

Something has changed!!!!
ID: 1628 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sygopet

Send message
Joined: 28 Nov 17
Posts: 5
Credit: 34,481,761
RAC: 1,126
   
Message 1629 - Posted: 4 Nov 2022, 11:31:18 UTC
Last modified: 4 Nov 2022, 11:37:21 UTC

Same here. Elderly computer and normal running is about 10 hours.
One unit cancelled after 5% in 10 hours; current unit at 90% after 44 hours.
Ruud - don't expect more credits: the 6,836.19 (where on earth did such a figure come from?!) seems to be hard-wired in the software.
ID: 1629 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
bluestang

Send message
Joined: 26 Apr 18
Posts: 11
Credit: 2,185,284,992
RAC: 1,208
   
Message 1630 - Posted: 4 Nov 2022, 23:43:47 UTC

I've had some rogue very long GPU tasks as well so it's just not CPU ony.

And yeah...if it takes that much longer to tune then it should pay that much more as well!
ID: 1630 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sygopet

Send message
Joined: 28 Nov 17
Posts: 5
Credit: 34,481,761
RAC: 1,126
   
Message 1631 - Posted: 5 Nov 2022, 18:13:58 UTC - in response to Message 1629.  

Same here. Elderly computer and normal running is about 10 hours.
One unit cancelled after 5% in 10 hours; current unit at 90% after 44 hours.
Ruud - don't expect more credits: the 6,836.19 (where on earth did such a figure come from?!) seems to be hard-wired in the software.

This unit still "running" after 65 hours. In truth it hasn't moved an inch from yesterday. Big waste of computer time and will now be given the chop.
ID: 1631 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ruud van der Kroef

Send message
Joined: 29 May 22
Posts: 4
Credit: 64,793,443
RAC: 0
   
Message 1632 - Posted: 6 Nov 2022, 22:56:25 UTC - in response to Message 1627.  

What I was afraid of, happened today: one of the tasks - https://sech.me/boinc/Amicable/workunit.php?wuid=22422555 - exceeded its deadline and the task was given to another machine, which finished it in a flash.
That will probably mean, that I won't get any credit after I finished the task. It is running for 130h now, and will need another day.

I have a total of 7 tasks, which will run from 6 to 8 days, and will certainly finish after the deadline.
I will keep the above mentioned task running, and let it finish to see what happens.
The other tasks have been put on hold.
ID: 1632 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sygopet

Send message
Joined: 28 Nov 17
Posts: 5
Credit: 34,481,761
RAC: 1,126
   
Message 1633 - Posted: 8 Nov 2022, 10:41:05 UTC

The recent problem of long-running units appears to have ended after a week or so, with units now completing satisfactorily. My "average work done" figure has taken a hit but is gradually recovering.
ID: 1633 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ruud van der Kroef

Send message
Joined: 29 May 22
Posts: 4
Credit: 64,793,443
RAC: 0
   
Message 1634 - Posted: 8 Nov 2022, 14:01:10 UTC - in response to Message 1632.  
Last modified: 8 Nov 2022, 14:04:56 UTC

What I was afraid of, happened today: one of the tasks - https://sech.me/boinc/Amicable/workunit.php?wuid=22422555 - exceeded its deadline and the task was given to another machine, which finished it in a flash.
That will probably mean, that I won't get any credit after I finished the task. It is running for 130h now, and will need another day.

I have a total of 7 tasks, which will run from 6 to 8 days, and will certainly finish after the deadline.
I will keep the above mentioned task running, and let it finish to see what happens.
The other tasks have been put on hold.


The above mentioned task has finished, and has been marked: Completed, too late to validate.
OK, that was to be expected. Next I aborted all tasks that I had put on hold.
Earlier I mentioned 7, but it turned out to be 12 tasks.
Lost about 1000 hours of processing time; not very nice.

I looked at the aborted tasks, and the ones that have been processed by a wingman and granted credit, all have run on a GPU.
So, that does not seem to be a problem.

Things look to be good now, as Sygopet mentioned.
What we have to do now, is keep a close eye on these tasks, and watch progress and time remaining.
ID: 1634 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ruud van der Kroef

Send message
Joined: 29 May 22
Posts: 4
Credit: 64,793,443
RAC: 0
   
Message 1635 - Posted: 17 Nov 2022, 21:20:45 UTC
Last modified: 17 Nov 2022, 21:24:50 UTC

There really is something wrong with these tasks (at least if run on a CPU, they seem to run fine on a GPU).

Today I aborted a number of tasks that did not show much progress (<1%) after more than an hour.

Right now I am at my computer(s), and noticed one task starts, progress goes to 0.288%, and sits there, even after 13 minutes.
While I am watching this, I see a task on the same system, that runs for about 7 hours, and progress is at about 65%, suddenly fall back to 0.212% and keeps sitting there. Very weird.
Time to abort these too.
ID: 1635 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Corsair
Avatar

Send message
Joined: 17 Aug 22
Posts: 2
Credit: 68,614,875
RAC: 0
   
Message 1636 - Posted: 20 Nov 2022, 16:24:37 UTC

Hi all,

same to me running CPU tasks with 8 cores and in 1,5 days running only 14%, aborted
again, another CPU task with same 8 cores and don't know how many hours and only 2%, aborted.
today another unit with 8 cores 19 hours and only 16%

really don't know what to say because computer is an AMD Ryzen 7 3870X with 32 GB DDR4 RAM and SSD.
before I've been crunching units in less than 1.5 hours
Roses don't bloom on the sailor's grave

ID: 1636 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[AF>Le_Pommier] Jerome_C2005

Send message
Joined: 24 Apr 17
Posts: 23
Credit: 200,366,731
RAC: 2
   
Message 1701 - Posted: 27 Jul 2023, 18:13:44 UTC

Today I had a GPU task that ran for 12 hours, and at the moment I have another one almost running for 5 hours / 65% completion...

I have nothing else running on my mac, before this the tasks were taking between 30mn and 1h to complete.

Anyone else experiencing the same ?
ID: 1701 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Fuzzy Duck

Send message
Joined: 7 Mar 19
Posts: 14
Credit: 4,962,085,307
RAC: 3,975,485
    
Message 1702 - Posted: 29 Jul 2023, 9:16:52 UTC - in response to Message 1701.  

Yes. I run on two machines and have had tasks taking ten times longer than usual.
ID: 1702 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[AF>Le_Pommier] Jerome_C2005

Send message
Joined: 24 Apr 17
Posts: 23
Credit: 200,366,731
RAC: 2
   
Message 1703 - Posted: 29 Jul 2023, 11:20:39 UTC

I had another one than ran 7 hours and then it went down again, very variable though, maybe "globally longer" than before but I should compile data to confirm this feeling.
ID: 1703 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[AF>Le_Pommier] Jerome_C2005

Send message
Joined: 24 Apr 17
Posts: 23
Credit: 200,366,731
RAC: 2
   
Message 1705 - Posted: 30 Jul 2023, 11:29:10 UTC

Bloody hell I just found out I got another one that took 21 hours to complete, the other cruncher who got it took less than half the time to perform (but still it was 8 hours run).

Most of the tasks take from 40mn to 1h20 I would say, with some "peak" tasks around 2 hours from time to time, I only got 3 very long ones in the past week : 12h, 7h and now 21h.

Needless to say my RAC is falling down the black hole...
ID: 1705 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 1706 - Posted: 30 Jul 2023, 15:40:00 UTC - in response to Message 1705.  

I commented regarding this issue before, it's not the first time the project goes through slow search ranges. The positive side of this is we're finding a lot of new amicable pairs now (more than 4000 new pairs today).
ID: 1706 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[AF>Le_Pommier] Jerome_C2005

Send message
Joined: 24 Apr 17
Posts: 23
Credit: 200,366,731
RAC: 2
   
Message 1707 - Posted: 30 Jul 2023, 15:56:59 UTC

OK thanks for the feedback, happy to see it is not an issue and it is helping the project !

Hopefully then I do find some of these pairs, because as credit is concerned a 21 hours task awards the same as a 30mn task...
ID: 1707 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[AF>Le_Pommier] Jerome_C2005

Send message
Joined: 24 Apr 17
Posts: 23
Credit: 200,366,731
RAC: 2
   
Message 1709 - Posted: 31 Jul 2023, 11:35:27 UTC

Duration suddenly dropped from 10-20mn now.

We should call this project yoyo :D

(oops, already used ;) )
ID: 1709 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Corsair
Avatar

Send message
Joined: 17 Aug 22
Posts: 2
Credit: 68,614,875
RAC: 0
   
Message 1718 - Posted: 23 Aug 2023, 10:27:51 UTC - in response to Message 1709.  
Last modified: 23 Aug 2023, 10:28:13 UTC

same here,

I have one AMD running with six cores each unit and uses to last from two to three hours for each, tonight has been gone to more than six hours two units and got same credit that the "normal" ones.

now again running in the range of 2.5 hours.
Roses don't bloom on the sailor's grave

ID: 1718 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
yoshi838

Send message
Joined: 23 Feb 23
Posts: 1
Credit: 3,613,365,835
RAC: 14,583,686
    
Message 1719 - Posted: 27 Aug 2023, 1:12:01 UTC
Last modified: 27 Aug 2023, 1:12:18 UTC

Same observation. My task typically takes 70 minutes to complete. Starting yesterday, a task is at 10% after 1 hour, but the estimated time grown to 2d plus as the progress is only 0.001% every several seconds.

Running on intel 11th gen i5 with hyper threading. I just turned off HT to see if that makes any diff.
ID: 1719 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Long running tasks


©2024 Sergei Chernykh