Message boards : Number crunching : Unrecoverable error after 4100 hours :(
Message board moderation
Author | Message |
---|---|
Send message Joined: 12 Jan 05 Posts: 12 Credit: 40,824 RAC: 0 |
My work unit errored out after over 4100 hours and almost 80% complete. That\'s really too bad. What might have caused this? The machine was dedicated only to this project. No task switching, no new version installed, no Windows update. Just crunching happily and then the error :( See http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=5276332 BOINC.BE: For Belgians who love the smell of glowing red cpu's in the morning Tutta55's Lair |
Send message Joined: 12 Jan 05 Posts: 12 Credit: 40,824 RAC: 0 |
What are the chances of success if I restore a backup from that directory? It has trickled about an hour before the error, and the backup is at least a day old. So the same trickle would be sent more than once to the server. Any idea what would happen? BOINC.BE: For Belgians who love the smell of glowing red cpu's in the morning Tutta55's Lair |
Send message Joined: 9 Jan 07 Posts: 497 Credit: 342,899 RAC: 0 |
The server would just ignore the repeated trickle and would start acknowledging trickles again from the first new one. (Says one who has restored from backups often!) However, if it crashes again in the same place, you\'ve probably got a looper :-( and should abandon it. Visit the Scotland team |
Send message Joined: 12 Jan 05 Posts: 12 Credit: 40,824 RAC: 0 |
The server would just ignore the repeated trickle and would start acknowledging trickles again from the first new one. (Says one who has restored from backups often!) However, if it crashes again in the same place, you\'ve probably got a looper :-( and should abandon it. Thanks for reply. But it turns out this was a hypothetical question. I checked the backup. As this machine is no longer in production, it is also no longer included in our backup cycle. Pity. |
Send message Joined: 29 Sep 04 Posts: 2363 Credit: 14,611,758 RAC: 0 |
This error could have been caused by shutting down the computer without exiting from boinc first, or perhaps if you had a freeze-up and had to shut boinc down using Task Manager. It can sometimes mean your graphics card needs an update. To avoid future crashes I\'d have a look at the READMEs in my sig: *In the README about Running the model, the top tips *In the one about crashes, items #5 by Mike and #6 by Thyme Lawn *Same crashes README - item #1 by Les gives an easy manual backup method that you could use in future. There are extra methods in the README specifically about backups. One can\'t rely on automatic backups of one\'s whole hard drive or of a roomful of computers at one\'s place of work if they\'re carried out while the model\'s running. Backups made while the model\'s running can\'t be successfully restored. Congratulations on completing so much of the model. What you\'ve crunched will be used by the researchers. Cpdn news |
Send message Joined: 31 Oct 04 Posts: 336 Credit: 3,316,482 RAC: 0 |
Crash at 97.75% / 278 days :´( Tutta, you\'re not alone ;-) I still have one coupled model going at 97,27% now - no idea how many days, BOINC lost track when it moved from a dual Xeon to a dual P3s. If that one works, it will be my 1st HadCM3 full run - had a few HadSM3 and Sulphurs full runs but a coupled one is not in my collection yet. |
Send message Joined: 13 Jan 06 Posts: 1498 Credit: 15,613,038 RAC: 0 |
I\'ve got my fingers crossed for your model :-) 97% does count as \'completed\' from the scientists viewpoint since virtually all of the key data has already been uploaded by that point. So congratulations for that one too, although I understand completely why you\'d prefer to be at 100%! I'm a volunteer and my views are my own. News and Announcements and FAQ |
Send message Joined: 31 Oct 04 Posts: 336 Credit: 3,316,482 RAC: 0 |
It ignores duplicate trickles btw., they do not even cause trouble, if they come from a different machine than the one the model has been delivered to. Thanks for the head up :-) A full run at CPDN is something special, yes - and at least once I want to see the 100% |
Send message Joined: 31 Oct 04 Posts: 336 Credit: 3,316,482 RAC: 0 |
|
Send message Joined: 13 Jan 06 Posts: 1498 Credit: 15,613,038 RAC: 0 |
Congratulations :-) (And you won\'t even be busted by the cops) I'm a volunteer and my views are my own. News and Announcements and FAQ |
Send message Joined: 29 Sep 04 Posts: 2363 Credit: 14,611,758 RAC: 0 |
|
Send message Joined: 30 Apr 06 Posts: 8 Credit: 10,884,632 RAC: 1,681 |
I just had one end with a computation error when it was in the mid 90s percentile. My last backup was at 83%. It was reporting some kind of file I/O errors. I had to really struggle to get the last one to complete, many restores. This time it\'s to far back and I abandoned it. My last WU was on a network drive, but I had this one on a local drive. I have no idea what caused the errors this time around. This was the first error it had, made it all the way to the mid 90s with no problem, then BANG. I\'m abandoning the project for now, maybe forever. They need to work on this thing more to break the WUs up more, a single run takes far to long right now and there are too many possibilities for errors. Backing it up myself is not a viable solution, BOINC is not supposed to require user intervention. |
Send message Joined: 13 Jan 06 Posts: 1498 Credit: 15,613,038 RAC: 0 |
Given that the model uploads it\'s climate as it goes (summary at each year, more detailed summary at the model decades, and a full restart dump at 1960, 2000, and 2040), it doesn\'t really matter a huge amount if it doesn\'t reach it\'s end. Not only that, but your model reached 2071 - which is as good as done as far as the scientists are concerned. It will already have been added to the count of completed models on the front page (successfully completed = 2050 as far as the first page is concerned). And finally - the reason it was stopped was that it was aborted 10 minutes before you posted. http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6206053 <core_client_version>5.4.11</core_client_version> <message> aborted by user </message> <stderr_txt> I'm a volunteer and my views are my own. News and Announcements and FAQ |
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
And if (or when) you do come back, remember to ask about the automatic, set-and-forget, backup system. |
Send message Joined: 6 Oct 06 Posts: 204 Credit: 7,608,986 RAC: 0 |
:-) Les! not fair :-) if there is such a method for back up :-( at least let us know ;-) i at least am not leaving but on such Long WU\'s :-) i do tend to get night mares ;-) LoL Regards Masud. |
Send message Joined: 13 Jan 06 Posts: 1498 Credit: 15,613,038 RAC: 0 |
There\'s a backup-and-restore page here, which discusses links to loads of different ways of doing backups. I tend to stick to the \'old school\' way of shutting down boinc, copying the boinc folder, and restarting it, at weekly intervals, but a lot of people use more sophisticated methods! The thread is here: http://www.climateprediction.net/board/viewtopic.php?t=5895 The automatic backup programme is here: http://bbc.cpdn.org/forum_thread.php?id=2748 It was written by RRodway, a participant originally on the BBC/CCE variant of the project (which is why you\'ll see \'BBC\' all over it). Sometimes a backup just won\'t help - for example, if a model crashes with a \'NEGATIVE PRESSURE DETECTED\' error, it\'ll almost always crash at the same spot again if you restore it. But in this case, it means that the model has reached as far as it\'ll ever go. I'm a volunteer and my views are my own. News and Announcements and FAQ |
Send message Joined: 30 Apr 06 Posts: 8 Credit: 10,884,632 RAC: 1,681 |
I restored from backup and then aborted that, probably not the best idea but I already deleted the one that had the computation failure. I don\'t think it made much difference though, but it may affect stats concerning computation errors vs user aborted. |
Send message Joined: 30 Sep 06 Posts: 18 Credit: 93,623 RAC: 0 |
Maybe it was corruption caused by Cosmic Rays. What is your elevation there? |
Send message Joined: 31 Oct 04 Posts: 336 Credit: 3,316,482 RAC: 0 |
A team mate just posted (in the team forum), that his model crashed after shutting down / restarting his computer. When I checked the result, I found \"aborted by user,, CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Abort request from BOINC...\" CC is 5.8.8 Is there some known issue with CPDN and shutdowns or is it cosmic rays again? resultid=6513196 He might not have stopped BOINC before he shut down but it would be good if a model survived that. |
Send message Joined: 29 Sep 04 Posts: 2363 Credit: 14,611,758 RAC: 0 |
Hi Ananas Yes, there\'s a big issue with shutting down the computer. Your friend needs to exit from boinc first by right-clicking on the B icon and selecting Exit. Then wait for the icon to disappear, then begin the shutdown process. Sometimes a model survives if boinc isn\'t exited, sometimes it doesn\'t. However, the model\'s results page doesn\'t show that. It shows \'aborted by user\': http://bbc.cpdn.org/show_user.php?userid=156404 All your friend\'s models are quickly crashing. Eg I think he must have run an AV scan without exiting from boinc. The scans locked a file and crashed these: http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6497343 http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6492724 http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6465859 and maybe this one though it crashed with a 107 code which is different from the others http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6458633 and he aborted his first model: http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=6447344 His computer looks good - dual-core, lots of memory. But I don\'t think he has any idea how to keep his models safe. Please explain to him that he mustn\'t abort his models or he\'ll never finish one. Would it be possible for you to translate the following items into German for your team members and put them in a sticky on your forum? Go to the project READMEs through my signature. *Running the model README - the top tips *README about avoiding crashes - you\'d need to make a summary of item #5 by Mike *Crashes README again - item #1 by Les about how to back models up. That\'s the essential info that cpdn crunchers need. If you did translate that, it would be a good idea if you could post the German translation here so that Saenger, Tomcat and the many other German team members can copy it to their forums for people whose English isn\'t so good. I\'m sure they would do this. There are several German forums with cpdn crunchers. If you can do that, it would be better to post a translation & discuss it in a separate thread which could all be in German. With a thread title in German to attract the people who need to read it. Cpdn news |
©2024 cpdn.org