I\'ve now re-run my latest back-up, 07/09/2036, three times. Each time it crashes at 21:00hrs on 12/09/2036. But something strange is happening. When it first gets to 21:00hrs, it pauses some 10 seconds or so, then reverts to 00:30hrs on 07/09/2036, but the checkpoint counter shows 311, not 432. It counts down to 264, then, at 01:00hrs (ocean) on 07/09/2036, it jumps up from 264 to 383. This then counts down to 30, ie 21:00hrs on the 12th. It does exactly the same twice more before crashing with a 161 error. Could it possibly be that the back-up is in some way corrupted? Should I restart an earlier back-up, although these only go back three or four model months?

If it\'s always failing at the same point, that implies that it\'s an unrecoverable error, such as NEGATIVE PRESSURE or similar. These are much more likely when the model climate is extreme. We won\'t be able to see the cause from the website due to the earlier abort, but it might be in a log file somewhere on the disk (possibly in one of the slots directories).

I don\'t think it\'s worth going back to a really old backup, since it\'ll almost certainly stop at the same point again (given that it has done so several times already).

-- Edit:

Hi ya :-)

---

I'm a volunteer and my views are my own.
News and Announcements and FAQ

This behavour is typical of a model in trouble, with the program trying to recover it.

I\'ve seen this \"strange\" behavour of the counter myself. It\'s just the point at which something goes wrong. And it won\'t necessarily happen at EACTLY the same point each time around. After all, what is being modelled is chaotic at small time scales, and each time this is modelled, it may be slightly different.

Best to just let it crash and report this to the project. There are millions more possibilities to try.
Although whether or not any more info will get reported is hard to know, seeing as how it\'s already reported an error earlier on.

edit
Snap :)

I believe that I completed WU 6119163 on 8th Feb, but the Server State is In Progress and the Outcome Unknown. Do I need to kick something?

I believe that I completed WU 6119163 on 8th Feb, but the Server State is In Progress and the Outcome Unknown. Do I need to kick something?

Has the model disappeared from the tasks tab in BOINC Manager? The last set of trickles were submitted as a batch and it\'s just about possible that the model has rewound to the start of phase 3, even though the last trickle has been submitted.

Yes, it has disappeared from the tasks tab. The graphs of temperature and rainfall are complete. My m/c is definitely no longer processing this model.

I believe that I completed WU 6119163 on 8th Feb, but the Server State is In Progress and the Outcome Unknown. Do I need to kick something?

Some of the servers are still behind from the outage, so relax a couple of days, have a cold one (or 6, or ...), and check back.

---

The graph is so hot that I\'m posting it for all to see. This could be the hottest model ever!

Some models reach the same temperature much earlier. It looks like they always crash at 21 degrees:

---

Hi Alex

I see you\'ve posted the graph of one of your own models plus one run by Tomcat who\'s an experienced cruncher with a very good model completion record. The temperatures are as you say amazingly high. I wonder whether the model code contains a script that causes them to abort if the temperature reaches 21C. The exit messages are different for both these models but this could be caused by the fact that BOINC changed to more detailed error reporting.

Thanks for pointing these models out. I\'ll send Tomcat a private message so he sees what you\'ve noticed.

---

Cpdn news

Many models seem to have exceeded 21 degrees C without crashing (in this context). Any further information? If there is such a script as you suggest there might be, then is \"global warming\" limited by political, or other, decree? If so, then what a farce this all is!

The modelling is NOT about Real World Situations; it\'s about \"What If ...\".
And politics are NOT a factor in weather/climate modelling.

The modelling program does contain code that checks for valid conditions, such as the air pressure becoming negative, but I very much doubt that there is anything that limits a model to 21 degrees C.
It\'s more likely to be 100 degrees C, (boiling water), or some other VERY high number, because temperatures in the real world do get up to, and over, 45 degrees C. e.g. Marble Bar, in the far North West of Australia.

And what\'s to say that, internally, the tempreture DID suddenly rise to 100 degrees C, and then the model was aborted without this extreme temperature being reported? Only the project people can tell all of the details of individual failures, by looking at all of the information returned, AND knowing what it all means.

Hi MW. You said

Many models seem to have exceeded 21 degrees C without crashing (in this context). Any further information? If there is such a script as you suggest there might be, then is \"global warming\" limited by political, or other, decree? If so, then what a farce this all is!

No, I\'m sure there is no such script. I was really just \'thinking aloud\' about what Alex had said - he\'d commented that

It looks like they always crash at 21 degrees

but I think that this was just a coincidence in the particular crashed models he\'d looked at.

There\'s a page here where Nick Faull, one of the CPDN researchers (he\'s now left Oxford), explained some of the BBC Climate Change Experiment results:

http://www.bbc.co.uk/sn/climateexperiment/theresult/abouttheresults.shtml

You can see from the graph that some of the models, which were 160-year HADCMs identical to the ones we\'ve also crunched on CPDN, produced very large or very small temperature changes with increased CO2 but continued to the end - they didn\'t self-abort.

Part of the reason for processing ensembles (= large numbers) of models is to be able to predict the % probability that temperatures and other climatic phenomena will change by a given amount if emissions of CO2 continue to rise. So the average rise predicted by 2080 may be 3 - 3.5ºC ie this is very likely, but much smaller or much larger rises are possible though less probable. In the BBC experiment the less probable outliers weren\'t eliminated from the dataset - not even the small number of results that predicted a temperature fall.

Usually CPDN models begin with a control period or phase to test whether each model runs stably and sometimes to compare the results each model produces for past climate with real past measurements. You\'ll see that on the page I linked to, Nick Faull says \'Model runs which didn’t closely match the recorded temperature data for 1960 to 2005 were downweighted.\'. But these models still weren\'t eliminated from the dataset.

There are some sorts of instabilities in results that cause a small % of completed model results to fail CPDN quality control. As far as I know, one of the most frequent reasons for models failing QC is that the computer was unstably overclocked. This must produce numerical errors that lead to results so implausible that they have to be discounted.

If model calculations produce phenomena that are impossible in the real world eg negative atmospheric pressure, they do automatically self-abort. Sometimes this happens on very stable computers and restoring these models from a backup usually produces the same impossible phenomenon again. So we have to accept that certain combinations of initial parameter values are unviable.

I hope that makes the general situation clearer.

---

Cpdn news

Thank you, Les, for your response, with most of which I would have hoped to concur. I am just a little surprised that it has taken you so long to take issue with mo.v\'s message 32898.

Hi MW and Les

I think all three of us were posting at the same time just a moment ago.

I\'m also sure that the reason why Les and others didn\'t pick up on my previous post and refute what I\'d said is that they assumed I was just \'thinking/wondering aloud\'. Both Les and I and other moderators have said the same things (as in our most recent posts) on previous occasions.

I apologise for giving a mistaken impression in my earlier post.

---

Cpdn news

Yes, we often \'think out loud\' about things that occur to us, but usually in our private area. So I wasn\'t planning on \'taking issue\' here.

As for \"all the models that are crashing at 21 degrees\", perhaps the ones looked at by Alex were/are all from the same batch of \'parameter space\'.

And when he said: Some models reach the same temperature much earlier. It looks like they always crash at 21 degrees:, perhaps he meant \"they always crash at 21 degrees when rerun several times from a backup\".

I guess that one could say similar things about models that crash with \"negative pressure\", or \"negative theta\", if one spent time searching for large numbers of them.

Whatever, they all produce info for the project people to check, and for the dozens of research groups around the world, who are apparently very anxious to get their hands on the results.
Me, I only run the models to produce data, not to understand them.