Continuing the thread from the php board.

Here are another two runs with the same parameters but very different results.

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=438879&amp;phase=AT#graph">here</a> and <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=438878&amp;phase=AT#graph">here</a>

Again it seems that there is a cold equator on the second run, though the climate sensitivity seems to be similar.

---

____________________________<br>
<a href="http://www.boincforum.info/boinc/">boinc forum</a> and <a href="http://www.uk4cp.co.uk/">United Kindom</a> team, my climate change <a href="http://www.livejournal.com/users/mike_atkinson/">blog</a>.

Unusual to see a cold equator where critical relative humidity gets as low as 0.6. However both seem to have been produced by a reliable computers.

I find this more worrying than other pairs seen.

I am not sure how you calculate climate sensitivity to be similar. Isn't it the difference between the equilibrium level that would be reached if phases 2 and 3 were continued?

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

&gt; Unusual to see a cold equator where critical relative humidity gets as low as
&gt; 0.6. However both seem to have been produced by a reliable computers.
&gt;
&gt; I find this more worrying than other pairs seen.
&gt;
&gt; I am not sure how you calculate climate sensitivity to be similar. Isn't it
&gt; the difference between the equilibrium level that would be reached if phases 2
&gt; and 3 were continued?
&gt;

Looking at the graphs again I find I was wrong. I thought that the second run had a phase 3 that showed signs of a cold equator but at about 3 degrees above the phase 2 value. 3 degrees is the climate sensitivity of the first run and so they could be said to be similar. However, phase 3 of the second run seems to be stable and reach equilibrium after about a 1.5 degrees rise in temperature.

I agree it looks worrying. If the reasons for this are not contained in the papers about to be published then perhaps a technical note could be placed on this site explaining what is going on. There are several unexplained things happening with the experiment:

1. the same workunit on different computers seems to be giving different weather. This is true even if exactly the same download is used as per the threads on the php board.

2. the same workunit may give both stable and unstable phase 2. Probably caused by some weather conditions in the unstable case leading to cold equators, but this would be nice to be confirmed.

3. there are hints that the runs in this thread have different climate sensitivities for the same workunit (though this is difficult to verify as the second run has an unstable phase 2). Is this the case and if so what is the cause?

4. Are all unstable runs grouped in the parameter space, or are some scattered thoughout the parameter space? Is there a difference between those that are always unstable and those that are just sometimes unstable?

5. climate sensitivity seems to be usually measured as the difference between the average of the control (phase 2) and the 2xCO2 (phase 3), averaged over 10 years after equilibrium has been reached. Many of the phase 3 temperatures are still increasing, and most only reach equilibrium a few years before the end of phase 3, so how is climate sensitivity going to be measured? A result from experiment 1 + experiment 2 could be a pdf of the climate sensitivity for HadSM3 (weighted by the hindcast of experiment 2 using HadCM3), but the extremes (which cpdn was designed to cover) won't be represented if they are not allowed to reach equilibrium.

6. Given the same workunit giving different weather on different PCs is there any need for an initial condition ensemble?

---

____________________________<br>
<a href="http://www.boincforum.info/boinc/">boinc forum</a> and <a href="http://www.uk4cp.co.uk/">United Kindom</a> team, my climate change <a href="http://www.livejournal.com/users/mike_atkinson/">blog</a>.

Two flavors from the same Model, both run on P4's but with different versions of Windoze. They're similar but far from identical.

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=212401

(HostID #4741 is mine.)

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

(Accidental duplication, sorry.)

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

&gt; Two flavors from the same Model, both run on P4's but with different versions
&gt; of Windoze. They're similar but far from identical.
&gt;
&gt; http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=212401
&gt;
&gt; (HostID #4741 is mine.)

This is an example of what I meant by:

&gt; 1. the same workunit on different computers seems to be giving different
&gt; weather. This is true even if exactly the same download is used as per the
&gt; threads on the php board.

Some of the experiments on the php board showed this occuring for exactly the same model download with the weather being significantly different after a week or two.

---

____________________________<br>
<a href="http://www.boincforum.info/boinc/">boinc forum</a> and <a href="http://www.uk4cp.co.uk/">United Kindom</a> team, my climate change <a href="http://www.livejournal.com/users/mike_atkinson/">blog</a>.

I certainly agree it would be nice to have some explanation of those 6 items Mike mentioned earlier. I expect we may have to wait longer for some. DaveF has said 1 is due to different math libraries being used to solve the same equations. Presumably a different technique to estimate a solution to an equation could lead to a different least significant digit. This will lead to chaotically different results - the butterfly effect in operation.

Re 4. I believe they are quite well grouped.

Re 5. They use the last 8 years for diagnostics. An exponential extrapolation can be used to estimate the equilibrium though I think I heard at the open day a suggestion that such an extrapolation was tending to overestimate the climate sensitivity (though that may have been underestimate).

Re 6. I expect we do need the different ic sets. Maybe with some more work to ensure that results from different computers produce the same sort of differences as initial conditions, the team would then be in a position to use different computers in the same way as an ic ensemble. I doubt they could have taken the risk of using different computer to provided an ic ensemble from the start.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

&gt; Two flavors from the same Model, both run on P4's but with different versions
&gt; of Windoze. They're similar but far from identical.
&gt;
Has someone tried to crunch exactly the same model on the same computer (same hardware and same Windows version).
Of course everybody would expect the same result, but I wouldn't be surprised to see differences if the model is chaotic.

---

Arnaud

&gt; Has someone tried to crunch exactly the same model on the same computer (same
&gt; hardware and same Windows version).

I've tried to crunch a model the same timesteps several times.
(To be exactly: I stopped cpdn, made a backup, started cpdn, after 144 steps the model was stopped again, saved the restart.day file, backup restored, cpdn run again for 144 steps then stopped, compare restart.day with the saved restart.day file)

The restart.day file was always different!
I don't exactly know what is saved in this file, I thought it would be the field variables of the model. If this was right, this would mean, that there must be some random numbers thrown into the calculations.

---

Interesting Josti

Did you do a file compare and was there any indication of how much and what was different. Just a possibility that something like the date &amp; time completed is stored in there.

I have just had a model that went back to the start on me. I copied the pe12 file and let the model continue. Doing a file compare found some differences. However using CPView I dumped the temperatures for a season to a CSV file from each. 7008 temperatures each recorded to 4 decimal places were identical.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

Hi

Not bad final temperature agreement (at the final display resolution) between these 3 results of WU235513 - 2mo6_100143935. All from different CPU's and O/S's:


Host 70466 - P4 2GHz, Win2k (Pro) OS

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=438997&amp;phase=AT#graph


Host 78240 - Zeon 2.8GHz, Linux 2.4.21-1SCORE_xeon OS

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=438998&amp;phase=AT#graph

Pity this one had a server error in collection of some of the data points in Phase 3.


Host 3708 (my PC, Amy) P4 3.2GHz(H/T), WinXP (Home) O/S

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=438999&amp;phase=AT#graph

Once the php boards are back up, I will post further results from my 'unofficial' run looking at the behaviour of exactly the same downloaded model run on two different PC's. As I thought might happen, up to 50% of Phase 2, there has been much more deviation between the two machines in Phase 2 than there was in Phase 1.

Pete

Does anyone know when different maths libraries may be used. eg Could a different stepping of the same chip at the same speed use a different maths library? And/or Could 2 chips of the same stepping but run at different speed use a different maths library?

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

&gt; Did you do a file compare and was there any indication of how much and what
&gt; was different. Just a possibility that something like the date &amp; time
&gt; completed is stored in there.

Oh, you are right. There must be something stored like cputime or date or something.
I've run a model twice for 40 modeldays (on the same system) with the result that the temperature fields were identical.
So I conclude that there are *no* random numbers involved. Otherwise the weather in a model would be significantly different after some weeks, due to the nonlinearity of the equations.
I hope this conclusion is more correct than my previous one :-)

---

I know from my fortran programing, that results can be different, when different compiler options are used. This difference comes from different internal precision of the calculations in the cpu, if I remember right. Usually only the last decimal place is different.
But in a weather model this would lead to different weather after a few weeks.
[speculation]
I think the cpdn model for linux is more optimized than the one for windows, which could explain the difference when the same model is run on different OS.
[/speculation]

But where do the differences within the same OS come from?
From machine errors??? Don't really know.

EDIT:
I found two models with the same parameters which got exactly the same result (one on a athlon64, the other on a athlon, both under windows):

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=421090&amp;phase=AT#graph">Link1</a>
<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=421092&amp;phase=AT#graph">Link2</a>

EDIT 2:
Sorry, I've posted the wrong resultids

---

Stumbled over the same thing a few days ago :

Same Model, but definitely notable differences :
(my) AthlonXP 3000+, Win2000 SP4 :
<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=399030&amp;phase=AT#graph">http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=399030&amp;phase=AT#graph</a>
(other) Pentium4 3000MHz, WinXP SP2 :
<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=399028&amp;phase=AT#graph">http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?field=Temperature&amp;resultid=399028&amp;phase=AT#graph</a>

---

Scientific Network : 44800 MHz - 77824 MB - 1970 GB

&gt; Re 5. They use the last 8 years for diagnostics. An exponential extrapolation
&gt; can be used to estimate the equilibrium though I think I heard at the open day
&gt; a suggestion that such an extrapolation was tending to overestimate the
&gt; climate sensitivity (though that may have been underestimate).
&gt;

I think I must have been mistaken or at least misinterpreted the size/importance of the error. The results paper says the error is small.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

&gt;
&gt; 2. the same workunit may give both stable and unstable phase 2. Probably
&gt; caused by some weather conditions in the unstable case leading to cold
&gt; equators, but this would be nice to be confirmed.
&gt;
&gt; 3. there are hints that the runs in this thread have different climate
&gt; sensitivities for the same workunit (though this is difficult to verify as the
&gt; second run has an unstable phase 2). Is this the case and if so what is the
&gt; cause?
&gt;
&gt; 4. Are all unstable runs grouped in the parameter space, or are some scattered
&gt; thoughout the parameter space? Is there a difference between those that are
&gt; always unstable and those that are just sometimes unstable?
&gt;

The supplementary information to the first results paper in nature
http://www.nature.com/nature/journal/v433/n7024/suppinfo/nature03301.html
says:

Cooling Simulations
There is a well understood mechanism for models with a mixed layer ocean to produce dramatic, unphysical cooling. The process begins in the eastern tropical Pacific. During the calibration phase there is a mean level of low cloud over the region and consequently a negative cloud radiative forcing. The heat flux convergence field is negative here, mimicking the cooling effect of upwelling colder ocean water. During a subsequent phase the regional amount of cloud may become greater than was typical in the calibration phase. As a result the amount of short wave radiation penetrating to the surface is reduced and the sea surface temperatures fall. This can lead to further low-level cloud and a positive feedback mechanism, ultimately cooling the whole planet. This runaway effect is an unphysical consequence of using a mixed layer ocean; it could happen at any point in the control or double CO2 phases. Some model versions may be more susceptible but any model versions with sufficient variability, potentially including the unperturbed model, could produce it given sufficiently long simulations.

The climateprediction.net simulations highlight this generic problem of mixed layer models, as a consequence of three factors: i) the total number of model years is far greater than in any previous experiment so the effect is simply more likely to occur, ii) some perturbed model versions may be more susceptible due to increased variability, and iii) the experimental design requires that all model versions use a 15 year calibration phase – for model versions with higher than standard variability it may be necessary to increase the length of this phase to reduce the likelihood of encountering the problem.

Of the simulations removed from the analysis due to a drift in the control, many exhibit only slow drifts in global mean temperature; a result either of an insufficiently long calibration or a spurious deduction as a consequence of natural variability in the control. The remaining ones appear to be a result of the above described effect. Its consequences are seen in the surface temperature field which show a dramatic cooling in the eastern equatorial pacific. Supplementary figure 2 shows the difference in the surface temperature field between the control and calibration phases for (a) a stable simulation and (b) an unstable simulation showing this effect. These unstable simulations are removed from the analysis by the requirement that the drift in Tg be less than 0.02 K/year in the last eight years of the control (see Methods – Data Quality). It is impractical to assess these fields by eye for all simulations in our grand ensemble but such manual verification confirms that none of the simulations which pass the quality and stability checks and have sensitivity greater than 8.5K, exhibit this problem.

Six simulations have stable controls but show a negative drift in the double CO2 phase. These are also a result of the above described effect as can be seen in supplementary figure 3; in the simulation shown the 8 year mean surface temperature in the double CO2 phase shows a strong cooling in the tropical pacific with temperatures down to 27K below the calibration phase. We consider it justified, therefore, to remove them from the current analysis. However, it will be important not to omit such parameter combinations from future experiments using models with dynamic oceans since they would not exhibit the same problem.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

&gt;Some model versions may be more susceptible but any model versions with &gt;sufficient variability, potentially including the unperturbed model, could &gt;produce it given sufficiently long simulations.

So when I have tried to find the parameters that cause cold equators, I am probably finding the parameters that cause more variability. This probably explains why there is a considerable overlap between the parameters that apparently cause cold equators and the parameters that cause hot models.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.

I think all my questions are answered by the paper, supplementary discussion and answers on this thread.

Thanks crandles and all who replied.

---

____________________________<br>
<a href="http://www.boincforum.info/boinc/">boinc forum</a> and <a href="http://www.uk4cp.co.uk/">United Kindom</a> team, my climate change <a href="http://www.livejournal.com/users/mike_atkinson/">blog</a>.

oops mistake.

---

Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.