<b>Announcement: Database residual problem - misallocated WUs</b>

Carl kindly fixed the database when it ran out of room. Unfortunately before it was fixed some WUs have got allocated to the wrong host and/or to more than one host.

At this time, we are not sure of the best way to fix the problem, but would like to try to gather some information to determine extent of the problem.

A partial solution/ recommendation is given below, but first:

<b>Determining if the problem affects you</b>
Look at your account, select view computers. Select each ComputerID and then results then each recent ResultID (sent April or May) to find the WU name (something like 3iw8_200186109_1); compare to the WUs that that computer has (shown on work tab or in the terminal window).

<b>There are 2 ways you could be affected</b>
1. If you have a WU on your computer which is not in the list of results,
2. If there is a WU in your list of result <b>AND</b> there is work done by another computer.

<b>Things that are OK</b>
1. If there are WUs allocated to you in results list, but not on your computer, where there is no work done. (There have always been loads and loads of these.)
2. If there are WUids sent out more than once with different resultids. This is supposed to happen. (WUids can confuse people, try to work with resultids.)

<b>What to do</b>
If you have a WU which is not in your list of results for that host: If you have done less than a couple of hours of work on the WU then it is easiest and safest to just abort the run. If you have done a lot of work that you would rather not loose, it may be sensible to suspend work on this WU rather than risk doing more work when there is no guarantee that we will find a way to fix the problem. You can try to download another WU by setting your connect to server at most every x days preference very high. (Try not to let other projects communicate with this setting.)

BOINC Versions over 4.19 have options for suspending and aborting WUs. If you have v4.19 or less, you can either upgrade to get these options. Alternatively, for abort just delete some of the files so that the model will fail. For suspend on 4.19 without upgrading, the only possibility is to temporarily change the resource shares.

We are very sorry about these problems and any confusion, lost credit/work and any investigative work this may cause. Please bear with us while we try to work out how best to deal with the problems.

<b>Added Info 23 May</b>

<b>Good info we now know</b>
It appears units processed by wrong host will be accepted so the science does not go to waste.

<b>Bad info we now know</b>
The credit is not granted during nor at the end of a run. The credit may eventually be fixed but we cannot guarantee this.

<b>What we don't know (Probably Bad) </b>
How the upload servers will react to host trying to upload a run that is already uploaded. It could overwrite the first upload, reject the upload, or it might be saved because it went to a different upload server.

<b>Consequences</b>
Therefore we still don't want 2 hosts completing a resultid.

<b>Further suggested possible actions</b>
So if you are only interested in the science and not the credits, it should be ok to continue unless you think someone else is also running the resultid.

If you want the credits and are very trusting that this issue will be fixed, it should also be ok to continue unless you think someone else is also running the resultid.

If you are still unsure about continuing but don’t want to leave CP suspended too long: Stop BOINC and backup your BOINC folder. After restarting abort the WU or reset the project. Then if it later becomes clear that you can finish the WU, you can take another backup then restore the old backup. You would also have to suspend all work other than CP in the old version because you may have already returned that work. When the problem WU uploads, you can restore the second backup.

I know that this suggestion is a mess. Sorry I cannot offer anything better or simpler until Tolu looks at the problem.

<b>Added info 1 June</b>
When you download a new workload, it is advisable to check whether it is ok or not. The method is the same as indicated above. The WU should show up in the results list almost immediately so there is no need to wait for the first trickle or for a 4 hourly update.

<b>Edits 4th June</b>
Removed intructions to report resultids, host numbers and WU names. We would still be grateful for any reports that indicate a new aspect or different affected time period eg affected resultids less than 719000 or greater than 904515.

Clarified that WUids sent out more than once as different resultids are NOT a problem under new title of 'things that are OK'.

Added note about trying to get another model if suspended by setting connect to server preference high.

Edit 11 June: revised upper limit of range.

Edit <b>29 July 2005</b>

<b>Credit has been fixed</b>.

Hi there

The following WU, 2vy7_300156078_0, <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=850098">Result ID#850098</a>

was downloaded by Amy, <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=168273">ID#168273</a>

on Monday 16th May. It was not registered to Amy, but later appeared as registered to the following
machine <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=141454">ID#141454</a>

Checking the results for that machine shows the outcome as unknown and with a credit of 472.59. Tracing ths credit through the 5 trickles for the WU show that they were obtained, not by the registered machine ID#141454, but by machine ID#168273, i.e. Amy.

That WU has now been stopped and deleted from Amy on 18th May after 5 trickles so no further credit will ever appear that is allocated to Amy. A replacement model was downloaded which is properly registered and running.

Pete

My computer ID 5429 downloaded 2mb3_300143459, result ID 836495 http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=836495
which appears to be on host 167123 with a client error but not in my results list. It had five trickles and 472.59 credits. I aborted it and it downloaded a new model which does show up in the results for this computer.

Pam

---

<img border=\"0\" src=\"http://boinc.mundayweb.com/one/stats.php?userID=2247\" />

I've had 6 results that fall into this category.

1q6k_600101414_1: downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=20652">host id 20652</a>. Registered host unknown as I deleted before it started running.

2mab_300143431_0: downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=20652">host id 20652</a>. Registered host unknown as I deleted before it started running.

2sw4_300152076_1: <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=786878">result id 786878</a>, downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=39976">host id 39976</a>. I'm still running this one as it had failed on its registered <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=81872">host id 81872</a>.

3gk4_200183051_1: <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=823392">result id 823392</a>, downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=39968">host id 39968</a>. Killed this one after 7 trickles (only the last 2 are listed) as it hadn't failed on its registered <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=164243">host id 164243</a>.

3ncs_200192043_0: <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/result.php?resultid=814709">result id 814709</a>, downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=39968">host id 39968</a>. I'm still running this one as it had failed on its registered <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=96608">host id 96608</a>.

3nfg_200192043_0: downloaded to <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=34">host id 34</a>. Registered host unknown as I deleted before it started running.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

I have one result with this problem. Some others were allocated but I have deleted them already. This is the info on the troublesome one:

Result: 2svq_300152062_1 (result id 786821)

Host ID working on the result: 69015; got to phase 3 step 226842 so far, so I am completing it. No credit is being granted and is not showing in my results list. The previous result by this computer (69015) is completed but is shown as 'in progress' in the results list (this is result 758126 = 2ofa_200146229_1).

The database shows, however that 2svq_300152062_1 (786821) was allocated to host 73891, which is not mine, and which got a download error when geting this work unit (now that host is working on a different WU).

Two other work units I have deleted already are:

ResultID WU HostID DB_Status
853059 2y7f_300159031_0 129362 Unsent Deleted after ph 1/10802
853400 2ygs_300159372_0 129362 Unsent Deleted before starting

Host 76637 also got an 'unsent' unit queued which I deleted, but I do not have the information on that one anymore.

Thanks and regards,

LS (user id 28498)

---

LS

Some possible good news
After you complete the unit, it is possible that the 4 hourly sweep will grant you credit. It would be good to know if this happens so we can give better advice to others as to whether to continue.

Some possible bad news
Your upload may be rejected because it is not the host that was allocated the WU. It would be a good idea to backup your BOINC folder before it completes. That way if the model is rejected but Tolu subsequently fixes things so that the model can be uploaded, you may be able to restore the backup, get the credit due to you, and ensure the scientists can use your model.

Please let us know what happens.

That is unless Carl or someone else is able to tell us beforehand. ;)

Is it know WHICH database is messed up? User accounts, science, both?

I suspect that the credits db / xml files are also useless, and hope to get around to writing it up on another thread.

Les

&gt; Is it know WHICH database is messed up? User accounts, science, both?
&gt;
&gt; I suspect that the credits db / xml files are also useless, and hope to get
&gt; around to writing it up on another thread.
&gt;
&gt; Les
&gt;
&gt;
I suspect the science database is much more secure against problems. It is possible that the science database will reject models from the wrong host and is therefore problem free. If the science database does accept models from the wrong host, will this matter? Where the scientists are interested in the details of the computer that did the run, they could get confused. However I don't think that is a big issue.

The thing that may be a big issue for the science database would be if the model files downloaded came from one model but the name and result id indicated a different model. I am not aware of having seen anything to indicate this is happening. I could be wrong about that, however even if this is happening then it may not be a problem if the files uploaded contain confirmation of the parameters used. That seems a sensible precaution that should have been taken.

Whatever the situation there could still be affects on information such as the proportion of complete runs returned for each model sent out but hopefully only a small number of runs would be affected and the effect would be unnoticably small compared to normal random variation.

Carl mentioned 196 runs with userid=0. I suspect runs where 2 hosts are processing it are additional to this but I hope this provides reason to believe it is not many thousands of runs with problems.

&gt; Is it know WHICH database is messed up? User accounts, science, both?

It's the results table. Each result has a unique number and name (suffix of '_0', '_1', etc) and should only be processed by one host. A combination of scheduler problems caused some results to be sent to more than one host.

Read on if you want the full details ...

When a result is created its associated hostid is left empty. This means it is available for transmission to any host that requests work.

When a work request is received the scheduler searches for an unassigned result it can send to the requesting host. Having found one it updates the results table to record the result as being processed by that host. This removes it from the pool of results the scheduler can select from for subsequent requests.

After the disk problems on the server the other week some table updates started timing out. Not a problem if the allocation was rolled back, but the BOINC scheduler wasn't doing that. It was sending the result to the selected host without the table being updated, which meant the result could be sent to any number of hosts until the update finally succeeded (this has been fixed in the latest BOINC code).

As the announcement says, the only way you can detect if this has happened to you is by checking for results on each of your computers that don't appear in the result list for that host. If trickles for the result appear on the host's list you can use that to track which host the result was finally allocated to (it might still appear as 'unsent').

If you have uploaded trickles for results not registered to your host and they don't appear on the host's trickle list it's likely that another host has already passed that trickle point, in which case your trickles will have been discarded as repeats by the scheduler.

The other thing to check for is results which are registered to your host and list trickles from another user's host. One example of this is <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/trickle.php?resultid=786718">result id 786718</a>, which lists trickles from hosts 102841 (the registered host) and 133215 (result now discarded).

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

I received an "unsent" workunit to process - 2nf4_300144914_0, result ID 838087.
The machine in question is 18108.

I have now aborted and am runing normally with a new wu.

---

I will report tomorrow on the completion of result <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/trickle.php?resultid=786821">786821</a> , and I will also back up the whole directory as crandles suggested. I hope the result can be uploaded and get used. I consider the credit problem only a minor matter.

Regards,

LS

---

Not a missing WU on the list, but My stats show a WU that I never received,
or should say, BOINC doesn't list it.. but CPDN shows I did....


Result
Result ID 32091
Name 025u_400027787_0
Workunit 22240
Created 25 Aug 2004 11:24:39 UTC
Sent 31 Aug 2004 5:03:33 UTC
Received ---
Server state In Progress
Outcome Unknown
Client state New
Exit status 0 (0x0)
Host ID 6914
Report deadline 13 Aug 2005 10:23:33 UTC
CPU time 0.00
stderr out

Granted credit 0.00
Client version ---
Trickle # 0
Perturbed Parameters for Result # 32091
UK Met Office HadSM3 Slab Model
Description Value Used Default Value Unit
threshold for precipitation over land 2.e-3 2.e-4 kg/m3
entrainment coefficient 0.6 3
initial condition parameter 0.05 0.
accretion constant 4.e-4 1.e-4 /s
threshold for precipitation over sea 5.e-4 5.e-5 kg/m3
critical relative humidity 0.95, 0.9, 0.85, 0.6
0.6, 0.6, 0.6, 0.6
0.6, 0.6, 0.6, 0.6
0.6, 0.6, 0.6, 0.6
0.6, 0.6, 0.6
0.95, 0.90, 0.85, 0.70
0.70, 0.70, 0.70, 0.70
0.70, 0.70, 0.70, 0.70
0.70, 0.70, 0.70, 0.70
0.70, 0.70, 0.70

Run Information Received
Temperature





Precipitation




WUs 24506 is still running, and 474786 is waiting to be run...
but is it 'safe' to abort the 474786 and hope someone will get it to work on it?

running Boinc v4.43

---

Mark<br><br><A HREF="http://www.boincsynergy.com"><IMG SRC="http://www.boincsynergy.com/images/stats/comb-1215.jpg"></A>

host id 165678
work id 47911
result id 719682

work unit 26sp_300123158_0

This is still running but result indicates done with client error

---

&gt; host id 165678
&gt; work id 47911
&gt; result id 719682
&gt;
&gt; work unit 26sp_300123158_0
&gt;
&gt; This is still running but result indicates done with client error

That one's not a problem Allan. <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/trickle.php?resultid=719682">That result</a> is registered to your host and the other system that was running it no longer exists (I guess the owner must have merged it). And there's no need to worry about losing credits because of the first 46 trickles being sent by the other system as you get the credits appropriate for your most recent trickle.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

&gt; Result ID 32091

There are loads and loads of these, don't worry about it.

&gt;
&gt; WUs 24506 is still running, and 474786 is waiting to be run...
&gt; but is it 'safe' to abort the 474786 and hope someone will get it to work on
&gt; it?
&gt;
&gt; running Boinc v4.43
&gt;
Result Id 826900 (WU 474786) is allocated to you, so you should NOT want to abort it. (If you haven't got the WU and someone else has then it may be sensible for it to be aborted, but you won't be able to do that. )

I think you may have got confused between resultid 703595 (named 23hq_200118833_0) and resultid 826900 (named 23hq_200118833_1) both are WU 474786 but that is not a problem because of the different endings on the name of the result.

To avoid confusion, it is better to work on ResultIDs' rather than WU numbers. I have edited the announcement to hopefully make this clearer.

It wouldn't matter if you did abort it. However if there was work done...., I am trying to avoid loss of work. Sorry if my instructions have not been very clear, I know it is easy to get confused.

&gt; Not a missing WU on the list, but My stats show a WU that I never received,
&gt; or should say, BOINC doesn't list it.. but CPDN shows I did....
&gt;
&gt; Result ID 32091
&gt;
&gt; WUs 24506 is still running, and 474786 is waiting to be run...
&gt; but is it 'safe' to abort the 474786 and hope someone will get it to work on
&gt; it?

Your current status looks fine to me Mark.

Although result id 32091 was allocated to you it never made it to your system (I've had a fair few of them!). It should eventually be rescheduled.

Result id 826900 (WU 474786) has been downloaded ready for your system to start when it finishes result id 34357 (WU 24506), so I definitely wouldn't delete it.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

&gt; &gt; host id 165678
&gt; &gt; work id 47911
&gt; &gt; result id 719682
&gt; &gt;
&gt; &gt; work unit 26sp_300123158_0
&gt; &gt;
&gt; &gt; This is still running but result indicates done with client error
&gt;
&gt; That one's not a problem Allan. <a> href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/trickle.php?resultid=719682"&gt;That
&gt; result</a> is registered to your host and the other system that was running it
&gt; no longer exists (I guess the owner must have merged it). And there's no need
&gt; to worry about losing credits because of the first 46 trickles being sent by
&gt; the other system as you get the credits appropriate for your most recent
&gt; trickle.
&gt;
Thanks for reporting it. It is the oldest one I have seen so far. It was sent 7th April with first trickle by wrong host on 9th April. So I wonder if, to be safe, runs sent March should also be checked.

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=530252">39vg_200174302</a> got broken in a switch to the new 4.35 BOINC version. So it should be saying "aborted" or Client error by now.

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=542626">3jbu_200186677</a> is an unknown to me (although the date is prior to around the 15/16th).

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=564251">2tg5_300152804</a> never graced my system.

<a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/workunit.php?wuid=564265">2tgi_300152818</a> <i>IS</i> the one I am currently running, but which doesn't seem to want to trickle. Seeing how it has run alone for the past 2 days due to negative long term debt with CC4.42 / CC4.43, I would've expected at least one trickly, if not 4 by now. ;)

I have one Aborted unit on my BOINC system that's nowhere in my list though: 2u6l_300153766

My host #ID is <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=167707">167707</a>.

---

Jord.

&gt; &gt; Result ID 32091
&gt;
&gt; There are loads and loads of these, don't worry about it.
&gt;

Okay.. thats what I thought... ;)

&gt; &gt;
&gt; &gt; WUs 24506 is still running, and 474786 is waiting to be run...
&gt; &gt; but is it 'safe' to abort the 474786 and hope someone will get it to work
&gt; on
&gt; &gt; it?
&gt; &gt;
&gt; &gt; running Boinc v4.43
&gt; &gt;
&gt; Result Id 826900 (WU 474786) is allocated to you, so you should NOT want to
&gt; abort it. (If you haven't got the WU and someone else has then it may be
&gt; sensible for it to be aborted, but you won't be able to do that. )
&gt;

I've got it in Boinc... it waiting (actually suspended at the moment waiting to get my system backlog cleaned out...)

&gt; To avoid confusion, it is better to work on ResultIDs' rather than WU numbers.
&gt; I have edited the announcement to hopefully make this clearer.
&gt;
&gt; It wouldn't matter if you did abort it. However if there was work done...., I
&gt; am trying to avoid loss of work. Sorry if my instructions have not been very
&gt; clear, I know it is easy to get confused.

Confusion is always easy to happen.. though it may seem clear at the time it's written....

---

Mark<br><br><A HREF="http://www.boincsynergy.com"><IMG SRC="http://www.boincsynergy.com/images/stats/comb-1215.jpg"></A>

Good call with this thread :) I had one extra CPDN WU on one of my hosts and had actually planned to run it (I admit, I never thought to check the results list).

The WU in question is 2m8o_300143371_0 - NOT in my results list - 0 (zero) work done on it, and now aborted.

Host in question is 52531.

Hope that helps

---