climateprediction.net (CPDN) home page
Thread 'Missed daily trickle?'

Thread 'Missed daily trickle?'

Message boards : Number crunching : Missed daily trickle?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Profilemo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 30650 - Posted: 22 Sep 2007, 18:12:59 UTC

We\'re very fortunate on all four ClimatePrediction projects that the programmers - Tolu, Milo and Carl (who has now left) - have always been accessible and very approachable.

We\'re also very fortunate that on the five ClimatePrediction forums, it\'s extremely rare (almost unheard-of) for members to be rude or cause a nuisance. This saves the mods and admins a lot of time which can be better spent in other ways. The newish registration process on the independent forum now excludes most of the spammers, which is a big improvement compared with before and also saves lots of time.
Cpdn news
ID: 30650 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,743,089
RAC: 6,177
Message 31175 - Posted: 29 Oct 2007, 20:58:03 UTC
Last modified: 29 Oct 2007, 20:59:29 UTC

Another statistical glitch - not the \'trickle to credit\' problem we had last time.

The XML stats dump used by external sites such as BOINCStats doesn\'t seem to have run on either Sunday or Monday: see the timestamp on my signature graphic.

It used to run once every 24 hours, around 12:00 UTC. Recently it\'s drifted to mid-afternoon: it would be better if it went back to mid-day, because that fits in better with the other (external) processing cycles, but it doesn\'t really matter - in the timescale of climate change, or even CPDN processing, a few hours or days doesn\'t make that much difference.

Edit - I wonder if something got knocked out of kilter by the clock-change in the UK this weekend?
ID: 31175 · Report as offensive     Reply Quote
Profilemo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 31176 - Posted: 29 Oct 2007, 23:05:23 UTC

Hi Richard

Thanks very much for reporting this. I\'ve had a feeling since yesterday that something was wrong. Have you looked at more stats sites than BoincStats? It sounds as if you have.

http://boincstats.com/index.php?list=full&or=0

It looks to me as if BBC and SAP are exporting their credits normally, but CPDN hasn\'t exported anything for 2 days since Saturday. If you mouse over the right-hand column you see the last contact times.

I think I\'d better ask Milo to look at your post.
Cpdn news
ID: 31176 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,743,089
RAC: 6,177
Message 31178 - Posted: 29 Oct 2007, 23:59:11 UTC - in response to Message 31176.  

Hi Richard

Thanks very much for reporting this. I\'ve had a feeling since yesterday that something was wrong. Have you looked at more stats sites than BoincStats? It sounds as if you have.

I noticed it first on BOINCStats, but it\'s visible too from BOINC Synergy - that\'s why I put my signature in the post. It\'s a live link, and you can monitor whether the XML has been picked up externally just by looking at this thread. When the credit shown under my name to the left matches the numbers in my signature, you\'ll know the problem has been resolved!
ID: 31178 · Report as offensive     Reply Quote
Profilemo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 31179 - Posted: 30 Oct 2007, 1:35:32 UTC

I\'ve announced this briefly in the news threads of all the CPDN forums and 3 stats forums. Could you please mention the problem on the Synergy forum?
Cpdn news
ID: 31179 · Report as offensive     Reply Quote
ProfileMilo Thurston
Volunteer moderator
Volunteer developer

Send message
Joined: 2 Mar 06
Posts: 253
Credit: 363,646
RAC: 0
Message 31181 - Posted: 30 Oct 2007, 11:05:36 UTC

Thanks for letting us know.
The server has the correct time and the logs suggest that the stats scripts have been running, but some of the output files are indeed 2 days old. This is odd.
Anyway, we\'re looking into it now.
ID: 31181 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,743,089
RAC: 6,177
Message 31183 - Posted: 30 Oct 2007, 13:50:33 UTC - in response to Message 31179.  
Last modified: 30 Oct 2007, 13:52:05 UTC

I\'ve announced this briefly in the news threads of all the CPDN forums and 3 stats forums. Could you please mention the problem on the Synergy forum?

No need - everything has synch\'d up already (to within a rounding error, at any rate).

Thanks Mo and Milo.

Edit - 1 minute 6 seconds between \"We\'re looking into it\" and \"XML updated\". That\'s fast.
ID: 31183 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,743,089
RAC: 6,177
Message 31218 - Posted: 31 Oct 2007, 19:20:18 UTC

Stats dump missed again today. I wonder if it\'s linked to the lack of upload storage space reported in the next thread? Those stats files can be big.
ID: 31218 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 31219 - Posted: 31 Oct 2007, 19:38:36 UTC


The core team know about both problems, and they\'re working on them.
Some of the problem is getting staff from other areas to fix their part of the problems. Like installing new servers.

ID: 31219 · Report as offensive     Reply Quote
Profilemo.v
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 2363
Credit: 14,611,758
RAC: 0
Message 31225 - Posted: 31 Oct 2007, 22:47:14 UTC
Last modified: 31 Oct 2007, 22:52:03 UTC

Milo won\'t be around until next Tuesday so the problem may not get fixed until the middle of next week.
Cpdn news
ID: 31225 · Report as offensive     Reply Quote
ProfileJohnofWem
Avatar

Send message
Joined: 15 Feb 06
Posts: 16
Credit: 7,234,664
RAC: 9,842
Message 31237 - Posted: 1 Nov 2007, 12:28:22 UTC

I have a slab model at 99.999% which early this morning showed this over and over again for several minutes

01/11/07 10:33:29|climateprediction.net|Sending scheduler request: To send trickle-up message
01/11/07 10:33:29|climateprediction.net|(not requesting new work or reporting completed tasks)
01/11/07 10:33:34|climateprediction.net|Scheduler RPC succeeded [server version 509]

This appears to say its last trickle succeeded but obviously it didn\'t as it is still stuck at 2 minutes to go.

Eventually after about 3 pages of these messages it went to \'waiting to run\' and downloaded a new 80 year model which is running OK together with an earlier 160 year model. Neither will have a decadal upload for several days.

Is this the same cause as reported here and in the top sticky or is something else going wrong here?
ID: 31237 · Report as offensive     Reply Quote
Profileold_user17525

Send message
Joined: 13 Sep 04
Posts: 161
Credit: 284,548
RAC: 0
Message 31238 - Posted: 1 Nov 2007, 12:34:23 UTC

Hi John,

It looks like the same problem as the slabs do a final zip upload rather than a normal trickle at the end. Its these uploads which aren\'t getting through but it should resolve itself (hopefully) once the server problem is resolved.

Marj
_________________________________
ID: 31238 · Report as offensive     Reply Quote
ProfileIain Inglis

Send message
Joined: 9 Jan 07
Posts: 467
Credit: 14,549,176
RAC: 317
Message 31239 - Posted: 1 Nov 2007, 14:53:46 UTC
Last modified: 1 Nov 2007, 16:32:54 UTC

John,

As Marj says, the final zip file may be delayed, but the slab models do trickle in a different way to the coupled models.

Here\'s an example of a slab model completing on a machine of mine. It trickles 26 times before sending the file (i.e. the 24 third-phase trickles plus two others).


2007-10-26 06:57:21 [climateprediction.net] [checkpoint_debug] result hadsm3fub_e062_005905825_5 checkpointed
2007-10-26 06:59:07 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 06:59:07 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 06:59:12 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 06:59:17 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 06:59:17 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 06:59:23 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 06:59:33 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 06:59:33 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 06:59:38 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 06:59:48 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 06:59:48 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 06:59:59 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:00:09 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:00:09 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:00:14 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:00:45 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:00:45 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:00:50 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:00:55 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:00:55 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:01:10 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:01:16 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:01:16 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:01:21 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:01:26 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:01:26 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:01:31 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:01:36 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:01:36 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:01:41 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:01:47 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:01:47 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:01:52 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:01:57 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:01:57 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:02:04 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:02:09 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:02:09 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:02:19 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:02:24 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:02:24 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:02:29 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:02:35 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:02:35 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:02:40 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:02:47 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:02:47 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:02:53 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:02:57 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:02:57 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:03:02 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:03:09 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:03:09 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:03:13 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:03:20 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:03:20 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:03:30 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:03:35 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:03:35 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:03:40 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:03:46 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:03:46 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:03:51 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:03:56 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:03:56 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:04:01 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:04:07 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:04:07 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:04:12 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:04:17 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:04:17 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:04:22 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:04:27 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:04:27 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:04:39 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:06:18 [climateprediction.net] Sending scheduler request: To send trickle-up message
2007-10-26 07:06:18 [climateprediction.net] (not requesting new work or reporting completed tasks)
2007-10-26 07:06:23 [climateprediction.net] Scheduler RPC succeeded [server version 509]
2007-10-26 07:06:35 [climateprediction.net] Computation for task hadsm3fub_e062_005905825_5 finished
2007-10-26 07:06:36 [climateprediction.net] [file_xfer] Started upload of file hadsm3fub_e062_005905825_5_3.zip
2007-10-26 07:07:42 [climateprediction.net] [file_xfer] Finished upload of file hadsm3fub_e062_005905825_5_3.zip
2007-10-26 07:07:42 [climateprediction.net] [file_xfer] Throughput 45381 bytes/sec
2007-10-26 09:31:46 [climateprediction.net] Sending scheduler request: To report completed tasks
2007-10-26 09:31:46 [climateprediction.net] Reporting 1 tasks
2007-10-26 09:31:51 [climateprediction.net] Scheduler RPC succeeded [server version 509]


Slab models also trickle when resuming, which coupled models don\'t.

Iain

[Edit: My stalled Zip has cleared. If you disabled network activity after the upload failure then there\'s no harm in enabling it to see whether the upload now works. Anyone who pre-emptively disabled networking to prevent the upload expiry count starting might want to wait until a few uploads have been reported as successful before re-enabling network activity.]
ID: 31239 · Report as offensive     Reply Quote
ProfileJohnofWem
Avatar

Send message
Joined: 15 Feb 06
Posts: 16
Credit: 7,234,664
RAC: 9,842
Message 31241 - Posted: 1 Nov 2007, 16:38:12 UTC

Just counted the trickles. There were 37 (!!) of them before it finally suspended itself and a new model downloaded.

Not to worry, I will leave it for a few days and then activate it again to see what happens
ID: 31241 · Report as offensive     Reply Quote
ProfileastroWX
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1496
Credit: 95,522,203
RAC: 0
Message 31243 - Posted: 1 Nov 2007, 18:40:37 UTC


I think there was a batch of Slab Models that generated spurious Trickles near Phase ends. To my knowledge, those Trickles are nothing more than a nuisance. We\'re running Beta tests on a new version, which looks good, so far.
"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.
ID: 31243 · Report as offensive     Reply Quote
ProfileJohnofWem
Avatar

Send message
Joined: 15 Feb 06
Posts: 16
Credit: 7,234,664
RAC: 9,842
Message 31265 - Posted: 3 Nov 2007, 0:11:55 UTC

Thanks all.
Started it up again by suspending one of the others. It uploaded a zip file and now says the magic words \'Ready to report\'
ID: 31265 · Report as offensive     Reply Quote
ProfileMikeMarsUK
Volunteer moderator
Avatar

Send message
Joined: 13 Jan 06
Posts: 1498
Credit: 15,613,038
RAC: 0
Message 31269 - Posted: 3 Nov 2007, 11:40:27 UTC


Excellent :-)

I'm a volunteer and my views are my own.
News and Announcements and FAQ
ID: 31269 · Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : Number crunching : Missed daily trickle?

©2024 cpdn.org