website is back up but getting

Fri 06 Aug 2021 17:21:54 BST | climateprediction.net | Project is temporarily shut down for maintenance

On project update. Unless someone is doing overtime, it will be Monday before everything stands a chance of returning to normal. Will post again when more bits return to normal. Once things do start working again, then and only then let us know if anything is behaving oddly.

Thanks.

Edit:1 hour later, 9 completed tasks uploaded and 8 new ones now downloading.

I had two that had competed a day or two ago, but had not been reported yet.
A manual update fixed it, and I am now all up to date.
https://www.cpdn.org/results.php?hostid=1520871

I have had transient http errors on all 8 downloads and Richard has had all 4 of his new tasks allocated fail to download. Clearly they need to pay more overtime.

I had one that was ready to report but stuck - no contact with site since this morning. Solved by suspending then resuming network activity manually (no transfers waiting). Now reported and another task has failed to download.

New tasks still not downloading. I have informed Andy but don't expect anything to change till Monday. I don't know enough about the output from the flags I enabled to work out if it is a script needs restarting, internal addresses changed or something more obscure.

They have all gone fishing, finned or? Anyway, I am not being able to download any WU and I suppose I will have to wait till Monday. Let us all pray to the God's of CPDN.

From Andy:

A number of key machines still have no networking access following the switch work on Tuesday.

------------------------

They're going to be spoken to.

One task finished a few days ago and it uploaded yesterday. A second task finished today and it uploaded OK soon after.
My client tried to download a new task yesterday, but is not getting the files. After the second task finished, my client tried to download a second task, but that is stuck too.
So it seems most of CPDN is working, but not downloads yet.

---

From Andy:

A number of key machines still have no networking access following the switch work on Tuesday.

------------------------

They're going to be spoken to.

________________________
Good. At least someone can speak to machines.
My computers have run out of WU's. Which reminds me, trickles are not uploading.

Good. At least someone can speak to machines.
My computers have run out of WU's. Which reminds me, trickles are not uploading.

It seems to me that my trickle files are uploading...

Sat 07 Aug 2021 03:37:41 AM EDT | climateprediction.net | Started upload of hadam4h_b0sf_201211_5_882_012036031_0_r1205737584_5.zip
Sat 07 Aug 2021 03:38:05 AM EDT | climateprediction.net | Finished upload of hadam4h_b0sf_201211_5_882_012036031_0_r1205737584_5.zip
Sat 07 Aug 2021 03:58:22 AM EDT | climateprediction.net | Computation for task hadam4h_b0sf_201211_5_882_012036031_0 finished
Sat 07 Aug 2021 03:58:24 AM EDT | climateprediction.net | Started upload of hadam4h_b0sf_201211_5_882_012036031_0_r1205737584_out.zip
Sat 07 Aug 2021 03:58:28 AM EDT | climateprediction.net | Finished upload of hadam4h_b0sf_201211_5_882_012036031_0_r1205737584_out.zip
Sat 07 Aug 2021 04:58:33 AM EDT | climateprediction.net | Sending scheduler request: To report completed tasks.
Sat 07 Aug 2021 04:58:33 AM EDT | climateprediction.net | Reporting 1 completed tasks
Sat 07 Aug 2021 04:58:33 AM EDT | climateprediction.net | Not requesting tasks: some download is stalled
Sat 07 Aug 2021 04:58:35 AM EDT | climateprediction.net | Scheduler request completed

---

The trickles server is not running, as per the Project Status page.

It seems to me that my trickle files are uploading...

That is the zip files uploading which are produced at the same time as the trickles.

That is the zip files uploading which are produced at the same time as the trickles.

Does that mean my trickles will never be uploaded since, since then, the tasks that produced them have exited, my machine got some updates needing updates, and so my machine has been rebooted?

---

The best place to look to see if trickle_up files have been sent, is in the Event log.

Failing that, as in your case, you should be able to see them on your computer if they're still there. (They're very small.)

Else, if they did upload, you just have to wait until the work at Oxford is finished and all of the servers are working, then check the task page to see if several trickle_ups are listed with the same date/time stamp.

Waiting is where I'm at right now, both for the trickle_ups to show up, and the files for the next task to download.
And BOINC is Suspended, so as not to waste time on futile attempts to communicate.

The best place to look to see if trickle_up files have been sent, is in the Event log.

Failing that, as in your case, you should be able to see them on your computer if they're still there. (They're very small.)
...
Waiting is where I'm at right now, both for the trickle_ups to show up, and the files for the next task to download.
And BOINC is Suspended, so as not to waste time on futile attempts to communicate.

My Event Log does not go back far enough. I do not know where to look for the trickle up messages. slots? projects/climate...?
However it is still in /var/log/messages....

# grep trickle messages-20210808
Aug  4 01:14:32 localhost boinc[2021]: 04-Aug-2021 01:14:32 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 03:55:41 localhost boinc[2021]: 04-Aug-2021 03:55:41 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 04:02:59 localhost boinc[2021]: 04-Aug-2021 04:02:59 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 04:12:17 localhost boinc[2021]: 04-Aug-2021 04:12:17 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 04:24:33 localhost boinc[2021]: 04-Aug-2021 04:24:33 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 04:44:50 localhost boinc[2021]: 04-Aug-2021 04:44:50 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 05:38:06 localhost boinc[2021]: 04-Aug-2021 05:38:06 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 07:17:35 localhost boinc[2021]: 04-Aug-2021 07:17:35 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 10:36:45 localhost boinc[2021]: 04-Aug-2021 10:36:45 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 12:56:50 localhost boinc[2021]: 04-Aug-2021 12:56:50 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  4 16:54:14 localhost boinc[2021]: 04-Aug-2021 16:54:14 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  6 17:01:52 localhost boinc[2021]: 06-Aug-2021 17:01:52 [climateprediction.net] Sending scheduler request: To send trickle-up message.
Aug  7 03:37:24 localhost boinc[2021]: 07-Aug-2021 03:37:24 [climateprediction.net] Sending scheduler request: To send trickle-up message.

I infer things were running OK including most of August 4, and the trickles started going out again starting late on August 6. So maybe all the ones I tried to sent actually went up and I will see them once the web site catches up. I now have three work unit stuck trying to download. I imagine patience will fix this.

I did not suspend Boinc-client, or even just climateprediction because I still have one work unit working, and it might as well upload trickle-up messages and those files that go up at the same time. I suppose I should have set climateprediction to no new tasks, but I did not think of it until you suggested it, and now it is so close to working that I might as well just keep running. It looks at most once an hour.

---

I think they appear under:
/var/lib/boinc-client/projects/climateprediction.net

This is what is in one of mine that I saved ages ago:

<variety>year</variety>
<wu>hadam4_a01y_200611_12_785_011729848</wu>
<result>hadam4_a01y_200611_12_785_011729848_1_r1940024311</result>
<ph>1</ph>
<ts>51941</ts>
<cp>728765</cp>
<vr>8.08</vr>

The complaints have changed in the last day or two. I imagine this is actually improvement. From my event log it now shows like this:

Mon 09 Aug 2021 07:06:51 AM EDT | climateprediction.net | Temporarily failed download of a019_915_atmos.gz: transient HTTP error
Mon 09 Aug 2021 07:06:51 AM EDT | climateprediction.net | Backing off 05:28:01 on download of a019_915_atmos.gz
Mon 09 Aug 2021 07:06:53 AM EDT |                       | Internet access OK - project servers may be temporarily down.

I infer that means they are actually trying to send me stuff, but it is not getting through.

---

The complaints have changed in the last day or two. I imagine this is actually improvement. From my event log it now shows like this:

Mon 09 Aug 2021 07:06:51 AM EDT | climateprediction.net | Temporarily failed download of a019_915_atmos.gz: transient HTTP error
Mon 09 Aug 2021 07:06:51 AM EDT | climateprediction.net | Backing off 05:28:01 on download of a019_915_atmos.gz
Mon 09 Aug 2021 07:06:53 AM EDT |                       | Internet access OK - project servers may be temporarily down.

I infer that means they are actually trying to send me stuff, but it is not getting through.

That is what I was getting on Saturday. I do hope the IT support people can sort it out soon though. No point in my sending another email as I know Andy is aware of it and chasing them.

Yes,
upload is OK
download NOK, at this time.

---

Do we abort these WU's in the pipeline or do we wait and see? Will the server start again from halfway or have these WU's entered the Black Holes on the Internet? Over the years I have accumulated a lot of WU's which can best be described, they are in some Black Hole and be done with the matter. No record on my machines, no record on the server.