Indeed. I had a similar number. I now have a couple of zips showing a transient error, but those invariably clear, so I'm not worried. The server shows the number of WUs in progress dropping very fast.

From something posted on the BOINC forum, Richard Haselgrove (and team?) deserve kudos for a lot of hard work getting the servers back online. Nice work. Hope you get time for a beer or several.

The backlog of uploads is now slowly clearing, but I am seeing a problem with _13 files failing.
For example:
15/08/2014 01:47:07 | climateprediction.net | Started upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip
15/08/2014 01:54:55 | climateprediction.net | Finished upload of hadam3p_eu_iawg_2013_1_008778564_2_6.zip
15/08/2014 01:54:55 | climateprediction.net | Started upload of hadam3p_eu_h71v_2013_1_008862964_0_5.zip
15/08/2014 02:00:42 | climateprediction.net | Sending scheduler request: To send trickle-up message.
15/08/2014 02:00:42 | climateprediction.net | Not requesting tasks: "no new tasks" requested via Manager
15/08/2014 02:00:46 | climateprediction.net | Scheduler request completed
15/08/2014 02:10:11 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_h71v_2013_1_008862964_0_13.zip: No such file or directory
15/08/2014 02:10:11 | climateprediction.net | Temporarily failed upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip: transient upload error
15/08/2014 02:10:11 | climateprediction.net | Backing off 00:05:12 on upload of hadam3p_eu_h71v_2013_1_008862964_0_13.zip
All my _13 files are failing in the same way. Other files are uploading just fine.

approximately 97 % of my Zip files have uploaded. Just 3 left in the BOINC transfer tab. here are some messages I'm getting in the BOINC event Log.


14/08/2014 10:50:27 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:51:05 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip: transient HTTP error
14/08/2014 10:51:05 PM | climateprediction.net | Backing off 00:03:58 on upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:51:05 PM | | Project communication failed: attempting access to reference site
14/08/2014 10:51:09 PM | | Internet access OK - project servers may be temporarily down.
14/08/2014 10:55:04 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:55:27 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip: transient HTTP error
14/08/2014 10:55:27 PM | climateprediction.net | Backing off 00:04:37 on upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 10:55:28 PM | | Project communication failed: attempting access to reference site
14/08/2014 10:55:30 PM | | Internet access OK - project servers may be temporarily down.
14/08/2014 11:00:04 PM | climateprediction.net | Started upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 11:07:34 PM | climateprediction.net | Finished upload of hadam3p_eu_o4rv_2013_1_008832772_0_5.zip
14/08/2014 11:29:43 PM | climateprediction.net | Started upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip
14/08/2014 11:29:45 PM | climateprediction.net | Started upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip
14/08/2014 11:32:56 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_o8qu_2013_1_008837919_0_13.zip: No such file or directory
14/08/2014 11:32:56 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip: transient upload error
14/08/2014 11:32:56 PM | climateprediction.net | Backing off 03:52:14 on upload of hadam3p_eu_o8qu_2013_1_008837919_0_13.zip
14/08/2014 11:32:57 PM | climateprediction.net | Started upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip
14/08/2014 11:34:00 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_o8vv_2013_1_008838100_0_13.zip: No such file or directory
14/08/2014 11:34:00 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip: transient upload error
14/08/2014 11:34:00 PM | climateprediction.net | Backing off 03:59:51 on upload of hadam3p_eu_o8vv_2013_1_008838100_0_13.zip
14/08/2014 11:35:26 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_obgq_2013_1_008841443_0_13.zip: No such file or directory
14/08/2014 11:35:26 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip: transient upload error
14/08/2014 11:35:26 PM | climateprediction.net | Backing off 03:39:50 on upload of hadam3p_eu_obgq_2013_1_008841443_0_13.zip

Yeah, same here;

15-Aug-2014 09:46:53 [climateprediction.net] [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_l685_2013_1_008819294_1_13.zip: No such file or directory

From something posted on the BOINC forum, Richard Haselgrove (and team?) deserve kudos for a lot of hard work getting the servers back online. Nice work. Hope you get time for a beer or several.

Not me. I'm simply a messenger passing information back and forth. If that small cog in the wheel was helpful to anyone, then it was worth doing.

On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers. I'm sure the team will be wanting to review the performance of the new database server first this morning, but after that there should be time (personal guess) to look at uploads too.

Fearless prediction -- since it is Friday.

Servers will fail at about 1700 - 1800 UTC .

Anybody on for some serious betting?

---

On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers.

My _13.zip uploads started working at around 0930 UTC.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

And what will that do?

maybe some "... 13 'files get uploaded -- good.

The totally broken upload situation -- let us pretend --. :)

On that subject, I see that the problem of the _13.zip file uploads failing with the error "No such file or directory" has already been passed directly into the lab by one of the other messengers.

My _13.zip uploads started working at around 0930 UTC.

---

We have another upload problem. The zip files in my transfer tab that built up during the server outage have stopped clearing and are building up again. Is this just server overload, or is there a new problem?

OPPS.

I just saw that cpdn.uploader2orec is listed at not running. Hopefully it won�t be Monday before the staff can get to it and get it running again>

---

Agreed. This time it's the 13.zips that seem to be moving, while it appears everything else is stuck.

Yep, same here.

Agreed. This time it's the 13.zips that seem to be moving, while it appears everything else is stuck.

The zip 13 files are definitely being accepted while all other hadam3p zips are hanging. I have had 2 WU�s finish since this problem started and there is a zip file 6, 7, 8, 9, two 10�s, two 11�s and two 12�s stuck in my transfer tab, but, no sign of the 13�s. They uploaded fine.

---

What is the server name that the stuck files want to upload to?

I think it is cpdnupload2.oerc but I don't know the right file to look at in my BOINC folder to find it.

Hi Dave

The file is client_state.xml
I always copy this and paste it "elsewhere", then look at the copy. Just to be safe. :)

Scan it for the 4 character file name, and keep going until you reach the upload section.

If it is that uploader, then it's a Monday job. :(
Possibly the storage section needs re-mounting.

Thanks Les,

It is http://cpdn-upload2.oerc.ox.ac.uk/cgi-bin/file_upload_handler
which is had gone red on the Server Status page which is what made me suspect it.

I think those of us who have been around a while had already worked out that it was a Monday, post 9.00am job. I have suspended internet access for BOINC and will wait an hour or so after the colour has changed before trying again.

Upload problem - after a lot of work units could be uploaded last week, again since a couple of days some are remaining with the message transient HTTP error - see excerpt of log below:

Sun Aug 17 13:38:23 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:38:23 2014 | climateprediction.net | Backing off 04:40:08 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:38:31 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:38:31 2014 | climateprediction.net | Backing off 04:50:51 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:38:32 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:38:39 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:38:39 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:39:56 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:39:56 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:39:56 2014 | climateprediction.net | Backing off 04:39:06 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:39:56 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:39:56 2014 | climateprediction.net | Backing off 05:07:26 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:39:57 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:40:47 2014 | | Re-reading cc_config.xml
Sun Aug 17 13:40:47 2014 | | cc_config.xml not found - using defaults
Sun Aug 17 13:40:47 2014 | | log flags: file_xfer, sched_ops, task
Sun Aug 17 13:40:54 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:40:54 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:42:05 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:42:05 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:42:05 2014 | climateprediction.net | Backing off 03:44:57 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:42:06 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:42:06 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:42:06 2014 | climateprediction.net | Backing off 05:48:43 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:42:36 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:42:36 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:43:44 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:43:44 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:43:44 2014 | climateprediction.net | Backing off 05:51:52 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:43:46 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:43:52 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:43:52 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:43:52 2014 | climateprediction.net | Backing off 03:46:31 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:43:54 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:43:55 2014 | climateprediction.net | Started upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:43:55 2014 | climateprediction.net | Started upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:45:06 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:45:06 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip: transient HTTP error
Sun Aug 17 13:45:06 2014 | climateprediction.net | Backing off 05:17:53 on upload of hadam3p_eu_lbw2_2013_1_008826635_1_10.zip
Sun Aug 17 13:45:07 2014 | | Internet access OK - project servers may be temporarily down.
Sun Aug 17 13:45:11 2014 | | Project communication failed: attempting access to reference site
Sun Aug 17 13:45:11 2014 | climateprediction.net | Temporarily failed upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip: transient HTTP error
Sun Aug 17 13:45:11 2014 | climateprediction.net | Backing off 04:01:20 on upload of hadam3p_eu_l9o4_2013_1_008823757_0_6.zip
Sun Aug 17 13:45:12 2014 | | Internet access OK - project servers may be temporarily down.

---

I see that server cpdnupload2.orec is listed as being back �up�. This is good news as I presently have more than 50 zip files stuck in my transfer tabs on 3 machines ready to go. Backlog of zip files starting to clear. Lets hope they all go this time.

---

The transfer tab on my fastest machine now completely empty. Second machine is uploading. Some of those zip files are had been there for more than a week.

---

same here also I got some new work :)