Someone has kicked something.

My hadcm3s zip is now uploading. With my bored band speed, I suspect that even were it not for the problems with space for data at Oxford, a really fast computer would be producing data more quickly than I could upload it if running all new global models!
........
........
........


And finally upload finished!

I have been stuck uploading some zips for batch 691 for several days. Transient HTTP error.

---

I wonder if they go somewhere else, I don't have any of that batch but the four different batch numbers I do have tasks for all seem to be uploading their zips normally.

Ian has/had batch 691 upload problems as well. Andy just changed a timeout value on the server, but I don't know if that will solve the problem.

The detail for batch 691 uploads is -


12/17/2018 12:27:16 PM | climateprediction.net | Started upload of wah2_cam25_a09n_200405_18_691_011369896_0_r1612571459_7.zip
12/17/2018 12:27:16 PM | climateprediction.net | [file_xfer] URL: http://upload6.cpdn.org/cgi-bin/file_upload_handler
12/17/2018 12:27:18 PM | | Internet access OK - project servers may be temporarily down.

---

I have been stuck uploading some zips for batch 691 for several days. Transient HTTP error.

As George mentioned, I also have stuck uploads for a task in this batch. The task has completed and everything has been successfully sent to upload6 except for:

• wah2_cam25_a0fm_200405_18_691_011370111_2_r66429025_13.zip is stuck at 54.19%, first failed at 13:07:05 on 15th, has been retried 24 times and is getting up to 54.41% on each attempt.
  
  
• wah2_cam25_a0fm_200405_18_691_011370111_2_r66429025_16.zip is stuck at 8.56%, first failed at 05:58:34 on 16th, has been retried 15 times and is getting up to 8.68% on each attempt.
  

In both cases the BOINC event log shows that the file was "locked by file_upload_handler" for 2 hours after the first attempt and gives no indication why subsequent attempts have been failing.

Wireshark shows that retries are successfully negotiating the restart offset and data is being sent before timing out. The same restart point is negotiated on every retry, indicating that none of the retransmitted data is being received.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

My first one of these cam25's had 3 zips that refused to upload.
As back then no one knew if they were even needed anymore, I let the model run, then made a copy of the 3 zips and aborted them to let the model finish.
But I've now got a second one, which has just uploaded it's zip 15, and there are no problems so far.

I've been having similar problems with a few sas50's but they eventually get through.

It looks like those two research places need better control of their servers.
Possibly a lot of people are putting files and programs onto the one server, and it can't handle it.

I'm getting these messages for all of my wah2 zip files:

1/2/2019 1:01:59 AM | climateprediction.net | Temporarily failed upload of wah2_safr50_n0nt_198812_14_781_011715489_1_r171020809_7.zip: connect() failed
1/2/2019 1:01:59 AM | climateprediction.net | Backing off 00:06:25 on upload of wah2_safr50_n0nt_198812_14_781_011715489_1_r171020809_7.zip

There are 5 of them queued to transfer all with the same type error.

---

I also get Transient HTTP error for the safr WUs, it should go to a UK server starting 192.171..

Getting problems on both sas50 and safr50s. Email sent to project.

Might get some response but it was only their first day back after Xmas - possibly.

Might get some response but it was only their first day back after Xmas - possibly.

Hilary term doesn't start until 13th so I am not holding my breath, though some things were sorted after last term finished. I just don't know who starts work before the new term and who is on a strict academic time table.

I would expect support staff to be back as of yesterday - but there again.... If the project aren't checking email.

All of the safr50 779, 780 and 781 zips on one machine cleared out within the last 20 minutes, but the 777 on another machine remain stuck.

I have one 691 stuck

wah2_cam25_a0de_200405_18_691_011370031

---

My 777 are now uploading, with four zips remaining. They are 92 MB each. I know that has been commented on before, and it is no problem for my ISP. But it helps explain why the servers are struggling.

All my backlog has uploaded. Now to see how long it takes to get credit :)

---

mine are still sitting there.

from the logs it ssems to be a spefically upload6.cpdn.org which is having an issue.

04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Info: Found bundle for host upload6.cpdn.org: 0x355d500 [can pipeline]
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Info: Re-using existing connection! (#227) with host upload6.cpdn.org
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Info: Connected to upload6.cpdn.org (158.97.9.11) port 80 (#227)
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Host: upload6.cpdn.org
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.14.2)
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Accept: */*
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Accept-Encoding: deflate, gzip
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Content-Type: application/x-www-form-urlencoded
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Accept-Language: de_DE
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Content-Length: 70022930
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: Expect: 100-continue
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server:
04.01.2019 12:22:26 | climateprediction.net | [http] [ID#130] Sent header to server: ain=.google.com; HttpOnly
04.01.2019 12:22:27 | climateprediction.net | [http] [ID#130] Received header from server: HTTP/1.1 100 Continue
04.01.2019 12:22:48 | climateprediction.net | [http] [ID#130] Info: Recv failure: Connection was reset
04.01.2019 12:22:48 | climateprediction.net | [http] [ID#130] Info: Closing connection 227
04.01.2019 12:22:48 | climateprediction.net | [http] HTTP error: Failure when receiving data from the peer
04.01.2019 12:22:48 | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
04.01.2019 12:22:48 | climateprediction.net | [file_xfer] file transfer status -184 (transient HTTP error)
04.01.2019 12:22:48 | climateprediction.net | Temporarily failed upload of wah2_cam25_a0de_200405_18_691_011370031_0_r1337013121_7.zip: transient HTTP error
04.01.2019 12:22:48 | climateprediction.net | Backing off 04:46:21 on upload of wah2_cam25_a0de_200405_18_691_011370031_0_r1337013121_7.zip

---

[Rayburner wrote:]

mine are still sitting there.

from the logs it ssems to be a spefically upload6.cpdn.org which is having an issue.

...

I believe that there is a particular problem with old CAM25 models, such as the batch #691 example you've got (issued on 13 December 2017). However, I can't immediately find a solution, other than waiting or aborting.

I will just wait … :-)

---