Author | Message |
JimMcCarthy_StellarSolns

Send message Joined: 3 Sep 08 Posts: 23 Credit: 42,176,240 RAC: 10,608
|
The subject line summaries what I'm seeing. Maybe the following details will help ...
3/7/2014 2:55:08 PM | | Resuming network activity
3/7/2014 2:55:08 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
3/7/2014 2:55:08 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
3/7/2014 2:55:08 PM | climateprediction.net | Started upload of hadam3p_eu_l1ct_2013_1_008537308_0_13.zip
3/7/2014 2:55:08 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
3/7/2014 2:55:08 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
3/7/2014 2:55:08 PM | climateprediction.net | Started upload of hadam3p_eu_l1cy_2013_1_008537313_0_13.zip
3/7/2014 2:55:08 PM | climateprediction.net | [http] [ID#866] Info: Connection #0 seems to be dead!
3/7/2014 2:55:08 PM | climateprediction.net | [http] [ID#866] Info: Closing connection #0
3/7/2014 2:55:08 PM | climateprediction.net | [http] [ID#866] Info: Connection #1 seems to be dead!
3/7/2014 2:55:08 PM | climateprediction.net | [http] [ID#866] Info: Closing connection #1
3/7/2014 2:55:09 PM | climateprediction.net | [http] [ID#866] Info: About to connect() to cpdn-restarts.oerc.ox.ac.uk port 80 (#0)
3/7/2014 2:55:09 PM | climateprediction.net | [http] [ID#866] Info: Trying 129.67.195.121...
3/7/2014 2:55:09 PM | climateprediction.net | [http] [ID#867] Info: About to connect() to cpdn-restarts.oerc.ox.ac.uk port 80 (#1)
3/7/2014 2:55:09 PM | climateprediction.net | [http] [ID#867] Info: Trying 129.67.195.121...
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Info: Connected to cpdn-restarts.oerc.ox.ac.uk (129.67.195.121) port 80 (#0)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Info: Connected to cpdn-restarts.oerc.ox.ac.uk (129.67.195.121) port 80 (#0)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.28)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: Host: cpdn-restarts.oerc.ox.ac.uk
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: Accept: */*
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: Accept-Encoding: deflate, gzip
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: Content-Type: application/x-www-form-urlencoded
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server: Content-Length: 294
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Sent header to server:
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Info: Connected to cpdn-restarts.oerc.ox.ac.uk (129.67.195.121) port 80 (#1)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Info: Connected to cpdn-restarts.oerc.ox.ac.uk (129.67.195.121) port 80 (#1)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.28)
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: Host: cpdn-restarts.oerc.ox.ac.uk
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: Accept: */*
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: Accept-Encoding: deflate, gzip
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: Content-Type: application/x-www-form-urlencoded
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server: Content-Length: 294
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Sent header to server:
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server: HTTP/1.1 200 OK
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server: Date: Fri, 07 Mar 2014 22:56:28 GMT
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server: Server: Apache
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server: Content-Length: 93
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server: Content-Type: text/plain
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Received header from server:
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#866] Info: Connection #0 to host cpdn-restarts.oerc.ox.ac.uk left intact
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server: HTTP/1.1 200 OK
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server: Date: Fri, 07 Mar 2014 22:56:28 GMT
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server: Server: Apache
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server: Transfer-Encoding: chunked
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server: Content-Type: text/plain
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Received header from server:
3/7/2014 2:55:10 PM | climateprediction.net | [http] [ID#867] Info: Connection #1 to host cpdn-restarts.oerc.ox.ac.uk left intact
3/7/2014 2:55:11 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
3/7/2014 2:55:11 PM | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Info: Re-using existing connection! (#0) with host (nil)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Info: Connected to (nil) (129.67.195.121) port 80 (#0)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.28)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Host: cpdn-restarts.oerc.ox.ac.uk
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Accept: */*
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Accept-Encoding: deflate, gzip
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Content-Type: application/x-www-form-urlencoded
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Content-Length: 36868995
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server: Expect: 100-continue
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Sent header to server:
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Info: Re-using existing connection! (#1) with host (nil)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Info: Connected to (nil) (129.67.195.121) port 80 (#1)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.28)
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Host: cpdn-restarts.oerc.ox.ac.uk
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Accept: */*
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Accept-Encoding: deflate, gzip
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Content-Type: application/x-www-form-urlencoded
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Content-Length: 36730299
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server: Expect: 100-continue
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#867] Sent header to server:
3/7/2014 2:55:11 PM | climateprediction.net | [http] [ID#866] Received header from server: HTTP/1.1 100 Continue
3/7/2014 2:55:12 PM | climateprediction.net | [http] [ID#867] Received header from server: HTTP/1.1 100 Continue
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server: HTTP/1.1 200 OK
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server: Date: Fri, 07 Mar 2014 22:56:29 GMT
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server: Server: Apache
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server: Content-Length: 214
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server: Content-Type: text/plain
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Received header from server:
3/7/2014 3:00:42 PM | climateprediction.net | [http] [ID#867] Info: Connection #1 to host (nil) left intact
3/7/2014 3:00:43 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_l1cy_2013_1_008537313_0_13.zip: No such file or directory
3/7/2014 3:00:43 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_l1cy_2013_1_008537313_0_13.zip: transient upload error
3/7/2014 3:00:43 PM | climateprediction.net | Backing off 20 min 6 sec on upload of hadam3p_eu_l1cy_2013_1_008537313_0_13.zip
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server: HTTP/1.1 200 OK
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server: Date: Fri, 07 Mar 2014 22:56:29 GMT
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server: Server: Apache
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server: Content-Length: 214
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server: Content-Type: text/plain
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Received header from server:
3/7/2014 3:00:55 PM | climateprediction.net | [http] [ID#866] Info: Connection #0 to host (nil) left intact
3/7/2014 3:00:56 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_l1ct_2013_1_008537308_0_13.zip: No such file or directory
3/7/2014 3:00:56 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_l1ct_2013_1_008537308_0_13.zip: transient upload error
3/7/2014 3:00:56 PM | climateprediction.net | Backing off 33 min 18 sec on upload of hadam3p_eu_l1ct_2013_1_008537308_0_13.zip
3/7/2014 3:01:37 PM | | Suspending network activity - user request
[/quote]
|
|
Richard Haselgrove
Send message Joined: 1 Jan 07 Posts: 1066 Credit: 36,887,369 RAC: 1,533
|
3/7/2014 3:00:43 PM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_l1cy_2013_1_008537313_0_13.zip: No such file or directory
sounds like the old problem where the server managing the transfer has lost its connection with the server managing the long-term storage disks.
|
|
JIM
Send message Joined: 31 Dec 07 Posts: 1152 Credit: 22,363,583 RAC: 5,022
|
I have this problem also.
hadam3p_eu_h1hv_2013_1_008529381_0_13.zip is stuck in the transfer tab. It uploads seemingly normally until it reaches 100% and then hangs. It has failed to upload at least twice. Other zips are uploading normally.
Looks like it is time to remount the server again.
|
|
Lockleys
Send message Joined: 13 Jan 07 Posts: 195 Credit: 10,581,566 RAC: 0
|
Yes, me too. All _13 zips are apparently uploading to 100% and then failing,
e.g.
08/03/2014 03:47:46 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_f0ax_2013_1_008523769_0_13.zip: No such file or directory
08/03/2014 03:47:46 | climateprediction.net | Temporarily failed upload of hadam3p_eu_f0ax_2013_1_008523769_0_13.zip: transient upload error
08/03/2014 03:47:46 | climateprediction.net | Backing off 00:06:15 on upload of hadam3p_eu_f0ax_2013_1_008523769_0_13.zip
zips _1 to _12 are OK.
|
|
Professor Desty Nova

Send message Joined: 19 Sep 04 Posts: 92 Credit: 2,026,547 RAC: 446
|
Same here. It seems only the upload server of the final zip is having problems.
08/03/2014 00:35:27 | climateprediction.net | Started upload of hadam3p_eu_m05t_2013_1_008537808_0_12.zip
08/03/2014 00:45:40 | climateprediction.net | Started upload of hadam3p_eu_m05t_2013_1_008537808_0_13.zip
08/03/2014 00:45:43 | climateprediction.net | Computation for task hadam3p_eu_m05t_2013_1_008537808_0 finished
08/03/2014 00:59:37 | climateprediction.net | Finished upload of hadam3p_eu_m05t_2013_1_008537808_0_12.zip
08/03/2014 01:10:03 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_m05t_2013_1_008537808_0_13.zip: No such file or directory
08/03/2014 01:10:03 | climateprediction.net | Temporarily failed upload of hadam3p_eu_m05t_2013_1_008537808_0_13.zip: transient upload error
PS: And since this is the weekend, we probably won't see this solved until Monday :-(
PPS: Decided to suspend the other WU that was almost finished. One less upload to fail. My other WU end after Monday, so by that time the problem should be solved.
Professor Desty Nova
Researching Karma the Hard Way
|
|
old_user608497
Send message Joined: 31 Dec 09 Posts: 12 Credit: 17,214 RAC: 0
|
My first WU is due in 6 hours. I think I'll suspend it too before it crosses the finish line.
|
|
Jean-David Beyer
Send message Joined: 5 Aug 04 Posts: 1121 Credit: 17,202,915 RAC: 2,154
|
Me, too.
But notice some stuff gets throough...
08-Mar-2014 12:08:57 [climateprediction.net] [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_m0aq_2013_1_008537985_0_13.zip: No such file or directory
08-Mar-2014 12:08:57 [climateprediction.net] Temporarily failed upload of hadam3p_eu_m0aq_2013_1_008537985_0_13.zip: transient upload error
08-Mar-2014 12:08:57 [climateprediction.net] Backing off 2 min 43 sec on upload of hadam3p_eu_m0aq_2013_1_008537985_0_13.zip
08-Mar-2014 12:11:47 [climateprediction.net] Started upload of hadam3p_eu_m0aq_2013_1_008537985_0_13.zip
08-Mar-2014 12:11:47 [climateprediction.net] Started upload of hadam3p_eu_m0ap_2013_1_008537984_0_12.zip
08-Mar-2014 12:13:54 [climateprediction.net] [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_m0aq_2013_1_008537985_0_13.zip: No such file or directory
08-Mar-2014 12:13:54 [climateprediction.net] Temporarily failed upload of hadam3p_eu_m0aq_2013_1_008537985_0_13.zip: transient upload error
08-Mar-2014 12:13:54 [climateprediction.net] Backing off 9 min 16 sec on upload of hadam3p_eu_m0aq_2013_1_008537985_0_13.zip
08-Mar-2014 12:14:00 [climateprediction.net] Finished upload of hadam3p_eu_m0ap_2013_1_008537984_0_12.zip
|
|
JIM
Send message Joined: 31 Dec 07 Posts: 1152 Credit: 22,363,583 RAC: 5,022
|
Those are the usual messages that you get when the server at Oxford that excepts the _13 zip files needs to be “remounted”. Being this is the weekend it will most likely be Monday before the project people can get access to the server room.
|
|
Lockleys
Send message Joined: 13 Jan 07 Posts: 195 Credit: 10,581,566 RAC: 0
|
Me, too.
But notice some stuff gets throough...
Yes, the zips numbered _1 to _12 get through; it is only the final wrap-up _13 zip which is failing.
|
|
JIM
Send message Joined: 31 Dec 07 Posts: 1152 Credit: 22,363,583 RAC: 5,022
|
Me, too.
But notice some stuff gets throough...
Yes, the zips numbered _1 to _12 get through; it is only the final wrap-up _13 zip which is failing.
Zips 1 - 12 go to a different server, so they are unaffected. I am not sure if that server is even at the same physical location as the server the _13 zip goes to.
|
|
JIM
Send message Joined: 31 Dec 07 Posts: 1152 Credit: 22,363,583 RAC: 5,022
|
Zip file _13 uploaded successfully. Good work guys.
|
|
Les Bayliss Volunteer moderator
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0
|
There is a server failure, which can't be fixed remotely.
So, tomorrow then.
|
|
old_user608497
Send message Joined: 31 Dec 09 Posts: 12 Credit: 17,214 RAC: 0
|
Zip file _13 uploaded successfully.
It didn't work for me when I tried it out yesterday. So, are there any news regarding the server issue, is it fixed yet?
|
|
Dave Jackson Volunteer moderator
Send message Joined: 15 May 09 Posts: 4571 Credit: 19,039,635 RAC: 18,944
|
They are likely to be working on it by now.
My first WU is due in 6 hours. I think I'll suspend it too before it crosses the finish line.
If you have work in the queue, why not just suspend network activity?
|
|
old_user608497
Send message Joined: 31 Dec 09 Posts: 12 Credit: 17,214 RAC: 0
|
I didn't want to suspend network activity because this would've affected other projects as well. I don't think it's possible to switch it off exclusively for CPDN, at least I haven't found a way to do it. After the failed upload yesterday I suspended network activity anyway.
|
|
tullus
Send message Joined: 16 May 13 Posts: 48 Credit: 475,901 RAC: 0
|
Seems to be working for me. Did not work this morning (few hours ago), but uploaded successfully now.
|
|
Professor Desty Nova

Send message Joined: 19 Sep 04 Posts: 92 Credit: 2,026,547 RAC: 446
|
Just uploaded the one that was trying since Saturday.
Professor Desty Nova
Researching Karma the Hard Way
|
|
old_user608497
Send message Joined: 31 Dec 09 Posts: 12 Credit: 17,214 RAC: 0
|
Upload completed successfully. Thanks guys.
|
|
old_user608497
Send message Joined: 31 Dec 09 Posts: 12 Credit: 17,214 RAC: 0
|
Hm, while the final upload seems to be working fine, I'm having a problem with uploading _5.zip now. This is what I found in the event log:
10/03/2014 21:52:12 | climateprediction.net | Started upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip
10/03/2014 21:57:07 | climateprediction.net | Temporarily failed upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip: transient HTTP error
10/03/2014 21:57:07 | climateprediction.net | Backing off 00:03:48 on upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip
10/03/2014 21:57:08 | | Project communication failed: attempting access to reference site
10/03/2014 21:57:11 | | Internet access OK - project servers may be temporarily down.
OK, I've just seen that the upload server (cpdnupload2.oerc) status has been changed to not running.
10/03/2014 22:00:56 | climateprediction.net | Started upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip
10/03/2014 22:04:43 | climateprediction.net | Temporarily failed upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip: transient HTTP error
10/03/2014 22:04:43 | climateprediction.net | Backing off 00:07:00 on upload of hadam3p_eu_i1yj_2013_1_008551243_0_5.zip
10/03/2014 22:04:44 | | Project communication failed: attempting access to reference site
10/03/2014 22:04:47 | | Internet access OK - project servers may be temporarily down.
|
|
Les Bayliss Volunteer moderator
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0
|
It says "Temporarily".
And apparently the servers are getting a bad hammering from all of the computers trying to upload data.
It'll be bad for sometime because of this Weather@Home effort, that's attracted a large number of new people.
I notice that we're out of these new models. Don't know if there's going to be more, to allow for the ones being abandoned.
|
|