Message boards :
Number crunching :
NZ25 file upload server problems?
Message board moderation
Author | Message |
---|---|
Send message Joined: 7 Aug 04 Posts: 2183 Credit: 64,822,615 RAC: 5,275 |
Anyone else getting file upload errors on these nz25 tasks? Mine uploads to 100% but then gives an error: 8/11/2022 5:52:48 PM | climateprediction.net | Started upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip 8/11/2022 5:58:10 PM | climateprediction.net | [checkpoint] result wah2_nz25_a141_199505_25_936_012151203_0 checkpointed 8/11/2022 6:06:07 PM | climateprediction.net | [checkpoint] result wah2_nz25_a141_199505_25_936_012151203_0 checkpointed 8/11/2022 6:08:26 PM | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 150376 8/11/2022 6:08:26 PM | climateprediction.net | Temporarily failed upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip: transient upload error 8/11/2022 6:08:26 PM | climateprediction.net | Backing off 01:12:41 on upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip This is reminiscent of previous problems with the ANZ model uploads that go to Tasmania/New Zealand servers. If others chime in with a problem, I'll notify Andy and Suzanne Rosier about it so someone can kick the server. |
Send message Joined: 7 Aug 04 Posts: 2183 Credit: 64,822,615 RAC: 5,275 |
Hmmm. On the 3rd retry, it finally went up so either they fixed it, or the problem is intermittent. |
Send message Joined: 12 Apr 21 Posts: 314 Credit: 14,559,045 RAC: 18,367 |
I'm having issues too but it does seem that eventually things upload as there are already 2 trickles uploaded, the third one is currently having upload issues though.. |
Send message Joined: 18 Feb 06 Posts: 73 Credit: 60,918,051 RAC: 45,912 |
I have suspended network activity for the moment. Wait and see. |
Send message Joined: 5 Aug 04 Posts: 1283 Credit: 15,824,334 RAC: 0 |
My task's third upload has had 7 failed attempts, the first starting at 0409 UTC, the last 20 minutes ago. "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
Also getting problems on my two testing site tasks from this project. Have left a message with extracts from log. If action in NZ needed, it may not get looked at for a few hours yet! |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
My uploads have started to upload again but still waiting to see if they finish. No idea if anyone has kicked server or not. Getting stuck at 100%. |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
Andy says the server appears to be up and he thinks it may be getting flooded with the number of machines trying to send zips. I will try again in the morning to see if anything has changed. |
Send message Joined: 5 Jun 09 Posts: 97 Credit: 3,673,031 RAC: 4,752 |
The arthritic snail that is uploads slides slowly up the sandpaper..... It's very patchy, sometimes a zip will go up up smoothly, but most of the time the snail is being very careful, if not stopped and soothing its painful foot. |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
I turned network activity off overnight and this morning, no data going through rather than getting to 100% and then giving the end of file error. Have turned it off again. |
Send message Joined: 1 Jan 07 Posts: 1058 Credit: 36,584,771 RAC: 15,932 |
Has anyone used a file manager to look at the actual output data file on their hard disk (it'll be in the CPDN project folder in their BOINC data tree). Does the byte count match the "asked for" number in the error message, the "got" number, or something else? It's possible - though unlikely - that a program or data error is mangling the file, and thus the error might be local, rather than in New Zealand. Best to eliminate the possibility, just to be on the safe side. |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
Has anyone used a file manager to look at the actual output data file on their hard disk (it'll be in the CPDN project folder in their BOINC data tree). Does the byte count match the "asked for" number in the error message, the "got" number, or something else? Just checked. On at least one of my testing site tasks it matches at 90,375,444bytes. That doesn't guarantee no problems elsewhere but given that George's task finished and uploaded OK from testing that suggests that it certainly isn't a universal problem. Also, I am now getting internet access OK, project servers may be down. |
Send message Joined: 1 Jan 07 Posts: 1058 Credit: 36,584,771 RAC: 15,932 |
Thanks - just checking. The next step would be http_debug (NOT xfer debug), to see what the server is actually telling you. |
Send message Joined: 9 Dec 05 Posts: 116 Credit: 12,528,032 RAC: 4,026 |
All of my uploads have gone thru, but the web site is missing all the trickles. The trickles were there on Thursday evening. |
Send message Joined: 5 Jun 09 Posts: 97 Credit: 3,673,031 RAC: 4,752 |
Richard - I just set http_debug.... "Instantly" all the tasks in transfer vanished from view, some of them reappeared about half a minute later. One thing I noticed was a number of blank lines - here's a sample @ 10:23:23: 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Server: Apache/2.4.7 (Ubuntu) 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Vary: Accept-Encoding 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Encoding: gzip 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Length: 84 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Type: text/plain 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: 13/08/2022 10:23:23 | climateprediction.net | 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info: Connection #57 to host upload4.cpdn.org left intact 13/08/2022 10:23:23 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [can pipeline] 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info: Re-using existing connection! (#57) with host upload4.cpdn.org 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info: Connected to upload4.cpdn.org (131.217.169.79) port 80 (#57) 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: Host: upload4.cpdn.org 13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20) Scrolling back in time to just before I set http_debug: 13/08/2022 10:13:36 | climateprediction.net | [sched_op] Starting scheduler request 13/08/2022 10:13:37 | climateprediction.net | Sending scheduler request: To send trickle-up message. 13/08/2022 10:13:37 | climateprediction.net | Not requesting tasks: don't need (CPU: job cache full; NVIDIA GPU: ) 13/08/2022 10:13:37 | climateprediction.net | [sched_op] CPU work request: 0.00 seconds; 0.00 devices 13/08/2022 10:13:37 | climateprediction.net | [sched_op] NVIDIA GPU work request: 0.00 seconds; 0.00 devices 13/08/2022 10:13:41 | climateprediction.net | Scheduler request completed 13/08/2022 10:13:41 | climateprediction.net | [sched_op] Server version 715 13/08/2022 10:13:41 | climateprediction.net | Project requested delay of 3636 seconds 13/08/2022 10:13:41 | climateprediction.net | [sched_op] Deferring communication for 01:00:36 13/08/2022 10:13:41 | climateprediction.net | [sched_op] Reason: requested by project 13/08/2022 10:13:45 | climateprediction.net | Started upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip 13/08/2022 10:18:53 | | Project communication failed: attempting access to reference site 13/08/2022 10:18:53 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip: transient HTTP error 13/08/2022 10:18:53 | climateprediction.net | Backing off 00:03:35 on upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip Then: 13/08/2022 10:20:09 | | Internet access OK - project servers may be temporarily down. 13/08/2022 10:20:37 | | Re-reading cc_config.xml 13/08/2022 10:20:37 | | Using proxy info from GUI 13/08/2022 10:20:37 | | Config: don't compute while Cities.exe is running 13/08/2022 10:20:37 | | Config: event log limit 20000 lines 13/08/2022 10:20:37 | | Config: use all coprocessors 13/08/2022 10:20:37 | | log flags: file_xfer, sched_ops, task, http_debug, sched_op_debug 13/08/2022 10:20:37 | DENIS@home | Found app_config.xml 13/08/2022 10:20:37 | Einstein@Home | Found app_config.xml 13/08/2022 10:20:37 | LHC@home | Found app_config.xml 13/08/2022 10:20:40 | | Re-reading cc_config.xml 13/08/2022 10:20:40 | | Using proxy info from GUI 13/08/2022 10:20:40 | | Config: don't compute while Cities.exe is running 13/08/2022 10:20:40 | | Config: event log limit 20000 lines 13/08/2022 10:20:40 | | Config: use all coprocessors 13/08/2022 10:20:40 | | log flags: file_xfer, sched_ops, task, http_debug, sched_op_debug 13/08/2022 10:20:40 | DENIS@home | Found app_config.xml 13/08/2022 10:20:40 | Einstein@Home | Found app_config.xml 13/08/2022 10:20:40 | LHC@home | Found app_config.xml Followed by a block like this (trying to upload the files that vanished from view?): 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt' 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1ga_199805_25_936_012151644_0_r2029078922_2.zip 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt' 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1ga_199805_25_936_012151644_0_r2029078922_3.zip 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt' 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_2.zip 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt' 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_2.zip 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt' 13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_2.zip And: 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#220] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#221] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#222] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#223] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#224] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#225] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] 13/08/2022 10:21:51 | climateprediction.net | [http] [ID#226] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [serially] Then a lad of lines like these: 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#219] Info: Trying 131.217.169.79... 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#220] Info: Hostname was found in DNS cache 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#220] Info: Trying 131.217.169.79... 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info: Hostname was found in DNS cache 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info: Trying 131.217.169.79... 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#222] Info: Hostname was found in DNS cache Then another block of lies like these: 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info: Connected to upload4.cpdn.org (131.217.169.79) port 80 (#56) 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Host: upload4.cpdn.org 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20) 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept: */* 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept-Encoding: deflate, gzip 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept-Language: en_GB 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Content-Length: 312 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Content-Type: application/x-www-form-urlencoded 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: poll_debug> 13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: <priority_debug>0</priority_debug> Loads more lines - the last few are: 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info: Connection #55 to host upload4.cpdn.org left intact 13/08/2022 10:32:18 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set 13/08/2022 10:32:18 | climateprediction.net | Finished upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_3.zip 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info: Found bundle for host upload4.cpdn.org: 0x43d7230 [can pipeline] 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info: Re-using existing connection! (#55) with host upload4.cpdn.org 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info: Connected to upload4.cpdn.org (131.217.169.79) port 80 (#55) 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Host: upload4.cpdn.org 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20) 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept: */* 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept-Encoding: deflate, gzip 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept-Language: en_GB 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Content-Length: 90401533 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Content-Type: application/x-www-form-urlencoded 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Expect: 100-continue 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: 13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Received header from server: HTTP/1.1 100 Continue 13/08/2022 10:37:36 | climateprediction.net | [http] [ID#228] Info: We are completely uploaded and fine I started with about 20 or so zip files to upload, now down to 7, plus one that's 100% progress, but in "upload: retry in 01:35:xx". Once the rest have cleared I might give that one a nudge.... Have fun (I should say I'm running BOINC version 7.16.20 on Windows 10 with Virtual box.) |
Send message Joined: 5 Jun 09 Posts: 97 Credit: 3,673,031 RAC: 4,752 |
Dialogue around a zip upload stalling: 13/08/2022 10:56:38 | climateprediction.net | 13/08/2022 10:56:38 | climateprediction.net | [http] [ID#257] Info: Connection #56 to host upload4.cpdn.org left intact 13/08/2022 10:56:39 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 159040 13/08/2022 10:56:39 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip: transient upload error 13/08/2022 10:56:39 | climateprediction.net | Backing off 00:02:05 on upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Info: We are completely uploaded and fine 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: HTTP/1.1 200 OK 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Date: Sat, 13 Aug 2022 09:46:57 GMT 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Server: Apache/2.4.7 (Ubuntu) 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Vary: Accept-Encoding 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Encoding: gzip 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Length: 123 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Type: text/plain 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: 13/08/2022 10:57:01 | climateprediction.net | Edit to add: A bit more arrived as I was typing: 13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Info: Connection #72 to host upload4.cpdn.org left intact 13/08/2022 10:57:02 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 157592 13/08/2022 10:57:02 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_4.zip: transient upload error 13/08/2022 10:57:02 | climateprediction.net | Backing off 00:04:34 on upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_4.zip |
Send message Joined: 1 Jan 07 Posts: 1058 Credit: 36,584,771 RAC: 15,932 |
13/08/2022 10:18:53 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip: transient HTTP errorThat's the one I was hoping to explore. The answer seems to be 13/08/2022 10:56:39 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 159040How big are your _4.zip files? The client seems to think it's sent it all, but neither 'asked for' nor 'got' match the numbers Dave posted earlier. I think the process goes like this - it should all be logged, though it's easier to see if you can retry a single file at a time: The client asks 'can I upload a file of 262144 bytes'? The server replies 'OK, go ahead' The client says 'here you are, then', and starts uploading what it finds on disk. Which is - ??? |
Send message Joined: 15 May 09 Posts: 4529 Credit: 18,663,251 RAC: 14,512 |
How big are your _4.zip files? The client seems to think it's sent it all, but neither 'asked for' nor 'got' match the numbers Dave posted earlier. The one I posted was a 5.zip from one of the main site batches. The messages from my testing branch tasks before I started getting servers problem messages have now dropped off the back of the log. Edit: now uploading again. will see what happens with the two uploads from testing and two from main site now running and check sizes of them if they fail Enabled http debug and the 25.zip from testing promptly succeeded. |
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
When the ANZ models first arrived years ago, the servers were at a big new data center in Hobart, the capitol of Tasmania. And the south east of Australia has been having severe weather with flooding, on and off all the year, including in Tasmania. The rain radar currently shows rain in the Hobart area. So server "flooding" could have two meanings. :( |
Send message Joined: 5 Jun 09 Posts: 97 Credit: 3,673,031 RAC: 4,752 |
BOINC reported all the files to be in the mid to high 80Mb. At the time of me seeing your comment I had one being uploaded, BOINC reported as ~86Mb, and was about the same when file was on the disc (needless to say it uploaded as I was looking at it and your message so I didn't catch the right sets of digits.....) |
©2024 cpdn.org