Anyone else getting file upload errors on these nz25 tasks? Mine uploads to 100% but then gives an error:

8/11/2022 5:52:48 PM | climateprediction.net | Started upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip
8/11/2022 5:58:10 PM | climateprediction.net | [checkpoint] result wah2_nz25_a141_199505_25_936_012151203_0 checkpointed
8/11/2022 6:06:07 PM | climateprediction.net | [checkpoint] result wah2_nz25_a141_199505_25_936_012151203_0 checkpointed
8/11/2022 6:08:26 PM | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 150376
8/11/2022 6:08:26 PM | climateprediction.net | Temporarily failed upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip: transient upload error
8/11/2022 6:08:26 PM | climateprediction.net | Backing off 01:12:41 on upload of wah2_nz25_a141_199505_25_936_012151203_0_r55943290_1.zip

This is reminiscent of previous problems with the ANZ model uploads that go to Tasmania/New Zealand servers. If others chime in with a problem, I'll notify Andy and Suzanne Rosier about it so someone can kick the server.

Hmmm. On the 3rd retry, it finally went up so either they fixed it, or the problem is intermittent.

I'm having issues too but it does seem that eventually things upload as there are already 2 trickles uploaded, the third one is currently having upload issues though..

I have suspended network activity for the moment. Wait and see.

---

My task's third upload has had 7 failed attempts, the first starting at 0409 UTC, the last 20 minutes ago.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

Also getting problems on my two testing site tasks from this project. Have left a message with extracts from log. If action in NZ needed, it may not get looked at for a few hours yet!

My uploads have started to upload again but still waiting to see if they finish. No idea if anyone has kicked server or not.

Getting stuck at 100%.

Andy says the server appears to be up and he thinks it may be getting flooded with the number of machines trying to send zips. I will try again in the morning to see if anything has changed.

The arthritic snail that is uploads slides slowly up the sandpaper.....
It's very patchy, sometimes a zip will go up up smoothly, but most of the time the snail is being very careful, if not stopped and soothing its painful foot.

I turned network activity off overnight and this morning, no data going through rather than getting to 100% and then giving the end of file error. Have turned it off again.

Has anyone used a file manager to look at the actual output data file on their hard disk (it'll be in the CPDN project folder in their BOINC data tree). Does the byte count match the "asked for" number in the error message, the "got" number, or something else?

It's possible - though unlikely - that a program or data error is mangling the file, and thus the error might be local, rather than in New Zealand. Best to eliminate the possibility, just to be on the safe side.

Has anyone used a file manager to look at the actual output data file on their hard disk (it'll be in the CPDN project folder in their BOINC data tree). Does the byte count match the "asked for" number in the error message, the "got" number, or something else?

Just checked. On at least one of my testing site tasks it matches at 90,375,444bytes. That doesn't guarantee no problems elsewhere but given that George's task finished and uploaded OK from testing that suggests that it certainly isn't a universal problem. Also, I am now getting internet access OK, project servers may be down.

Thanks - just checking. The next step would be http_debug (NOT xfer debug), to see what the server is actually telling you.

All of my uploads have gone thru, but the web site is missing all the trickles. The trickles were there on Thursday evening.

---

Richard - I just set http_debug....
"Instantly" all the tasks in transfer vanished from view, some of them reappeared about half a minute later.
One thing I noticed was a number of blank lines - here's a sample @ 10:23:23:

13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Server: Apache/2.4.7 (Ubuntu)
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Vary: Accept-Encoding
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Encoding: gzip
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Length: 84
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server: Content-Type: text/plain
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Received header from server:
13/08/2022 10:23:23 | climateprediction.net | 
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info:  Connection #57 to host upload4.cpdn.org left intact
13/08/2022 10:23:23 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [can pipeline]
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info:  Re-using existing connection! (#57) with host upload4.cpdn.org
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Info:  Connected to upload4.cpdn.org (131.217.169.79) port 80 (#57)
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: Host: upload4.cpdn.org
13/08/2022 10:23:23 | climateprediction.net | [http] [ID#228] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20)

Scrolling back in time to just before I set http_debug:

13/08/2022 10:13:36 | climateprediction.net | [sched_op] Starting scheduler request
13/08/2022 10:13:37 | climateprediction.net | Sending scheduler request: To send trickle-up message.
13/08/2022 10:13:37 | climateprediction.net | Not requesting tasks: don't need (CPU: job cache full; NVIDIA GPU: )
13/08/2022 10:13:37 | climateprediction.net | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
13/08/2022 10:13:37 | climateprediction.net | [sched_op] NVIDIA GPU work request: 0.00 seconds; 0.00 devices
13/08/2022 10:13:41 | climateprediction.net | Scheduler request completed
13/08/2022 10:13:41 | climateprediction.net | [sched_op] Server version 715
13/08/2022 10:13:41 | climateprediction.net | Project requested delay of 3636 seconds
13/08/2022 10:13:41 | climateprediction.net | [sched_op] Deferring communication for 01:00:36
13/08/2022 10:13:41 | climateprediction.net | [sched_op] Reason: requested by project
13/08/2022 10:13:45 | climateprediction.net | Started upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip
13/08/2022 10:18:53 |  | Project communication failed: attempting access to reference site
13/08/2022 10:18:53 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip: transient HTTP error
13/08/2022 10:18:53 | climateprediction.net | Backing off 00:03:35 on upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip

Then:

13/08/2022 10:20:09 |  | Internet access OK - project servers may be temporarily down.
13/08/2022 10:20:37 |  | Re-reading cc_config.xml
13/08/2022 10:20:37 |  | Using proxy info from GUI
13/08/2022 10:20:37 |  | Config: don't compute while Cities.exe is running
13/08/2022 10:20:37 |  | Config: event log limit 20000 lines
13/08/2022 10:20:37 |  | Config: use all coprocessors
13/08/2022 10:20:37 |  | log flags: file_xfer, sched_ops, task, http_debug, sched_op_debug
13/08/2022 10:20:37 | DENIS@home | Found app_config.xml
13/08/2022 10:20:37 | Einstein@Home | Found app_config.xml
13/08/2022 10:20:37 | LHC@home | Found app_config.xml
13/08/2022 10:20:40 |  | Re-reading cc_config.xml
13/08/2022 10:20:40 |  | Using proxy info from GUI
13/08/2022 10:20:40 |  | Config: don't compute while Cities.exe is running
13/08/2022 10:20:40 |  | Config: event log limit 20000 lines
13/08/2022 10:20:40 |  | Config: use all coprocessors
13/08/2022 10:20:40 |  | log flags: file_xfer, sched_ops, task, http_debug, sched_op_debug
13/08/2022 10:20:40 | DENIS@home | Found app_config.xml
13/08/2022 10:20:40 | Einstein@Home | Found app_config.xml
13/08/2022 10:20:40 | LHC@home | Found app_config.xml

Followed by a block like this (trying to upload the files that vanished from view?):

13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1ga_199805_25_936_012151644_0_r2029078922_2.zip
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1ga_199805_25_936_012151644_0_r2029078922_3.zip
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_2.zip
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_2.zip
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
13/08/2022 10:21:51 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:21:51 | climateprediction.net | Started upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_2.zip

And:

13/08/2022 10:21:51 | climateprediction.net | [http] [ID#220] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#221] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#222] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#223] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#224] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#225] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]
13/08/2022 10:21:51 | climateprediction.net | [http] [ID#226] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [serially]

Then a lad of lines like these:

13/08/2022 10:21:52 | climateprediction.net | [http] [ID#219] Info:    Trying 131.217.169.79...
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#220] Info:  Hostname was found in DNS cache
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#220] Info:    Trying 131.217.169.79...
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info:  Hostname was found in DNS cache
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info:    Trying 131.217.169.79...
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#222] Info:  Hostname was found in DNS cache

Then another block of lies like these:

13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Info:  Connected to upload4.cpdn.org (131.217.169.79) port 80 (#56)
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Host: upload4.cpdn.org
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20)
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept: */*
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept-Encoding: deflate, gzip
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Accept-Language: en_GB
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Content-Length: 312
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: Content-Type: application/x-www-form-urlencoded
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server:
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server: poll_debug>
13/08/2022 10:21:52 | climateprediction.net | [http] [ID#221] Sent header to server:         <priority_debug>0</priority_debug>

Loads more lines - the last few are:

13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info:  Connection #55 to host upload4.cpdn.org left intact
13/08/2022 10:32:18 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
13/08/2022 10:32:18 | climateprediction.net | Finished upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_3.zip
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info:  Found bundle for host upload4.cpdn.org: 0x43d7230 [can pipeline]
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info:  Re-using existing connection! (#55) with host upload4.cpdn.org
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Info:  Connected to upload4.cpdn.org (131.217.169.79) port 80 (#55)
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: POST /cgi-bin/file_upload_handler HTTP/1.1
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Host: upload4.cpdn.org
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.16.20)
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept: */*
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept-Encoding: deflate, gzip
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Accept-Language: en_GB
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Content-Length: 90401533
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Content-Type: application/x-www-form-urlencoded
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server: Expect: 100-continue
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Sent header to server:
13/08/2022 10:32:18 | climateprediction.net | [http] [ID#246] Received header from server: HTTP/1.1 100 Continue
13/08/2022 10:37:36 | climateprediction.net | [http] [ID#228] Info:  We are completely uploaded and fine

I started with about 20 or so zip files to upload, now down to 7, plus one that's 100% progress, but in "upload: retry in 01:35:xx". Once the rest have cleared I might give that one a nudge....
Have fun
(I should say I'm running BOINC version 7.16.20 on Windows 10 with Virtual box.)

Dialogue around a zip upload stalling:

13/08/2022 10:56:38 | climateprediction.net | 
13/08/2022 10:56:38 | climateprediction.net | [http] [ID#257] Info:  Connection #56 to host upload4.cpdn.org left intact
13/08/2022 10:56:39 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 159040
13/08/2022 10:56:39 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip: transient upload error
13/08/2022 10:56:39 | climateprediction.net | Backing off 00:02:05 on upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Info:  We are completely uploaded and fine
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: HTTP/1.1 200 OK
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Date: Sat, 13 Aug 2022 09:46:57 GMT
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Server: Apache/2.4.7 (Ubuntu)
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Vary: Accept-Encoding
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Encoding: gzip
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Length: 123
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server: Content-Type: text/plain
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Received header from server:
13/08/2022 10:57:01 | climateprediction.net |
 

Edit to add:
A bit more arrived as I was typing:

13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Info:  Connection #72 to host upload4.cpdn.org left intact
13/08/2022 10:57:02 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 157592
13/08/2022 10:57:02 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_4.zip: transient upload error
13/08/2022 10:57:02 | climateprediction.net | Backing off 00:04:34 on upload of wah2_nz25_a1k4_199905_25_936_012151782_0_r706971930_4.zip

13/08/2022 10:18:53 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1i4_199805_25_936_012151710_0_r2018106213_4.zip: transient HTTP error

That's the one I was hoping to explore. The answer seems to be

13/08/2022 10:56:39 | climateprediction.net | [error] Error reported by file upload server: EOF on socket read : asked for 262144, got 159040
13/08/2022 10:56:39 | climateprediction.net | Temporarily failed upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip: transient upload error
13/08/2022 10:56:39 | climateprediction.net | Backing off 00:02:05 on upload of wah2_nz25_a1g2_199805_25_936_012151636_0_r1401480091_4.zip
13/08/2022 10:57:01 | climateprediction.net | [http] [ID#265] Info: We are completely uploaded and fine

How big are your _4.zip files? The client seems to think it's sent it all, but neither 'asked for' nor 'got' match the numbers Dave posted earlier.

I think the process goes like this - it should all be logged, though it's easier to see if you can retry a single file at a time:

The client asks 'can I upload a file of 262144 bytes'?
The server replies 'OK, go ahead'
The client says 'here you are, then', and starts uploading what it finds on disk.

Which is - ???

How big are your _4.zip files? The client seems to think it's sent it all, but neither 'asked for' nor 'got' match the numbers Dave posted earlier.

The one I posted was a 5.zip from one of the main site batches. The messages from my testing branch tasks before I started getting servers problem messages have now dropped off the back of the log.

Edit: now uploading again. will see what happens with the two uploads from testing and two from main site now running and check sizes of them if they fail

Enabled http debug and the 25.zip from testing promptly succeeded.

When the ANZ models first arrived years ago, the servers were at a big new data center in Hobart, the capitol of Tasmania.
And the south east of Australia has been having severe weather with flooding, on and off all the year, including in Tasmania.
The rain radar currently shows rain in the Hobart area.

So server "flooding" could have two meanings. :(

BOINC reported all the files to be in the mid to high 80Mb.

At the time of me seeing your comment I had one being uploaded, BOINC reported as ~86Mb, and was about the same when file was on the disc (needless to say it uploaded as I was looking at it and your message so I didn't catch the right sets of digits.....)