My safr and sam zips have all cleared now as well. Hooray!

Thanks for the history on the cam25's. I found the one in question, and it has returned 18 zips.
https://www.cpdn.org/cpdnboinc/result.php?resultid=21709022

However, the ones that are stuck are #12 and #13. So it looks like they got lost in the shuffle.
If they have not uploaded by the time my other work has finished tomorrow, I will just can them
(as in trash can; I just realized that may not be clear to non-native English speakers).

What I've done with previous CAM25s that have persistently stuck uploads is just abort the transfers that are obviously stuck. After aborting the transfers, it will report the task, possibly as a success, and the scientists can determine whether the output is useful without the missing zips. I've only done this with the CAM25 tasks however since those are the ones that seem to occasionally have the rogue stuck uploads.

What I've done with previous CAM25s that have persistently stuck uploads is just abort the transfers that are obviously stuck. After aborting the transfers, it will report the task, possibly as a success, and the scientists can determine whether the output is useful without the missing zips.

It worked, and it is now listed as a success.
That is worthy of a FAQ, or sticky, or something.

What I've done with previous CAM25s that have persistently stuck uploads is just abort the transfers that are obviously stuck. After aborting the transfers, it will report the task, possibly as a success, and the scientists can determine whether the output is useful without the missing zips. I've only done this with the CAM25 tasks however since those are the ones that seem to occasionally have the rogue stuck uploads.

Good tip.

I reset the project which cleared everything, which is a bit extreme. But I had updated to WIN 10 Version 1903 and wanted a clean image file backup. All my other projects had already completed.

All is running OK now

---

Thanks for the history on the cam25's. I found the one in question, and it has returned 18 zips.
https://www.cpdn.org/cpdnboinc/result.php?resultid=21709022

However, the ones that are stuck are #12 and #13. So it looks like they got lost in the shuffle.
If they have not uploaded by the time my other work has finished tomorrow, I will just can them
(as in trash can; I just realized that may not be clear to non-native English speakers).

What I've done with previous CAM25s that have persistently stuck uploads is just abort the transfers that are obviously stuck. After aborting the transfers, it will report the task, possibly as a success, and the scientists can determine whether the output is useful without the missing zips. I've only done this with the CAM25 tasks however since those are the ones that seem to occasionally have the rogue stuck uploads.

This worked for me too. Trickle _15 was stuck at 51%. After reading this I checked and 18 trickles were uploaded according to the task stats. Aborted and it updated to successful.

I have lots of files to upload.

7/11/2019 4:56:59 PM | climateprediction.net | Backing off 00:06:55 on upload of wah2_sam50_n4ss_199112_24_822_011881375_0_r1252356358_9.zip
7/11/2019 4:57:01 PM | | [http] [ID#0] Info: Trying 172.217.164.100...
7/11/2019 4:57:01 PM | | [http] [ID#0] Info: Connected to www.google.com (172.217.164.100) port 443 (#12)
7/11/2019 4:57:01 PM | | [http] [ID#0] Info: ALPN, offering http/1.1
7/11/2019 4:57:01 PM | | [http] [ID#0] Info: Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: successfully set certificate verify locations:
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: CAfile: C:\Program Files\BOINC\ca-bundle.crt
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: CApath: none
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (OUT), TLS change cipher, Client hello (1):
7/11/2019 4:57:02 PM | | [http] [ID#0] Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS change cipher, Client hello (1):
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: TLSv1.2 (IN), TLS handshake, Finished (20):
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: ALPN, server accepted to use http/1.1
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: Server certificate:
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: start date: Jun 18 08:22:58 2019 GMT
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: expire date: Sep 10 08:15:00 2019 GMT
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: subjectAltName: www.google.com matched
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
7/11/2019 4:57:03 PM | | [http] [ID#0] Info: SSL certificate verify ok.
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: GET / HTTP/1.1
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: Host: www.google.com
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.14.2)
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: Accept: */*
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server: Accept-Language: en_US
7/11/2019 4:57:03 PM | | [http] [ID#0] Sent header to server:
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: HTTP/1.1 200 OK
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Date: Thu, 11 Jul 2019 23:57:04 GMT
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Expires: -1
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Cache-Control: private, max-age=0
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Content-Type: text/html; charset=ISO-8859-1
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Content-Encoding: gzip
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Server: gws
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: X-XSS-Protection: 0
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: X-Frame-Options: SAMEORIGIN
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Set-Cookie: 1P_JAR=2019-07-11-23; expires=Sat, 10-Aug-2019 23:57:04 GMT; path=/; domain=.google.com
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Set-Cookie: NID=187=Q_6Vo18HMPBsQwKMmbzWcDgsEpIa7bNHc4OLmSZNnC2BNok4sOzogAjg9X1tf-uCzASy2DVpw1P6_teABJqkfDiOin6CXzuG710cZrwgABiYMXs_Dx6SFINztkvuGidHQ4N510Tlea6Nx4r6KnWzO8l1IdfLY9VlMPUEq0BtZ8k; expires=Fri, 10-Jan-2020 23:57:04 GMT; path=/; domain=.google.com; HttpOnly
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: Transfer-Encoding: chunked
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server:
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 00000001
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 00000001
7/11/2019 4:57:04 PM | |
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 00000001
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 
7/11/2019 4:57:04 PM | | [http] [ID#0] Received header from server: 00000001
7/11/2019 4:57:04 PM | | [http_xfer] [ID#0] HTTP: wrote 744 bytes
7/11/2019 4:57:04 PM | | [http_xfer] [ID#0] HTTP: wrote 2884 bytes
7/11/2019 4:57:04 PM | | [http_xfer] [ID#0] HTTP: wrote 3579 bytes
7/11/2019 4:57:04 PM | | [http_xfer] [ID#0] HTTP: wrote 3693 bytes
7/11/2019 4:57:04 PM | | [http_xfer] [ID#0] HTTP: wrote 620 bytes
7/11/2019 4:57:04 PM | | [http] [ID#0] Info: Connection #12 to host www.google.com left intact
7/11/2019 4:57:04 PM | | Internet access OK - project servers may be temporarily down.

I have lots of files to upload.

Interesting, Three uploaded from my laptop (though only two to Jasmine) between midnight and one o clock UK time this morning which is when I have them set to go unless I want to look at things when they are uploading. Have just enabled internet access and another sam50 upload seems to be going through with no problems albeit slowly due to my internet connection.

Looks like a problem with Jasmin again.

20/07/2019 12:34:13 | climateprediction.net | Started upload of wah2_sam50_n30d_199312_24_822_011877856_0_r1583104809_14.zip
20/07/2019 12:34:13 | climateprediction.net | [file_xfer] URL: http://jasmin-upload.cpdn.org/cgi-bin/file_upload_handler
20/07/2019 12:34:13 | climateprediction.net | [fxd] starting upload, upload_offset -1
20/07/2019 12:34:13 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
20/07/2019 12:34:13 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
20/07/2019 12:34:13 | climateprediction.net | Started upload of wah2_sam50_n30d_199312_24_822_011877856_0_r1583104809_16.zip
20/07/2019 12:34:13 | climateprediction.net | [file_xfer] URL: http://jasmin-upload.cpdn.org/cgi-bin/file_upload_handler
20/07/2019 12:34:13 | climateprediction.net | [http] [ID#122] Info: Found bundle for host jasmin-upload.cpdn.org: 0x28f2f90 [serially]
20/07/2019 12:34:13 | climateprediction.net | [http] [ID#121] Info: Trying 192.171.139.103...
20/07/2019 12:34:13 | climateprediction.net | [http] [ID#122] Info: Hostname was found in DNS cache
20/07/2019 12:34:13 | climateprediction.net | [http] [ID#122] Info: Trying 192.171.139.103...
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#121] Info: connect to 192.171.139.103 port 80 failed: Connection refused
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#121] Info: Failed to connect to jasmin-upload.cpdn.org port 80: Connection refused
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#121] Info: Closing connection 364
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#122] Info: connect to 192.171.139.103 port 80 failed: Connection refused
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#122] Info: Failed to connect to jasmin-upload.cpdn.org port 80: Connection refused
20/07/2019 12:34:15 | climateprediction.net | [http] [ID#122] Info: Closing connection 365
20/07/2019 12:34:15 | climateprediction.net | [http] HTTP error: Couldn't connect to server
20/07/2019 12:34:15 | climateprediction.net | [http] HTTP error: Couldn't connect to server

My SAM50 tasks from two batches also built-up a sizeable backlog. They are in process of uploading now. Hope yours are, too.

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

Yes. Backlog cleared.

What message do you get in the event log. Enable file transfer debug in the event log if there is nothing informative there. I see 15 trickle up messages have gone through so credits will still be granted but more information is needed to see if this is likely to be a major problem or not.

Edit: Just noticed this is a cam25. I think the zips go to a server in Mexico and in the past that server has had problems so will probably go when the server problems resolved again. Still be good to get the event log messages so Oxford can tell them what is going on.

I get:

26.07.2019 08:23:54 | climateprediction.net | Started upload of wah2_cam25_a0bw_200405_18_691_011369977_1_r502703358_1.zip
26.07.2019 08:23:57 | climateprediction.net | Temporarily failed upload of wah2_cam25_a0bw_200405_18_691_011369977_1_r502703358_1.zip: can't resolve hostname
26.07.2019 08:23:57 | climateprediction.net | Backing off 04:39:40 on upload of wah2_cam25_a0bw_200405_18_691_011369977_1_r502703358_1.zip
26.07.2019 08:24:02 | climateprediction.net | Sending scheduler request: To fetch work.
26.07.2019 08:24:02 | climateprediction.net | Requesting new tasks for CPU
26.07.2019 08:24:05 | climateprediction.net | Scheduler request completed: got 0 new tasks
26.07.2019 08:24:05 | climateprediction.net | Project has no tasks available
26.07.2019 08:24:23 | | Project communication failed: attempting access to reference site
26.07.2019 08:24:26 | | Internet access OK - project servers may be temporarily down.

---

OK will send email.

Mexico time is about 4.30am at the moment so any email from Oxford to them won't even be read for a few hours.

OK will send email.

Mexico time is about 4.30am at the moment so any email from Oxford to them won't even be read for a few hours.

Hi Dave,


Thanks, this is accessible again now.


Best regards,

Andy

If it doesn't work let me know but nothing will happen till Monday at the earliest as unless they are working in their own time, there isn't anyone at Oxford to read the emails.

Thanks Dave. Most but not all zips uploaded due to access lost again:

One of the other mods had some but not all go and has notified Oxford. He thought it could be just the server getting hammered but if that was the case I would have expected them all to have gone by now.

So the backlog of the sam50 (Jasmin) server is now resolved? .. I still get transient http error. So it seems Climateprediction.net cannot handle a proxy server anymore?

Or should i just keep trying?

Seems like one file succeded in uploading though..

2019-07-30 09:35:18 | climateprediction.net | Temporarily failed upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_restart.zip: transient HTTP error
2019-07-30 09:35:18 | climateprediction.net | Backing off 00:02:34 on upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_restart.zip
2019-07-30 09:35:18 | climateprediction.net | Started upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_24.zip
2019-07-30 09:35:19 | | Internet access OK - project servers may be temporarily down.
2019-07-30 09:35:21 | | Project communication failed: attempting access to reference site
2019-07-30 09:35:21 | climateprediction.net | Temporarily failed upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_24.zip: transient HTTP error
2019-07-30 09:35:21 | climateprediction.net | Backing off 00:02:57 on upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_24.zip
2019-07-30 09:35:21 | climateprediction.net | Started upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_out.zip
2019-07-30 09:35:22 | | Internet access OK - project servers may be temporarily down.
2019-07-30 09:35:23 | climateprediction.net | Finished upload of wah2_sam50_n3ei_201412_24_808_011812012_0_r1312120817_out.zip
2019-07-30 09:35:24 | climateprediction.net | Started upload of wah2_anz50_a0k6_201612_20_793_011761372_1_r261502111_2.zip
2019-07-30 09:35:28 | | Project communication failed: attempting access to reference site
2019-07-30 09:35:28 | climateprediction.net | Temporarily failed upload of wah2_anz50_a0k6_201612_20_793_011761372_1_r261502111_2.zip: transient HTTP error

I will move my boinc installation to another computer that does not need a proxy server in between to see if the behaviour changes since i have had issues a long time now... :)

Ok, so now on a new computer.. But still same issue.. Good now atleast i know its not the proxy server :)

So i guess the project still have issues with transient HTTP errors. :(

OK, update:

The issue is one of bandwidth onto the system. We are constantly running 146 upload processes at the moment and have been since I announced on the board that we were restarting jasmin. This is one of the issues around backing up when not running.
We are investigating how we can setup another upload server within the infrastructure that can be run as part of a load balancing pair behind the jasmin-upload name.


David

Out of curiosity.. How much bandwidth does this server / or connection have available?

Is there some load graph available to view? :) I wanted to know if there is any hour during the day where it might be easier to upload results (since im manually uploading).