Bunch of files uploaded then it started getting http error at about 15:30. 15:50, started working again then more errors from 17:10 onwards. May just be the server being overloaded though.

For me, I got two batches. The second batch had a few skips, but usually a minute or less.

10:13:32 to 10:22:37 EST so add 5 hours for UTC
11:40:17 to 12:36:02 EST so add 5 hours for UTC

Since then, many tries but no successes.

---

Bunch of files uploaded then it started getting http error at about 15:30. 15:50, started working again then more errors from 17:10 onwards. May just be the server being overloaded though.

This posted on projects message board with folowing reply from Andy:

I am afraid you are right. It's lost both it's SSH and HTTP ports again. I have sent another email to Matt in JASMIN Support to ask him to investigate this again tomorrow morning.

Open file count hitting limits and it can't create new sockets?

If he could get serial console access to it, that should be independent of SSH connections, to investigate and see what's going on. Or remote syslog to something he's got access to. Troubleshooting when it "just locks up" is brutal. :( I've also seen that sort of behavior from running a machine very badly out of RAM. I've no idea how much processing is happening on the upload box vs later on (zips being extracted, etc), but with the amount of uploads flowing in, any sort of "post-processing" tasks could easily thrash the machine into oblivion - the fun of recovering from non-steady-state operation is that you now have to be able to handle all the traffic queued up.

A bunch of my machines refuse to download new tasks as there are too many uploads in progress - not sure if there's a way around it.

traceroute eventually makes its way to the proper destination.

It doesn't for me.
Actually, I think it does because it seems to fail the same way it is doing for others. I just wonder why traceroute fails like this Or is DNS failing somewhere along the line?

localhost:jeandavid8[~]$ date
Tue Jan  3 21:17:56 EST 2023
localhost:jeandavid8[~]$ traceroute upload11.cpdn.org
traceroute to upload11.cpdn.org (192.171.169.187), 30 hops max, 60 byte packets
 1  Fios_Quantum_Gateway.fios-router.home (192.168.0.1)  0.317 ms  0.454 ms  0.590 ms
 2  lo0-100.NWRKNJ-VFTTP-309.verizon-gni.net (71.127.205.1)  5.251 ms  8.144 ms  8.196 ms
 3  at-0-0-0-1716.ALT2-CORE-RTR1.verizon-gni.net (100.41.5.68)  8.258 ms at-0-0-0-1717.ALT2-CORE-RTR2.verizon-gni.net (100.41.5.70)  10.218 ms  8.314 ms
 4  0.csi1.NBWKNJNB-MSE01-BB-SU1.ALTER.NET (140.222.4.106)  10.335 ms 0.csi1.NWRKNJ02-MSE01-BB-SU1.ALTER.NET (140.222.4.104)  13.201 ms 0.csi1.NBWKNJNB-MSE01-BB-SU1.ALTER.NET (140.222.4.106)  12.735 ms
 5  * * *
 6  * * *
 7  * * *
 8  nyk-bb2-link.ip.twelve99.net (62.115.135.162)  8.699 ms * *
 9  ldn-bb1-link.ip.twelve99.net (62.115.113.21)  77.718 ms  77.820 ms  77.458 ms
10  ldn-b2-link.ip.twelve99.net (62.115.122.189)  79.894 ms ldn-b2-link.ip.twelve99.net (62.115.120.239)  77.523 ms  74.854 ms
11  jisc-ic345131-ldn-b2.ip.twelve99-cust.net (62.115.175.131)  77.402 ms  77.977 ms  75.352 ms
12  ae24.londhx-sbr1.ja.net (146.97.35.197)  77.833 ms  77.979 ms  77.380 ms
13  ae29.londpg-sbr2.ja.net (146.97.33.2)  97.289 ms  95.016 ms  89.015 ms
14  ae31.erdiss-sbr2.ja.net (146.97.33.22)  80.207 ms  80.350 ms  82.997 ms
15  * * *
16  ral-r26.ja.net (146.97.41.34)  79.900 ms  81.510 ms  84.112 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
localhost:jeandavid8[~]$ 

---

My traceroute:

saenger@Sanger-sein-Mint:~$ traceroute upload11.cpdn.org
traceroute to upload11.cpdn.org (192.171.169.187), 30 hops max, 60 byte packets
 1  fritz.box (192.168.178.1)  0.546 ms  0.803 ms  1.021 ms
 2  85.16.121.109 (85.16.121.109)  15.733 ms  15.779 ms  15.824 ms
 3  mprt-hb-70740110-xe-2-2-0.ewe-ip-backbone.de (85.16.250.59)  15.886 ms  15.922 ms  15.946 ms
 4  mprt-hb-70740101-xe-1-2-1.ewe-ip-backbone.de (85.16.250.61)  15.960 ms  15.986 ms  16.010 ms
 5  bbrt-hb-1-70730203-ae15.ewe-ip-backbone.de (85.16.249.103)  25.920 ms  25.982 ms  26.031 ms
 6  bbrt-hb-2-70730201-ae24.ewe-ip-backbone.de (80.228.90.30)  19.972 ms  5.301 ms  14.076 ms
 7  et-0-0-53.edge1.Hamburg1.Level3.net (62.67.25.41)  14.100 ms  14.124 ms  14.147 ms
 8  ae2.3201.ear1.London1.level3.net (4.69.141.66)  24.178 ms  25.188 ms  25.958 ms
 9  JANET.ear1.London1.Level3.net (212.187.216.254)  26.599 ms  27.495 ms  28.196 ms
10  ae24.londhx-sbr1.ja.net (146.97.35.197)  29.310 ms  30.612 ms  20.051 ms
11  ae29.londpg-sbr2.ja.net (146.97.33.2)  28.368 ms  28.394 ms  28.418 ms
12  ae31.erdiss-sbr2.ja.net (146.97.33.22)  24.889 ms  26.494 ms  26.519 ms
13  * * *
14  ral-r26.ja.net (146.97.41.34)  30.549 ms  31.887 ms  32.385 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
saenger@Sanger-sein-Mint:~$ 

And some messages from BOINC:

Mi 04 Jan 2023 06:30:41 CET | climateprediction.net | [file_xfer] http op done; retval -184 (transient HTTP error)
Mi 04 Jan 2023 06:30:41 CET | climateprediction.net | [file_xfer] file transfer status -184 (transient HTTP error)
Mi 04 Jan 2023 06:30:41 CET | climateprediction.net | Temporarily failed upload of oifs_43r3_ps_0219_1999050100_123_968_12184863_0_r1453880002_3.zip: transient HTTP error
Mi 04 Jan 2023 06:30:41 CET | climateprediction.net | Backing off 00:02:59 on upload of oifs_43r3_ps_0219_1999050100_123_968_12184863_0_r1453880002_3.zip
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 2147 bytes
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 3223 bytes
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 3456 bytes
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 3401 bytes
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 589 bytes
Mi 04 Jan 2023 06:30:41 CET |  | [http_xfer] [ID#0] HTTP: wrote 1031 bytes
Mi 04 Jan 2023 06:30:42 CET |  | Internet access OK - project servers may be temporarily down.

Some make it, but I fail to see any pattern, why they manage to get away.

---

Grüße vom Sänger

And some messages from BOINC:

I would recommend that you set http_debug, rather than http_xfer_debug. The output gives more detail about the reasons for errors.

From Andy

Update received from JASMIN Support this morning:

We believe that there is still a problem with the block storage device. We are migrating VMs off that device while we investigate the problem. I'll let you know when we have migrated the VMs.

I'll let you know when we have migrated the VMs.

This involves transfer of several TB of data so may be a while. This is also why we have gone back to "connect failed" from transient http error.

Joy! 2 uploads are now going. Doesn't really make any difference that more aren't as my upload bandwidth is saturated.
Edit: my configured maximum of four are now going.

And now the messages flying about are how to keep it running!

And some messages from BOINC:

I would recommend that you set http_debug, rather than http_xfer_debug. The output gives more detail about the reasons for errors.

I did so, but I can't post any errors any more, because all is running fine once I hit the "Try again" once more after returning from work.
Whatever you did: Thanks!

---

Grüße vom Sänger

Its looking hopeful....... one just uploaded. 400+ to go.

They sure have enough bandwidth. I am watching the transfers with my system monitor and I estimate I am uploading at a rate of 5 Megabytes/second and have hit 11 Megabytes per second from time-to-time. I hope it keeps up.

Edit 1: I spoke too soon. Transfers stopped, retry in about 10 minutes. It tried again and failed to transfer any more. It is now resting for about another half hour.

Edit 2: It tried again, but failed. Backing off another hour.

---

I am still getting transient HTTP error. Noting has uploaded yet.

I am still getting transient HTTP error. Noting has uploaded yet.

I have six files uploading currently, 4 on host machine and 2 on vm, the maximum the respective installations of BOINC are configured for. I susupect the transient errors at the moment are just down to the server getting hammered as several TB worth of files try and upload. I don't know what the maximum number of connections the server can cope with at once is?

Seems like each upload opens up a new connection, which isn't ideal... Then usually the folks with long RTT gets screwed more until some bandwidth frees up on the server. I just hope the server stays up so once European friends catches up mostly, we on other continents can have our uploads too...

I've got at least 4 connections worth streaming up at the moment. I've noticed that it takes a while for them to get going, but once they're going, things seem to continue flowing more or less smoothly. Or, at least, as smoothly as they can flow through space (all my stuff is going through Starlink right now). I've got enough sun to light up a few more boxes and they'll be uploading on a terrestrial link, hopefully...

The admins should plan for enough infrastructure to handle:
* "Computers with recent credit" as per the server status page. Right now, that number is 968 computers.
* With project backoff near 1 hour, that means: 16 uploads per minute average
* Total file size 224MB each model, means server needs to handle 3.5G per minute during peak time.

The admins should plan for enough infrastructure to handle:
* "Computers with recent credit" as per the server status page. Right now, that number is 968 computers.
* With project backoff near 1 hour, that means: 16 uploads per minute average
* Total file size 224MB each model, means server needs to handle 3.5G per minute during peak time.

They did plan accordingly, they've been doing this for a long time. The upload server has about 25Tb of storage, completed tasks are processed and then move on to another big 'transfer' server. It can easily handle the load. Yesterday when it was up they were processing 50,000 requests per hour. CPDN was let down by the infrastructure on JASMIN, or perhaps their support advice. I'll find out more at their next technical meeting.

In readiness for that technical meeting, it might be worth having a peek at https://github.com/BOINC/boinc/issues/139. That's a conversation initiated by CPDN volunteers some 16 years ago, as the original source https://boinc.berkeley.edu/trac/ticket/139 makes clear.

Uploads, and servers filling up or failing, were a problem then. too. It became apparent then, and again this last fortnight, that volunteers have very little in the way of controls which could ease that recovery. Basically, uploads are either "off" or "on", for all projects together in that boinc instance. About the only tunable parameter is

<max_file_xfers_per_project>N</max_file_xfers_per_project>
Maximum number of simultaneous file transfers per project (default 2).

And it's not enough.