Stony666 wrote:

I have waited nearly two weeks until I post here because I had the hope that it could be fixed in a few days.
My hosts resides in Germany. They are in 3 different locations and providers.
On all boxes:
69972 climateprediction.net 11.01.2023 19:11:41 Temporarily failed upload of oifs_43r3_ps_0932_2013050100_123_982_12199576_0_r933785131_43.zip: transient HTTP error

The upload server is still up and functioning ok. There is an enormous backlog, so please be patient for a couple of days. I'm sure your uploads will happen soon.

wujj123456 wrote:

Best case: Server is saturated and we just need to be patient and wait for our turn.
Worse case: It was not the storage at first place and the actual issue is still unknown yet.

It was the storage. The cloud provider over-provisioned the disks and we suspect also put the OS on the same bare-metal as the data devices (slaps palm to forehead). But I didn't say that...

leloft wrote:

Doubly unlucky: I've just had the same refusal from both the first machine and now a second one, both refer to the same value 7168.00 MB. The good news is that one of the hosts has managed to upload 8 tasks.
I'll keep trying, but I'm limited by the 3636 seconds rule.

The 7168Mb tasks are fine. Perhaps the disk is just too full of results files at the moment and it will sort itself out in good time.

I am currently uploading with 32MBit/s. It's all my internet connection can provide. Location is germany.

I am currently uploading with 32MBit/s. It's all my internet connection can provide. Location is germany.

I am currently uploading with 6.3 MegaBytes/second. 50 Megabits/second. My internet connection can provide 75 megabits/second. Location near New York City.

Actual speeds are:

Timestamp 	     Download 	Upload 	   Latency  Jitter Quality Score Test Server
1/11/2023 14:35:27  76.28 Mbps  79.91 Mbps 5 ms     1 ms   Excellent     newyork02.speedtest.windstream.net

---

... both of my connections are pegged at 3Mbit up. No, that's not MB/s. That's Mbit. I'm hoping Starlink will clear up overnight, but it's gonna be a while.

FWIW, when I came home from work 3 hours ago it worked somewhat. It quickly worsened, and now _all_ transfer attempts fail. I've got the exact same situation now as the one @Stony666 described.

Glenn Carver wrote:

The upload server is still up and functioning ok. There is an enormous backlog, so please be patient for a couple of days. I'm sure your uploads will happen soon.

There is some good news and some bad news:

The good: Relative to the upload server outage (now almost three weeks), the backlog can't actually be very big. That's because many of us had to stop computing rather soon after the upload server outage started. (Everybody whose production is either constrained by Internet connection throughput, or by disk space.)

The bad: The fact that _nothing_ is moving anymore for myself and evidently for some others doesn't make me optimistic that the backlog (however modest or enormous it might be) would clear anytime soon. And as long as nothing is moving, folks as myself can't resume computation.

I’ve now cleared my backlog and files are still trickling up slowly - only trouble is that I’m generating new files just slightly more often than the link is clearing them :-)

Definitely progress as all of the outstanding WUs have gone and I can see my task list again.

All mine have uploaded this afternoon and new ZIP files are disappearing and not being held up.

Fix for - Need more disk space. You currently have 0.00 MB available.

In the BOINC Manager, Options -> Computing Preferences -> Disk and memory -

Check the box "Use no more than" and put a number in the number box equal to about 3/4 of your disk size (or some other number you are comfortable with).

If you leave it this box UNCHECKED, it is the same as having it checked with 100 (GB) in the number box.

At least that is how it works for me.

Good advice, thank you. I set 'use no more than' to 1000G, 'leave at least' to 1G free, 'use at most' to 99% of disk, updated project and now the two hosts in question processing new units.
2/4 hosts uploading as well.
Onwards and forwards.
fraser

FWIW, when I came home from work 3 hours ago it worked somewhat. It quickly worsened, and now _all_ transfer attempts fail. I've got the exact same situation now as the one @Stony666 described.

The bad: The fact that _nothing_ is moving anymore for myself and evidently for some others doesn't make me optimistic that the backlog (however modest or enormous it might be) would clear anytime soon. And as long as nothing is moving, folks as myself can't resume computation.

There may be some boinc-ness things going on. I vaguely remember Richard saying something about uploads will stop processing if it tries & fails 3 times? Or something like that? Uploads are still ok for me, I've got another 1000 to do.

Richard?

There may be some boinc-ness things going on. I vaguely remember Richard saying something about uploads will stop processing if it tries & fails 3 times? Or something like that? Uploads are still ok for me, I've got another 1000 to do.

I have vague memories of a limit on the number of tries and fails but it is a lot more than 3!

Edit: Cleared all from the VM. The others after a nudge have just started again.

Edit2: Something like 300 tasks an hour are being uploaded. (based on the reduction in tasks in progress. I didn't check tasks available to send which would increase it but not sure by how much. Will perhaps check that tomorrow.

Curious for folks with no problem connecting, are you able to ping upload11.cpdn.org?

I still couldn't connect, but given the latency, I more or less expect to be the last wave. I just don't want to keep retrying in BOINC since it has the tendency to upload partial files. If ping can give me the signal, that would be much easier for me to know when I might be able to upload.

Curious for folks with no problem connecting, are you able to ping upload11.cpdn.org?

I still couldn't connect, but given the latency, I more or less expect to be the last wave. I just don't want to keep retrying in BOINC since it has the tendency to upload partial files. If ping can give me the signal, that would be much easier for me to know when I might be able to upload.

I've got 10 concurrent uploads going (all day in fact), and if I try ping upload11.cpdn.org now it just hangs. So either it's too busy or they have disabled ping on the system for security.

traceroute does 30 hops and ends in (several tries, same result):

....
27  * * *
28  * * *
29  * * *
30  * * *

But both ping & traceroute were working fine for me yesterday evening when the server was up but httpd was not enabled on the machine. Probably load. Still, as you say, might be prudent just to wait a few more hours for the load to ease a bit.

p.s. I'm in Cambridge so Oxford's not too far...

Curious for folks with no problem connecting, are you able to ping upload11.cpdn.org?

I still couldn't connect, but given the latency, I more or less expect to be the last wave. I just don't want to keep retrying in BOINC since it has the tendency to upload partial files. If ping can give me the signal, that would be much easier for me to know when I might be able to upload.

I was able to upload my files a while ago, but atm a ping isn't getting through.
Trying from Hanover, Germany.

---

- - - - - - - - - -
Greetings, Jens

Thanks. Looks like ICMP packets are blocked then. Guess will just let boinc client handle the retry on its own. Given others are making progress, hopefully it won't take too long to be my turn.

I haven't seen any uploads in the last week either wujj123456.

I haven't seen any uploads in the last week either wujj123456.

FYI. The upload server only came back about 10:00GMT today (weds 11th).

I was able to upload all files, 100GB gone.

There may be some boinc-ness things going on. I vaguely remember Richard saying something about uploads will stop processing if it tries & fails 3 times? Or something like that? Uploads are still ok for me, I've got another 1000 to do.

Richard?

If three uploads in succession fail to get through, BOINC will go into 'Project backoff' for a stated period - starting at a few minutes, increasing to several hours. During this time, existing uploads will not be retried.

But any new upload will be tried just once. If that succeeds, the project backoff will be cleared, and existing uploads will be retried - until the next batch of failures.

This is purely a temporary thing - the user can over-ride it at any time, either by retrying a single transfer, or choosing 'retry pending transfers' from the tools menu in BOINC Manager.

climateprediction.net 12-01-2023 01:22 [error] Error reported by file upload server: Server is out of disk space

---

Supporting BOINC, a great concept !

climateprediction.net 12-01-2023 01:22 [error] Error reported by file upload server: Server is out of disk space

Yes, I've seen it and reported it back. I believe that's 27Tb filled then. The uploaded tasks should be moving off to a transfer server, maybe that's not working.