I have everything Suspended, and am in the Wait-And See-Zone.

Do we abort these WU's in the pipeline or do we wait and see? Will the server start again from halfway or have these WU's entered the Black Holes on the Internet? Over the years I have accumulated a lot of WU's which can best be described, they are in some Black Hole and be done with the matter. No record on my machines, no record on the server.

I have 7 tasks still running (all 216's which means they will go for a while yet) and 8 whose downloads are stuck. In the past when this has happened, once the server problems are resolved the tasks have downloaded and run normally. I certainly wouldn't abort while waiting for the servers to be sorted but I have known the occasional task get lost in the system somehow.

Three of the eight tasks that were stuck downloading have now finished downloading so that seems to be fixed. Another five minutes and I will now if the trickle server is running. (Server status page says not but as it only updates about every two hours it may not be accurate.)

Three of the eight tasks that were stuck downloading have now finished downloading so that seems to be fixed.

Four (not a typo) of the three tasks that were stuck downloading have now finished downloading, so that seems to be fixed. The fourth showed up while the other three were downloading. They downloaded very quickly. One (an N144 one) is already running. The boinc-client is running other tasks: WCG, Rosetta.

---

Life is back to normal. Downloading is in progress. Thanks be to Jupiter, King of Gods and Climate.

From Andy,

Hi All,

All services have been restored now to climateprediction.net infrastructure. The Department of Engineering IT Support decided to roll back the changes they made to the networking. This has allowed us to restore all the CPDN services.

Best wishes,

Andy

I won't post my opinion of the IT people who let the work go on so long after it was scheduled to finish and still didn't get it right.

From Andy,

Hi All,

All services have been restored now to climateprediction.net infrastructure. The Department of Engineering IT Support decided to roll back the changes they made to the networking. This has allowed us to restore all the CPDN services.

Best wishes,

Andy

I won't post my opinion of the IT people who let the work go on so long after it was scheduled to finish and still didn't get it right.

Download appears to have failed again.

Download appears to have failed again.

Just seen this. Friday afternoon not the best time to get things sorted in a hurry. The server appears to be running. I have a full cache currently and don't really want to up my cache to 10+10 days to get more. If anyone else is getting this problem I or another mod will let Andy know.

If anyone else is getting this problem I or another mod will let Andy know.

Fri 13 Aug 2021 07:40:57 AM EDT | climateprediction.net | Started download of ic_N144_200311_000057.nc.gz
Fri 13 Aug 2021 07:40:57 AM EDT | climateprediction.net | Started download of batch_911_a07e_sst.gz
Fri 13 Aug 2021 07:42:13 AM EDT |                       | Project communication failed: attempting access to reference site
Fri 13 Aug 2021 07:42:13 AM EDT | climateprediction.net | Temporarily failed download of ic_N144_200311_000057.nc.gz: transient HTTP error
Fri 13 Aug 2021 07:42:13 AM EDT | climateprediction.net | Backing off 00:11:32 on download of ic_N144_200311_000057.nc.gz
Fri 13 Aug 2021 07:42:16 AM EDT |                       | Internet access OK - project servers may be temporarily down.
Fri 13 Aug 2021 07:42:16 AM EDT | climateprediction.net | Temporarily failed download of batch_911_a07e_sst.gz: transient HTTP error
Fri 13 Aug 2021 07:42:16 AM EDT | climateprediction.net | Backing off 00:11:51 on download of batch_911_a07e_sst.gz
Fri 13 Aug 2021 09:13:53 AM EDT | climateprediction.net | Started download of hadam4_a1xx_201310_6_915_012102977.zip
Fri 13 Aug 2021 09:13:53 AM EDT | climateprediction.net | Started download of a1xx_915_atmos.gz
Fri 13 Aug 2021 09:15:12 AM EDT |                       | Project communication failed: attempting access to reference site
Fri 13 Aug 2021 09:15:12 AM EDT | climateprediction.net | Temporarily failed download of hadam4_a1xx_201310_6_915_012102977.zip: transient HTTP error
Fri 13 Aug 2021 09:15:12 AM EDT | climateprediction.net | Backing off 01:03:09 on download of hadam4_a1xx_201310_6_915_012102977.zip
Fri 13 Aug 2021 09:15:12 AM EDT | climateprediction.net | Temporarily failed download of a1xx_915_atmos.gz: transient HTTP error
Fri 13 Aug 2021 09:15:12 AM EDT | climateprediction.net | Backing off 00:56:30 on download of a1xx_915_atmos.gz
Fri 13 Aug 2021 09:15:14 AM EDT |                       | Internet access OK - project servers may be temporarily down.

---

Message sent.

Friday afternoon not the best time to get things sorted in a hurry.

How do they say clock-puncher in British?

Friday afternoon not the best time to get things sorted in a hurry.

How do they say clock-puncher in British?

They say: Patience.

Seems to me like they stopped everything except webservers

Yes, there's been a hardware failure at Oxford Uni, so Andy has shut down the project to prevent data loss.

Then, where are our trickles and uploads going? They are going somewhere.

Uploads don't go to Oxford.
They go to wherever on the planet those researchers are located.

The trickles?
Andy may have arranged "somewhere" for them to go to for the time being.

Today I am finding the message of trickle pending but no trace yet of upload of zip files pending.

I find that my trickles from Sunday 15August and now (here at UTC-5) Monday are all still on my boxen, but with the suffix '.sent'
Sent, but not acknowledged. And somehow hidden, had to go root to see em.
In other words, trickles are being bounced, and possibly diverted. No worries from here.
Hope this helps.

---

Andy has cast some powerful magic spell on all of the user computers. :)

It's 10.23am Monday there, so hopefully IT has managed to drag some workers in to fix whatever it is.

5.30pm and it's still down. :(
It's going to be a long week/month/year.

My slow running N216 has finally reached it's first zip/trickle, and the trickle_up file has been moved to the mail room ready to send. :)