climateprediction.net (CPDN) home page
Thread 'Download issues'

Thread 'Download issues'

Message boards : Number crunching : Download issues
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68769 - Posted: 18 May 2023, 21:58:36 UTC - in response to Message 68768.  

Where'd you get that from? I've got 8 Windows and 1 Linux PC and I never get work.
I have some (old) Macs so I only get the Hadley CM3 jobs. These three were sitting in my Transfers tab, trying to be downloaded, for a month. The most previous work before that was from February. There's not a lot to go around from this project. I counted and my two Macs have been assigned 28 tasks since May 18th of last year.
I guess it's a good thing, the scientists (if the server was behaving!) get work done immediately. I'd much rather sign up to 20 projects and see bursts of work from each, instead of the scientists waiting years for a massive batch to get done. I guess we just need more power - I've just upgraded four machines to 96GB RAM to cater for LHC, but I'm still stuck on this pitiful 7Mbit uplink for 2 years because of my telecoms provider being useless. The whole town but three streets upgraded to 1000/220, and we're stuck on 32/7.
ID: 68769 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68770 - Posted: 19 May 2023, 3:24:47 UTC - in response to Message 68765.  

That is not the same issue. This issue in this thread is with the download server having a bad SSL cert. What you are showing is that the project has no tasks available.


That is one thing I thought I was showing: no complaints about bad SSL certificates.
The fact that there are no tasks available is true, but not really surprising.
ID: 68770 · Report as offensive     Reply Quote
bullschuck

Send message
Joined: 22 May 21
Posts: 39
Credit: 1,208,413
RAC: 3,997
Message 68772 - Posted: 19 May 2023, 21:15:38 UTC - in response to Message 68770.  

That is not the same issue. This issue in this thread is with the download server having a bad SSL cert. What you are showing is that the project has no tasks available.


That is one thing I thought I was showing: no complaints about bad SSL certificates.
The fact that there are no tasks available is true, but not really surprising.


Ouch, I misread that completely. I apologize for any and all snark in my response.
ID: 68772 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68773 - Posted: 19 May 2023, 21:28:23 UTC - in response to Message 68772.  

Ouch, I misread that completely. I apologize for any and all snark in my response.


No problem. Apology accepted.

Just a side-effect of non face-to-face live communication. It happens a lot.

It will probably happen again, knowing human nature -- including my own.
ID: 68773 · Report as offensive     Reply Quote
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68775 - Posted: 20 May 2023, 5:28:04 UTC - in response to Message 68773.  

Face to face doesn't make it any better. The English language is rubbish.
ID: 68775 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68776 - Posted: 20 May 2023, 5:45:08 UTC - in response to Message 68775.  
Last modified: 20 May 2023, 5:46:05 UTC

Face to face doesn't make it any better. The English language is rubbish.


The very reason face-to-face is superior to English..I think Latin is better than English. But face-to-face is better because in addition to the words and associated grammar, the affect communicates much more than the words do, and the immediacy permits errors to be detected sooner and corrections to be made much faster.
ID: 68776 · Report as offensive     Reply Quote
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68777 - Posted: 20 May 2023, 6:44:04 UTC - in response to Message 68776.  

Face to face doesn't make it any better. The English language is rubbish.
The very reason face-to-face is superior to English..I think Latin is better than English. But face-to-face is better because in addition to the words and associated grammar, the affect communicates much more than the words do, and the immediacy permits errors to be detected sooner and corrections to be made much faster.
Agreed, but using English as the base of communication even face to face, people have misunderstandings. What we need is a precise language, but without all the unnecessary nonsense, like distinguishing between less and fewer.
ID: 68777 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,729,836
RAC: 7,099
Message 68781 - Posted: 20 May 2023, 14:52:27 UTC - in response to Message 68780.  

If the program executable files can't be downloaded from the associated url, that's a bit more serious than the missing (task-specific) atmos.gz files that bullschuck reported earlier.

Program executable names and urls should be persistent in client_state.xml - we should be able to diagnose them further by trying to download them via a browser.
ID: 68781 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,729,836
RAC: 7,099
Message 68783 - Posted: 20 May 2023, 16:35:31 UTC - in response to Message 68782.  

It's possible that the new user already had the program files, retained after a previous job. If that's the case, there would be no need to attempt a fresh download.
ID: 68783 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68784 - Posted: 20 May 2023, 17:53:27 UTC - in response to Message 68783.  
Last modified: 20 May 2023, 17:56:43 UTC

It's possible that the new user already had the program files, retained after a previous job. If that's the case, there would be no need to attempt a fresh download.
Looks like that is the problem. I too have a failed download with

Sat 20 May 2023 17:58:27 BST | climateprediction.net | Scheduler request completed: got 1 new tasks
Sat 20 May 2023 17:58:27 BST | climateprediction.net | Project requested delay of 3636 seconds
Sat 20 May 2023 17:58:29 BST | climateprediction.net | Started download of hadam4_8.52_i686-pc-linux-gnu
Sat 20 May 2023 17:58:29 BST | climateprediction.net | Started download of hadam4_data_8.52_i686-pc-linux-gnu.zip
Sat 20 May 2023 17:58:31 BST | climateprediction.net | Giving up on download of hadam4_8.52_i686-pc-linux-gnu: permanent HTTP error
Sat 20 May 2023 17:58:31 BST | climateprediction.net | Giving up on download of hadam4_data_8.52_i686-pc-linux-gnu.zip: permanent HTTP error
Sat 20 May 2023 17:58:31 BST | climateprediction.net | Started download of hadam4_se_8.52_i686-pc-linux-gnu.zip
Sat 20 May 2023 17:58:31 BST | climateprediction.net | Started download of hadam4_um_8.52_i686-pc-linux-gnu.zip
Sat 20 May 2023 17:58:32 BST | climateprediction.net | Giving up on download of hadam4_se_8.52_i686-pc-linux-gnu.zip: permanent HTTP error
Sat 20 May 2023 17:58:32 BST | climateprediction.net | Giving up on download of hadam4_um_8.52_i686-pc-linux-gnu.zip: permanent HTTP error

I could probably copy the files across from somewhere else now I know but I wasn't at the computer when it happened. Because I did a clean install of XUbuntu 23.04 I don't have any of the executeables. I have let Andy know.
ID: 68784 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,729,836
RAC: 7,099
Message 68785 - Posted: 20 May 2023, 18:14:43 UTC

It also affects IFS executables (those are the only ones I've run on this incarnation of Linux).

client_state.xml gave me the download url as https://www.cpdn.org/applications//oifs_43r3_ps_1.05_x86_64-pc-linux-gnu, and the response was

Not Found

The requested URL /applications/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu was not found on this server.
I've added that to Dave's email to Andy.
ID: 68785 · Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68786 - Posted: 21 May 2023, 3:18:38 UTC - in response to Message 68762.  

The acid test will be when we start getting downloads across the board.

I do not know about across the board, but I notice I got a work-unit (a re-run) very recently. Boinc-manager is predicting a little over 8 days to complete it. It has run 46 minutes so far.

Machine ID: 1511241
Task        22318255
Name 	hadam4h_a05p_200011_5_931_012138767_2

ID: 68786 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68787 - Posted: 21 May 2023, 17:42:41 UTC

But first I need to work out how to copy files across from the guest to the host OS. Copying text works but not files so far. Am looking at various threads on intereweb about this!
ID: 68787 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68788 - Posted: 22 May 2023, 4:59:28 UTC
Last modified: 22 May 2023, 5:36:09 UTC

Hi Richard,

Thank you very much, this should be okay now.

Best wishes,

Andy


I can confirm the files download via a browser now.
Edit: Some do, there are still a couple of files not downloading for me.
ID: 68788 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,729,836
RAC: 7,099
Message 68789 - Posted: 22 May 2023, 7:37:46 UTC - in response to Message 68788.  

... there are still a couple of files not downloading for me.
In Dave's email to Andy he identified two distinct remaining problems.

1) Files which are still missing from the server - they can't be downloaded at all, either by BOINC or by a browser. There's nothing we can do about these - we just have to wait for Andy to find the files and put them back in the right place.

2) Files which have have been re-instated already, and can be downloaded by a browser - but not by BOINC.

This may be related to a bug I reported some time ago to BOINC. Under Linux, the client leaves behind a zero-length file when a download fails because the source file is missing from the server. This confuses subsequent attempts, because BOINC thinks the file is already present and skips the download - and of course the file fails subsequent integrity checks.

If you have full administrator access to the BOINC client folder on your machine, and you're still getting download errors other than 404 'file bot found', look for zero-length copies of the affected file in the CPDN project folder and delete them.

Reference: https://github.com/BOINC/boinc/issues/4642
ID: 68789 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68790 - Posted: 22 May 2023, 7:45:52 UTC - in response to Message 68789.  

This may be related to a bug I reported some time ago to BOINC. Under Linux, the client leaves behind a zero-length file when a download fails because the source file is missing from the server. This confuses subsequent attempts, because BOINC thinks the file is already present and skips the download - and of course the file fails subsequent integrity checks.
The one BOINC was failing to download, I had managed to find a copy but having put it into the climateprediction.net folder in projects, I had neglected to give it permission to run as a program. Would this cause the same problem as a zero byte size file? I suspect it may have been doing so but I will have to wait till another task comes along to check that having changed its permissions resolves the issue.
ID: 68790 · Report as offensive     Reply Quote
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68791 - Posted: 22 May 2023, 7:54:08 UTC - in response to Message 68790.  

Admin access, run as a program, I'm glad I use windows!
ID: 68791 · Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,729,836
RAC: 7,099
Message 68792 - Posted: 22 May 2023, 8:10:19 UTC - in response to Message 68790.  

No, not the 'same' problem, but it might well cause a different problem with the same end result - task failure.
ID: 68792 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68803 - Posted: 23 May 2023, 10:28:23 UTC - in response to Message 68792.  

Andy is saying it should be fixed now.
ID: 68803 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 68811 - Posted: 26 May 2023, 9:00:40 UTC - in response to Message 68803.  

Andy is saying it should be fixed now.
Fihnally got one of the resends and can confirm from my own experience that it is fixed.
ID: 68811 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Number crunching : Download issues

©2024 cpdn.org