climateprediction.net (CPDN) home page
Thread 'New work discussion - 2'

Thread 'New work discussion - 2'

Message boards : Number crunching : New work discussion - 2
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · 24 . . . 42 · Next

AuthorMessage
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68798 - Posted: 23 May 2023, 4:43:08 UTC - in response to Message 68797.  

I do not know anything about dropbox.

I have these hadam4 ones..

The first number is the size in bytes. The date is probably when it was put onto my machine.

-rwxr-xr-x. 1 boinc boinc 3634637 Dec 18  2021 hadam4_8.09_i686-pc-linux-gnu
-rwxr-xr-x. 1 boinc boinc 3635629 Dec 18  2021 hadam4_8.52_i686-pc-linux-gnu
-rwxr-xr-x. 1 boinc boinc   75539 Dec 18  2021 hadam4_data_8.09_i686-pc-linux-gnu.zip
-rwxr-xr-x. 1 boinc boinc   75539 Dec 18  2021 hadam4_data_8.52_i686-pc-linux-gnu.zip

-rwxrwxr-x. 1 boinc boinc 3736644 May  1  2019 hadam4_se_8.09_i686-pc-linux-gnu.so
-rwxr-xr-x. 1 boinc boinc 1166623 Dec 18  2021 hadam4_se_8.09_i686-pc-linux-gnu.zip
-rwxrwxr-x. 1 boinc boinc 3736644 May  1  2019 hadam4_se_8.52_i686-pc-linux-gnu.so
-rwxr-xr-x. 1 boinc boinc 1166623 Dec 18  2021 hadam4_se_8.52_i686-pc-linux-gnu.zip
-rwxrwxr-x. 1 boinc boinc 6574073 May  1  2019 hadam4_um_8.09_i686-pc-linux-gnu
-rwxr-xr-x. 1 boinc boinc 2723666 Dec 18  2021 hadam4_um_8.09_i686-pc-linux-gnu.zip
-rwxrwxr-x. 1 boinc boinc 6574073 May  1  2019 hadam4_um_8.52_i686-pc-linux-gnu
-rwxr-xr-x. 1 boinc boinc 2725214 Dec 18  2021 hadam4_um_8.52_i686-pc-linux-gnu.zip
I suggested dropbox as an easy place for you to dump the files so I could get them.

They're actually really small, perhaps you would be kind enough to email me them? peter@hucker.plus.com - that address takes 100MB files easily.

Mainly the four mine was requesting, but the others too if you like:

hadam4_8.09_i686-pc-linux-gnu
hadam4_data_8.09_i686-pc-linux-gnu.zip
hadam4_se_8.09_i686-pc-linux-gnu.zip
hadam4_um_8.09_i686-pc-linux-gnu.zip
ID: 68798 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68799 - Posted: 23 May 2023, 7:49:24 UTC

They're actually really small, perhaps you would be kind enough to email me them?


Having your email visible on forums like this is a good way to get spam. Do you want your message hidden after a couple of days or if I see you have got the files if that happens sooner.

When you have put the files into the correct directory assuming you get them, right click on each, go to properties then permissions and tick allow to run as a program. That gets set automagically when downloaded and placed there by BOINC. That doesn't happen when you place them there yourself.
ID: 68799 · Report as offensive
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68800 - Posted: 23 May 2023, 9:51:33 UTC - in response to Message 68799.  

They're actually really small, perhaps you would be kind enough to email me them?
Having your email visible on forums like this is a good way to get spam. Do you want your message hidden after a couple of days or if I see you have got the files if that happens sooner.
It's ok, that email address I've had for a couple of decades and it already gets 50 spams a day because I've used it everywhere, but my ISP has a very good filter.

When you have put the files into the correct directory assuming you get them,
I haven't got them yet. Can you send them? Can I get them using a browser or is that just as problematic?

right click on each, go to properties then permissions and tick allow to run as a program. That gets set automagically when downloaded and placed there by BOINC. That doesn't happen when you place them there yourself.
Thanks, I'd forgotten about that, since I mainly use Windows which doesn't have that [polite mode engaged] oddity.
ID: 68800 · Report as offensive
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68801 - Posted: 23 May 2023, 10:25:34 UTC

Actually, it just downloaded them successfully within Boinc. Andy must have sorted something.
ID: 68801 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68802 - Posted: 23 May 2023, 10:27:55 UTC - in response to Message 68800.  

I don't have them at the moment due to having done a clean install of ubuntu. However, Andy tells me that it should be fixed now. It should be possible to get them by pasting the link from the event or pasting the file names into the link Richard gave in the thread adjacent to this or wait till the next time BOINC tries to download a task and it should get them.
ID: 68802 · Report as offensive
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68805 - Posted: 23 May 2023, 11:39:50 UTC - in response to Message 68801.  

Actually, it just downloaded them successfully within Boinc. Andy must have sorted something.


... so you do not need me to e-mail them to you?
ID: 68805 · Report as offensive
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68807 - Posted: 24 May 2023, 10:51:07 UTC - in response to Message 68805.  

Actually, it just downloaded them successfully within Boinc. Andy must have sorted something.
... so you do not need me to e-mail them to you?
Correct, thanks for offering. I now actually have a task running!
ID: 68807 · Report as offensive
Mr. P Hucker

Send message
Joined: 9 Oct 20
Posts: 690
Credit: 4,391,754
RAC: 6,918
Message 68808 - Posted: 24 May 2023, 12:02:55 UTC

Then discover Virtualbox defaulted to ONE (?!) CPU out of the 24 available, as I'd downgraded VB to suit Cosmology and had to set it up again (although using the same image, so not setting Linux up again, just the VB settings, which I didn't bother going through). Oracle really is Brahmin excrement.
ID: 68808 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68810 - Posted: 25 May 2023, 13:33:34 UTC

No idea what the hold up on the Windows tasks is. The scientist asked for the next stage of producing them to be done a couple of days after I completed my four spin up tasks in testing. It may be the person who usually does it is on leave at the moment. I haven't seen any activity from her on the message boards acknowledging the request. Or it may be she has other work that is more important?
ID: 68810 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68845 - Posted: 5 Jun 2023, 11:01:53 UTC

I have posted a question asking when the East Asia Windows tasks are going to appear.
ID: 68845 · Report as offensive
nairb

Send message
Joined: 3 Sep 04
Posts: 105
Credit: 5,646,090
RAC: 102,785
Message 68847 - Posted: 5 Jun 2023, 21:24:26 UTC

Are all the future w/u's going to be statically linked.... Hope so.
ID: 68847 · Report as offensive
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68848 - Posted: 5 Jun 2023, 22:01:18 UTC - in response to Message 68847.  

Are all the future w/u's going to be statically linked.... Hope so.


Will that even be possible? I know in Linux, different distributions use different methods to resolve network security certificate procedures. To include all that networking stuff in a statically-linked module would be quite a trick. In Windows, that I do not know well, it might be possible.
ID: 68848 · Report as offensive
nairb

Send message
Joined: 3 Sep 04
Posts: 105
Credit: 5,646,090
RAC: 102,785
Message 68849 - Posted: 5 Jun 2023, 23:06:52 UTC - in response to Message 68848.  

I meant w/u's not needing those 32 bit libs, We had some of those earlier. Worked well on fedora 36 without installing those 32 bit libs.
ID: 68849 · Report as offensive
Jean-David Beyer

Send message
Joined: 5 Aug 04
Posts: 1120
Credit: 17,202,915
RAC: 2,154
Message 68850 - Posted: 6 Jun 2023, 0:01:27 UTC - in response to Message 68849.  

I meant w/u's not needing those 32 bit libs, We had some of those earlier. Worked well on fedora 36 without installing those 32 bit libs.

Well, these do not need 32-bit libraries since they are 64-bit programs. I do not know if they needed to be statically linked, but they are.
oifs_43r3_1.21_x86_64-pc-linux-gnu:            ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped

oifs_43r3_bl_1.11_x86_64-pc-linux-gnu:         ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped

oifs_43r3_ps_1.09_x86_64-pc-linux-gnu:         ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped


On the other hand, these are dynamically linked and presumably need 32-bit compatibility libraries.
hadam4_8.09_i686-pc-linux-gnu:           ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=4f76ff9a4376f0271470f6b5b25d9c6edf07b390, with debug_info, not stripped
hadam4_8.52_i686-pc-linux-gnu:           ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=ece375f9d653f056ef582ea3ec922334255cd7d4, with debug_info, not stripped

hadam4_se_8.09_i686-pc-linux-gnu.so:     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=b188b7ead1a6d631bc73b96e884867d1f414e9bb, with debug_info, not stripped

hadam4_se_8.52_i686-pc-linux-gnu.so:     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=b188b7ead1a6d631bc73b96e884867d1f414e9bb, with debug_info, not stripped
1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=39fb1eff23e50e7ecadbb9f23a9562b1deb9e6bb, with debug_info, not stripped

hadam4_um_8.52_i686-pc-linux-gnu:        ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=f8399277dbd2683528c8461d9d449f04c2de9932, with debug_info, not stripped
V), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=aa1442951dfabe184d184c6f7038b253276190dc, with debug_info, not stripped

hadcm3s_se_8.36_i686-pc-linux-gnu.so:    ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, with debug_info, not stripped

hadcm3s_um_8.36_i686-pc-linux-gnu:       ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=d27b7f7cfd67d5822b59259cfcafed319ee6c0cf, with debug_info, not stripped

hadsm4_8.02_i686-pc-linux-gnu:           ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=916745402e986b1155c409b64c7dd600d297f47d, with debug_info, not stripped

hadsm4_se_8.02_i686-pc-linux-gnu.so:     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=aa80731a515b872ca8876fe6c761cbc6bdf2a714, with debug_info, not stripped

hadsm4_um_8.02_i686-pc-linux-gnu:        ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=fad83a9dd4aafeb3c94c7ea4f506f6e51c4dd8eb, with debug_info, not stripped


All the above work on Red Hat Enterprise Linux release 8.7 (Ootpa)
ID: 68850 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68852 - Posted: 6 Jun 2023, 7:07:18 UTC
Last modified: 6 Jun 2023, 8:50:30 UTC

My objection to having the libraries statically linked and downloaded afresh with each task is the extra bandwidth needed. (I don't know if the files could be provided for just a single download the way the files specific to each task type are. One problem is I think that different distributions require different versions of these files. I am pretty sure someone asked the project about his a few years back. Because the CPDN license on the Hadley model code doesn't allow them to mess about with it I think was the sticking point as having the files downloaded by BOINC as without messing about with the code, it would be looking for the libraries in the wrong place.
ID: 68852 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68853 - Posted: 6 Jun 2023, 8:56:15 UTC

The researcher for the EAS tasks has discussed the results with her professor and there were some concerns about the spin up results but they have decided everything is within range and the mainsite tasks will be released, "very soon."
ID: 68853 · Report as offensive
Richard Haselgrove

Send message
Joined: 1 Jan 07
Posts: 1061
Credit: 36,699,995
RAC: 9,937
Message 68854 - Posted: 6 Jun 2023, 8:58:37 UTC - in response to Message 68852.  

My objection to having the libraries statically linked and downloaded afresh with each task is the extra bandwidth needed.
I think you're being a little over-pessimistic there. Programs and libraries would normally be defined at the "application version" level. That would have the effect you describe during development and testing, but once a stable production version has been chosen, there should be no further need to download the libraries - certainly not for each succeeding task of the same type.

Obviously, applications need data to work on, and that will change with every task. There might be scope for splitting the data up into two types: a basic climate model framework (which would be mostly unchanging), and a variable set of perturbations or parameters to define each separate task. The Einstein project has lots of experience of this system for their Gravity Wave application: users download a large initial group of recorded data files from the Hanford and Livingstone detectors, and keep it for potentially hundreds of tasks until all possible searches have been completed. Their server keeps track of what data each computer has downloaded, and tries to minimise downloads by reusing it for as long as possible, before finally deleting it and moving the machine on to a new block. I'm sure Einstein would help with the initial setup, but that's probably a discussion for another day.
ID: 68854 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68855 - Posted: 6 Jun 2023, 9:47:52 UTC - in response to Message 68854.  

I am sure you are right Richard. Currently suffering from Covid (first time I ttested positive was yesterday so not thinking things through fully.) The bit about the Met office code not being open source and the licence not allowing CPDN to tinker with it to look for the libraries in the right place is I think the real issue.
ID: 68855 · Report as offensive
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,432,494
RAC: 17,331
Message 68856 - Posted: 6 Jun 2023, 10:11:38 UTC - in response to Message 68855.  

I am sure you are right Richard. Currently suffering from Covid (first time I ttested positive was yesterday so not thinking things through fully.) The bit about the Met office code not being open source and the licence not allowing CPDN to tinker with it to look for the libraries in the right place is I think the real issue.
CPDN have a license to inspect the code, otherwise they can't compile it. And they/we do tinker with it (have to - to get it working on newer linuxes sometimes).

An advantage of static linking is less reliance on libraries installed on the client machines. Given the wide variety of operating systems, that's an advantage. It's why I compile OpenIFS to be static. It's why some applications use virtual machines or containers if they need custom libraries. The difference in size of the binary is not significant.

The 32bit issue could be solved if someone could spend a month or two at CPDN getting the models to compile into 64bit. But they only have Andy and he's swamped already, and I think they would rather spend more development time getting the latest MetO model into CPDN than mess about with the old one. That would add more value for them.

Data used by these models could be split into task variant and invariant though, that's true. Though the resolutions we are running at are small and the effort is probably not worth it unless we go to higher resolutions.
---
CPDN Visiting Scientist
ID: 68856 · Report as offensive
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4536
Credit: 18,999,047
RAC: 21,571
Message 68858 - Posted: 6 Jun 2023, 10:49:39 UTC - in response to Message 68856.  

Thanks Glen. It makes a big difference having someone on the inside who actually knows what is happening.
ID: 68858 · Report as offensive
Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · 24 . . . 42 · Next

Message boards : Number crunching : New work discussion - 2

©2024 cpdn.org