climateprediction.net (CPDN) home page
Thread 'Big models'

Thread 'Big models'

Message boards : Number crunching : Big models
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,022,240
RAC: 20,762
Message 63069 - Posted: 2 Dec 2020, 16:19:10 UTC - in response to Message 63067.  

All very interesting, but have you been able to run more than one at once?
Is there any obvious slowdown due to it?

No obvious slowdown running two at once but pretty sure there would be a slow down running more than four at once as my Ryzen only has 32GB of RAM. The latest batch I ran only had 4 tasks so wasn't able to do a lot of checking out especially as one crashed due to fpops_est being too small so to get others to finish I had to edit this value in client_state.xml

When some of these first came out and required about 5GB of RAM each, I was able to run four on my laptop which only has 8GB RAM on it and they slowed down dramatically compared with running one but throughput did keep increasing up to running the maximum of four.
Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.
ID: 63069 · Report as offensive     Reply Quote
KAMasud

Send message
Joined: 6 Oct 06
Posts: 204
Credit: 7,608,986
RAC: 0
Message 63075 - Posted: 3 Dec 2020, 1:35:27 UTC - in response to Message 62637.  

Survey time

Some testing going on, but DON'T get too excited yet.
(Linux at present.)

Question: How do people feel about a monthly upload of around 193Mb?

--------------------
I would love to have some but these do not sound like Windows.
ID: 63075 · Report as offensive     Reply Quote
KAMasud

Send message
Joined: 6 Oct 06
Posts: 204
Credit: 7,608,986
RAC: 0
Message 63076 - Posted: 3 Dec 2020, 1:35:28 UTC - in response to Message 62637.  
Last modified: 3 Dec 2020, 1:37:46 UTC

Sorry, somehow a double post.
ID: 63076 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 63077 - Posted: 3 Dec 2020, 1:46:44 UTC - in response to Message 63075.  
Last modified: 3 Dec 2020, 2:33:43 UTC

Question: How do people feel about a monthly upload of around 193Mb?

I think you mean MB?
No problem for me. My cable modem is rated at 15 Mb/sec upload, but in practice does 20 Mb/sec (I just measured it).
That is 2.5 MB/sec, so 193 MB would take a little over a minute.

EDIT: And maybe you mean 193 GB? That would be 21.4 hours for me, still OK for a whole month. Not a problem.
But anyone with less than maybe 2 Mb/sec uploads should not attempt it. It would probably interfere with other things.

Probably anyone on a cable modem or optical fiber is OK, but that is not everyone by any means.
I think they should just issue them with some big warnings, and see what happens.
ID: 63077 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 63078 - Posted: 3 Dec 2020, 1:53:20 UTC - in response to Message 63069.  

No obvious slowdown running two at once but pretty sure there would be a slow down running more than four at once as my Ryzen only has 32GB of RAM.
OK, thanks. Maybe it is just main memory limiting it, and not cache. The only way to know is to try.

I can add 32 GB more to bring my i7-9700 up to 64 GB, by adding two my 16 GB modules.
I had a bit of a stability problem, but think it is probably because the memory modules run at 1.2 volts by default.
Bringing that up to 1.3 volts or so should fix it. It seemed to be right on the edge of stability anyway.
ID: 63078 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 63463 - Posted: 2 Feb 2021, 17:41:34 UTC

Ya know... rather than making each model so big (multiple GB per task), could the programmers simply have each task share more files in common? That way each model takes less space, if that makes sense.
ID: 63463 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,022,240
RAC: 20,762
Message 63464 - Posted: 2 Feb 2021, 18:21:15 UTC - in response to Message 63463.  

Ya know... rather than making each model so big (multiple GB per task), could the programmers simply have each task share more files in common? That way each model takes less space, if that makes sense.


Much greater chance of errors that way. Also the program that all the current tasks use is around a million lines of Fortran before being compiled used under a license from the Met office that doesn't allow a lot of mucking about with it. Big files are inevitable with climate modelling because of the amount of data being processed. That means this project is not going to be for all.
Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.
ID: 63464 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 63469 - Posted: 2 Feb 2021, 19:32:22 UTC - in response to Message 63463.  

The models DO share data files.
There are data files for land use schemes, vegetation, and lots more.

Where the data is getting bigger, is in the results produced, which are unique to each model.

Early in the project, the models were about researching the effects on a suburban block.
Now, with more powerful computers available, and better models, it's about researching each front and back yard of each block separately.

**************

“My dear, here we must run as fast as we can, just to stay in place. And if you wish to go anywhere you must run twice as fast as that.”

― The Queen of Hearts, Alice in Wonderland
ID: 63469 · Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : Number crunching : Big models

©2024 cpdn.org