climateprediction.net (CPDN) home page
Thread 'Size of FAMOUS on HD'

Thread 'Size of FAMOUS on HD'

Message boards : Number crunching : Size of FAMOUS on HD
Message board moderation

To post messages, you must log in.

AuthorMessage
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39361 - Posted: 26 Mar 2010, 3:29:28 UTC - in response to Message 39319.  

What are the disk space requirements for one FAMOUS model? I have two running right now and they\'ve already used 2.5GB!
ID: 39361 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 39362 - Posted: 26 Mar 2010, 6:06:38 UTC

Moved to it\'s own thread.

I\'m running 4 of them on the beta site, and they\'re taking up 900Megs.
I\'ve just uploaded a about 50 trickles (9K each), plus 7 zips, (4.5Megs each), and with these, plus the files from several test versions, the folder was 980Megs.

This was measured using Windows Properties on the beta project folder.

So:
Q1 At which point are you measuring 2.5Gigs?
Q2 If at the same point, what else is in the projects folder?

ID: 39362 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 39364 - Posted: 26 Mar 2010, 6:55:07 UTC - in response to Message 39362.  
Last modified: 26 Mar 2010, 6:56:06 UTC

Q1 At which point are you measuring 2.5Gigs?
Q2 If at the same point, what else is in the projects folder?

Q3 Have you enabled graphics recording to disk? If you have the model will be creating a 73kb .cpdn file in its tmp directory for every timestep. If that\'s the case you\'ll see the line \"recording...\" above the model time line in the graphics.

Recording is enabled and disabled using <CTRL>Q and .cpdn files can be deleted without affecting the model.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 39364 · Report as offensive     Reply Quote
ProfileJIM

Send message
Joined: 31 Dec 07
Posts: 1152
Credit: 22,363,583
RAC: 5,022
Message 39365 - Posted: 26 Mar 2010, 7:34:33 UTC

Hi, Everyone:

Backup size doesn’t seem excessive to me. A backup made today, containing 4 Famous models (1 very early stage and 3 “waiting to run”) plus 1 very late stage AM3P WU and the backup took up only about 800 MB’s. This is not excessively large. I am used to 2 late stage AM3Ps plus thier replacements taking up about 1.2 GBs.

ID: 39365 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39370 - Posted: 26 Mar 2010, 12:07:53 UTC

[root@jasper boinc]# du -h -s projects/climateprediction.net/
1.8G projects/climateprediction.net/

And that\'s after one model failed overnight and reported already. So, I only have one model using 1.8G right now.
ID: 39370 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39371 - Posted: 26 Mar 2010, 14:10:05 UTC - in response to Message 39364.  
Last modified: 26 Mar 2010, 14:13:49 UTC

Q1 Climate folder, see my previous post
Q2 Not sure what to look for?
Q3 No. I never look at the graphics because BOINC runs under a protected account.

There is a folder with the task\'s name that contains a large number of 16MB files.

Tasks generating all these files are:
11390186
11390195
ID: 39371 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39375 - Posted: 26 Mar 2010, 19:50:25 UTC
Last modified: 26 Mar 2010, 19:50:55 UTC

Found this in the stdout file. What does it mean?

dlopen error: /var/lib/boinc/projects/climateprediction.net/famous_se_6.01_i686-pc-linux-gnu.so: cannot restore segment prot after reloc: Permission denied

Other climate models have fun just fine for a long time.
ID: 39375 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 39377 - Posted: 26 Mar 2010, 21:38:15 UTC

Q2 Not sure what to look for?

In this case, models that are no longer being crunched, but which have a folder in the climateprediction.net folder.
Which would be the 2 listed in your 2nd last post:
The 1st one failed, and the 2nd one was aborted.

The stderr out list for both of them contain messages about disk problems.
A) HD close to failure?
B) Another running program using the HD at the same time as the climate model wanted it?

In any case, if those 2 that you listed are no longer in the Tasks tab, then delete the folders.

As for your latest post, this may also be due to HD contention.


Backups: Here
ID: 39377 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39378 - Posted: 26 Mar 2010, 22:04:50 UTC - in response to Message 39377.  

If you look at the 1st error, it\'s something unrelated to the disk.
dlopen error: /var/lib/boinc/projects/climateprediction.net/famous_se_6.01_i686-pc-linux-gnu.so: cannot restore segment prot after reloc: Permission denied


The disk problems were because it was out of space because each models generates gigabytes of information, far more than a typical climate model. Is this by design? And if so, how much space is needed per model? Is not, then all this debugging files it generates needs to be quelled.

I reset the project and will try one more FAMOUS model, but if there\'s a way to run these models without the gigabytes of files generated, let me know.
ID: 39378 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 39379 - Posted: 26 Mar 2010, 22:04:59 UTC - in response to Message 39375.  
Last modified: 26 Mar 2010, 22:47:09 UTC

Found this in the stdout file. What does it mean?

dlopen error: /var/lib/boinc/projects/climateprediction.net/famous_se_6.01_i686-pc-linux-gnu.so: cannot restore segment prot after reloc: Permission denied

From this search it looks like it\'s related to the way shared libraries which require text relocation are handled in Security-Enhanced Linux when enforcing is enabled.

There are a couple of options:

  • disable enforcing as described here or here.

  • make the FAMOUS shared library trusted by running

    chcon -t textrel_shlib_t /var/lib/boinc/projects/climateprediction.net/famous_se_6.01_i686-pc-linux-gnu.so


I\'ll post this as a sticky in the Linux forum and pass it on to the developers (the shared library can almost certainly be built to avoid text relocation).


"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 39379 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39381 - Posted: 26 Mar 2010, 22:50:41 UTC - in response to Message 39379.  

===snip===

* make the FAMOUS shared library trusted by running

chcon -t textrel_shlib_t /var/lib/boinc/projects/climateprediction.net/famous_se_6.01_i686-pc-linux-gnu.so

If you take the last approach you\'d have to repeat it on every new version.

I\'ll post this as a sticky in the Linux forum and pass it on to the developers (the shared library can almost certainly be built to avoid text relocation).

Thanks for your help Thyme. I implemented a policy so the whole climateprediction.net folder is in that SEL context. Also ran the chcon so took effect immediately. Will that stop the runaway created-files?

The new task that I\'m testing is here: 11386757
ID: 39381 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 39382 - Posted: 26 Mar 2010, 23:04:22 UTC - in response to Message 39381.  
Last modified: 26 Mar 2010, 23:05:19 UTC

Thanks for your help Thyme. I implemented a policy so the whole climateprediction.net folder is in that SEL context. Also ran the chcon so took effect immediately. Will that stop the runaway created-files?

I hope so :)

Nobody reported the problem in CPDN Beta, so I\'m guessing it\'s directly related to the SELinux security policy. Assuming your changes fix the problem it would help others if you posted the details to this sticky.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 39382 · Report as offensive     Reply Quote
DJStarfox

Send message
Joined: 27 Jan 07
Posts: 300
Credit: 3,288,263
RAC: 26,370
Message 39419 - Posted: 30 Mar 2010, 14:26:42 UTC - in response to Message 39382.  

ID: 39419 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 39420 - Posted: 30 Mar 2010, 18:54:28 UTC - in response to Message 39419.  

Thanks!
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 39420 · Report as offensive     Reply Quote

Message boards : Number crunching : Size of FAMOUS on HD

©2024 cpdn.org