climateprediction.net (CPDN) home page
Thread 'Huge dataout folder'

Thread 'Huge dataout folder'

Questions and Answers : Unix/Linux : Huge dataout folder
Message board moderation

To post messages, you must log in.

AuthorMessage
Profilefjalar

Send message
Joined: 8 Sep 04
Posts: 4
Credit: 202,368
RAC: 0
Message 11728 - Posted: 9 Apr 2005, 21:42:25 UTC

Hi everybody,

I'm using core client 4.19 on SuSE Linux 9.2 (kernel 2.6.8-24-default).

My dataout folder contains lots of files with names like1x4maa.pa22c10.x1.nc and sizes between 2 and 6 MB. The whole folder is about 2 GB.

Something seems to be wrong here - it certainly can't be normal that the _out_ folder contains hundreds of files. Shouldn't they be purged after processing or whenever results are uploaded?

Apart from this, everything seems to be running OK.

Any suggestions?

TIA
Matthias
ID: 11728 · Report as offensive     Reply Quote
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 11765 - Posted: 11 Apr 2005, 16:09:24 UTC

It is normal for there to be lots of files and lots of disk space used by an experiment. At the end of that model run, there will be about 700 MB of disk space used. Then in the post phase processing, these files will be zipped and the experiment folder will be reduced to about 333 MB. Five large files totaling about 7 MB will be uploaded to CPDN servers. You can then either leave that model's folder on your HD, back it up to a CD/DVD, or delete it if you don't have space. Occasionally the project investigators will request more files from the person who ran an interesting experiment. If they are not available, that result may need to be run again.
ID: 11765 · Report as offensive     Reply Quote
Profilefjalar

Send message
Joined: 8 Sep 04
Posts: 4
Credit: 202,368
RAC: 0
Message 11778 - Posted: 12 Apr 2005, 7:55:03 UTC - in response to Message 11765.  

> It is normal for there to be lots of files and lots of disk space used by an
> experiment. At the end of that model run, there will be about 700 MB of disk
> space used.

Well, I'm way above 700 MB. Space is not the real problem - just wondered if everything was running properly. As long as I'm not running out of disk space I'll leav the files where they are then.

Matthias
ID: 11778 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 11779 - Posted: 12 Apr 2005, 10:07:44 UTC - in response to Message 11778.  
Last modified: 12 Apr 2005, 10:08:02 UTC

> Well, I'm way above 700 MB. Space is not the real problem - just wondered if
> everything was running properly. As long as I'm not running out of disk space
> I'll leav the files where they are then.

Do you have a load of *.da* files in the dataout directory Matthias? Another user reported something very similar <a href="http://climateapps2.oucs.ox.ac.uk/cpdnboinc/forum_thread.php?id=1878#10072">here</a>.

I think they are temporary work files which should be automatically deleted, but don't quote me on that.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 11779 · Report as offensive     Reply Quote
Profilefjalar

Send message
Joined: 8 Sep 04
Posts: 4
Credit: 202,368
RAC: 0
Message 11780 - Posted: 12 Apr 2005, 10:17:11 UTC - in response to Message 11779.  


&gt; Do you have a load of *.da* files in the dataout directory Matthias? Another
&gt; user reported something very similar

No, most of them look like this:

climateprediction.net/1x4m_100110504/dataout/1x4mba.pf26c10

about 7MB apiece, some only 2MB and some up to 8.
And there are some files with *.nc extension, about 2 MB each.

Matthias

ID: 11780 · Report as offensive     Reply Quote

Questions and Answers : Unix/Linux : Huge dataout folder

©2024 cpdn.org