climateprediction.net (CPDN) home page
Thread 'out of disk space, why?'

Thread 'out of disk space, why?'

Questions and Answers : Unix/Linux : out of disk space, why?
Message board moderation

To post messages, you must log in.

AuthorMessage
old_user44752

Send message
Joined: 28 Jan 05
Posts: 3
Credit: 2,382
RAC: 0
Message 9071 - Posted: 10 Feb 2005, 10:06:45 UTC

Hi, I set the boinc client to leave min .5GB free but it uses up all my disk space constantly so what can I do?
ID: 9071 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 9073 - Posted: 10 Feb 2005, 10:37:47 UTC

CP is a BIG project: 700Mb per processor during the run, plus 330Mb of zipped files at the completion of a model.
The project is really designed for modern computers, which usually have large HDs.

How much disk space do you have free to allocate?

Les

ID: 9073 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 9958 - Posted: 25 Feb 2005, 7:57:35 UTC - in response to Message 9073.  

I have a very similar problem. I have set boinc to use at most 2GB of HD space, but it has already used 3.3 GB. While there is still enough space left now, I wonder what will happen if boinc continues ignoring this setting... (only about 4GB left, and my CP model is not even halfway through yet). Is there any way to force boinc to free up some space without interrupting its calculations?

ID: 9958 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 9964 - Posted: 25 Feb 2005, 8:44:16 UTC

janhefti,

It's a bit hard to be specific because you have hidden your computers.
If it's used that much space, it must mean that you have a lot of old model data still in the project folder.
Move completed models out of the BOINC folder to a 'save' folder.

Les
ID: 9964 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 9993 - Posted: 25 Feb 2005, 15:20:18 UTC - in response to Message 9964.  

Hi Les,

thank you very much for your fast reply. Actually, most of the data is in the 'dataout' folder of my current project (over 600 files named '3ujcaa._______', e.g. '3ujcaa.ph22c10'). Are these supposed to be sent to cp.net, and could their still being in this folder signify a network problem? Or is it safe to back up the data and move it to another medium?

Jan

ID: 9993 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 10000 - Posted: 25 Feb 2005, 16:44:17 UTC - in response to Message 9993.  

> thank you very much for your fast reply. Actually, most of the data is in the
> 'dataout' folder of my current project (over 600 files named '3ujcaa._______',
> e.g. '3ujcaa.ph22c10'). Are these supposed to be sent to cp.net, and could
> their still being in this folder signify a network problem? Or is it safe to
> back up the data and move it to another medium?

Whatever you do don't delete them. They're the working files for the job and are required to generate the 5 result files that will be uploaded when the run completes.

After the model finishes the files will be compressed (down to about 330Mb) and moved up to the '3ujc*' directory. You can then delete that directory, preferably archiving it first so that it'll be available should the project team want to analyse more than the 7Mb that gets uploaded (some people were asked to do that for the CPDN classic THC experiment).
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 10000 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 10069 - Posted: 27 Feb 2005, 10:33:52 UTC - in response to Message 10000.  

> Whatever you do don't delete them. They're the working files for the job and
> are required to generate the 5 result files that will be uploaded when the run
> completes.
>
> After the model finishes the files will be compressed (down to about 330Mb)
> and moved up to the '3ujc*' directory. You can then delete that directory,
> preferably archiving it first so that it'll be available should the project
> team want to analyse more than the 7Mb that gets uploaded (some people were
> asked to do that for the CPDN classic THC experiment).

Ok, thank you, I won't. But does that mean that it is impossible to save disk space until the project is finished, and that I will have to make more space available to let boinc complete its task? I did not expect CPDN to take more than 2GB (the limit I set), but it will be around 10GB if disk usage keeps increasing at the current rate.

ID: 10069 · Report as offensive     Reply Quote
ProfileAndrew Hingston
Volunteer moderator

Send message
Joined: 17 Aug 04
Posts: 753
Credit: 9,804,700
RAC: 0
Message 10072 - Posted: 27 Feb 2005, 13:11:40 UTC - in response to Message 10069.  
Last modified: 27 Feb 2005, 13:11:56 UTC

> does that mean that it is impossible to save disk
> space until the project is finished, and that I will have to make more space
> available to let boinc complete its task? I did not expect CPDN to take more
> than 2GB (the limit I set), but it will be around 10GB if disk usage keeps
> increasing at the current rate.

I'm puzzled that you should be needing so much already. Your computers are hidden, so I cannot be sure whether you are running hyperthreaded with two models on one machine, but even if you are my experience is that you should not need much more than a gigabyte.

Each model run is in its own folder, and these are readily identifiable from their number. Other folders in the projects directory are from earlier runs and as Thyme Lawn says can be removed to a storage drive elsewhere or onto CD/DVD. Even toward the end of a model, the total size of a run folder should not get much above 600MB.

What is the size of the 'dataout' folder for your current model(s)?
ID: 10072 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 10104 - Posted: 28 Feb 2005, 9:50:48 UTC - in response to Message 10072.  

> I'm puzzled that you should be needing so much already. Your computers are
> hidden, so I cannot be sure whether you are running hyperthreaded with two
> models on one machine, but even if you are my experience is that you should
> not need much more than a gigabyte.
>
> Each model run is in its own folder, and these are readily identifiable from
> their number. Other folders in the projects directory are from earlier runs
> and as Thyme Lawn says can be removed to a storage drive elsewhere or onto
> CD/DVD. Even toward the end of a model, the total size of a run folder should
> not get much above 600MB.
>
> What is the size of the 'dataout' folder for your current model(s)?

That machine is crunching only one model at a time, and its 'dataout' folder stands at 3.5GB (it is really the only folder that uses a considerable amount of space).

Just in case it helps - I put a list of all files in the projects directory at http://www.janhefti.de/boinc_lsr.txt , and my boinc client is boinc_4.19_i686-pc-linux-gnu.

ID: 10104 · Report as offensive     Reply Quote
ProfileThyme Lawn
Volunteer moderator

Send message
Joined: 5 Aug 04
Posts: 1283
Credit: 15,824,334
RAC: 0
Message 10105 - Posted: 28 Feb 2005, 11:01:18 UTC - in response to Message 10104.  

> That machine is crunching only one model at a time, and its 'dataout' folder
> stands at 3.5GB (it is really the only folder that uses a considerable amount
> of space).
>
> Just in case it helps - I put a list of all files in the projects directory at
> http://www.janhefti.de/boinc_lsr.txt , and my boinc client is
> boinc_4.19_i686-pc-linux-gnu.

I've no idea what all those *.da* files in your dataout directory are. They're certainly not generated by the Windoze version of the program, but I'm afraid you'll have to wait for someone who knows the Linux version to say why they're being created and if it's safe to delete them.
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 10105 · Report as offensive     Reply Quote
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 10112 - Posted: 28 Feb 2005, 13:47:49 UTC - in response to Message 10104.  

> Just in case it helps - I put a list of all files in the projects directory at
> http://www.janhefti.de/boinc_lsr.txt , and my boinc client is
> boinc_4.19_i686-pc-linux-gnu.
>
>
Those *da* files are new to me. I don't have anything like that in the dataout directory of my linux PC. Very odd.
ID: 10112 · Report as offensive     Reply Quote
Profileold_user248

Send message
Joined: 6 Aug 04
Posts: 65
Credit: 1,605,224
RAC: 0
Message 10203 - Posted: 2 Mar 2005, 11:59:05 UTC - in response to Message 10112.  

> Those *da* files are new to me. I don't have anything like that in the
> dataout directory of my linux PC. Very odd.
>
My dataout directory doesn't contain those files either. Are you running the sulphur cycle or some other experiment? When running 4 concurrent models the maximum that CP took up was just over 3 Gb at the end. That includes new models for processing, etc. Once the 4 new ones started and I backed up the completed models it was back down to around 350 Mb.

I think you need to know more from the core CP staff on this to figure out what is going on.
ID: 10203 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 10915 - Posted: 15 Mar 2005, 13:34:33 UTC - in response to Message 10203.  

Thank you very much, I really appreciate everybody's help.

In the meantime, my model completed its run for the first period, presumably sent its data to cp.net (I can access the temperature and precipitation graphs), and aborted on a client error (exit status 251). My computer then started a new model, and it is building up those *da* files again. If they were just a nuisance, it wouldn't matter much, I could just clean them up with a cron job periodically, but if the new model fails on the same error, I guess I will have to give up, as gaining credit without presenting usable results just doesn't feel right. But I will definitely try again once a new version of boinc is available.
ID: 10915 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 10919 - Posted: 15 Mar 2005, 13:46:25 UTC

You could try the new 4.25 version of BOINC, available from <a href="http://boinc.berkeley.edu/download.php"> here.</a> Although it seems that they have moved to 4.26 now.

The 251 error seems to be confined to Linux users.

Les
ID: 10919 · Report as offensive     Reply Quote
old_user40440

Send message
Joined: 27 Jan 05
Posts: 6
Credit: 6,054
RAC: 0
Message 10948 - Posted: 15 Mar 2005, 16:50:00 UTC - in response to Message 10919.  

&gt; You could try the new 4.25 version of BOINC, available from <a> href="http://boinc.berkeley.edu/download.php"&gt; here.</a> Although it seems
&gt; that they have moved to 4.26 now.
&gt;
&gt; The 251 error seems to be confined to Linux users.
&gt;
&gt; Les

Good idea! I'll give 4.26 a try.

Jan
ID: 10948 · Report as offensive     Reply Quote

Questions and Answers : Unix/Linux : out of disk space, why?

©2024 cpdn.org