climateprediction.net (CPDN) home page
Thread 'Delete files from finished WU'

Thread 'Delete files from finished WU'

Questions and Answers : Wish list : Delete files from finished WU
Message board moderation

To post messages, you must log in.

AuthorMessage
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7084 - Posted: 28 Dec 2004, 19:11:46 UTC

Everytime my computers finish a WU, there are several hundreds of Mb left. I have to go and delete them myself. I know that these files may be valuable but I think we should know if they are really valuable by the time the WU finishes and if they are, CP should retreive them. In any other case, those files should be automatically deleted to leave space for other WUs.
Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7084 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 7087 - Posted: 28 Dec 2004, 21:37:52 UTC

The files ARE valuable! The matter has been discussed several times on these forums.
They should be saved onto CDs or some similar media for later use by the scientists.
This is a long term research project, and it may be a few years before the scientists
get to the stage in their analyses where they need these extras files. And then they may only
need some of them.

Les
ID: 7087 · Report as offensive     Reply Quote
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7088 - Posted: 28 Dec 2004, 21:43:59 UTC - in response to Message 7087.  

So, am I supposed to keep CD after CD for years??? That seems to me distributed storage not distributed computing. Why they don't retrieve the data? I never read that I had to store data for ages.

By the way, once I asked if I could delete the files and someone from CP told me I could.

G

> The files ARE valuable! The matter has been discussed several times on these
> forums.
> They should be saved onto CDs or some similar media for later use by the
> scientists.
> This is a long term research project, and it may be a few years before the
> scientists
> get to the stage in their analyses where they need these extras files. And
> then they may only
> need some of them.
>
> Les
>
>
Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7088 · Report as offensive     Reply Quote
crandles
Volunteer moderator

Send message
Joined: 16 Oct 04
Posts: 692
Credit: 277,679
RAC: 0
Message 7092 - Posted: 28 Dec 2004, 22:16:12 UTC - in response to Message 7088.  

> So, am I supposed to keep CD after CD for years??? That seems to me
> distributed storage not distributed computing. Why they don't retrieve the
> data? I never read that I had to store data for ages.
>
> By the way, once I asked if I could delete the files and someone from CP told
> me I could.
>
> G

I think David Anderson of SETI and BOINC wants to take BOINC in the direction of distributed storage as well as distributed computing.

You do not have to keep the data. However it seems a shame if your run does show something interesting and is either not available to be fully examined at a later stage or the model has to be calculated again.

Not essential to keep, but nice if they can be kept.
Visit BOINC WIKI for help

And join BOINC Synergy for all the news in one place.
ID: 7092 · Report as offensive     Reply Quote
old_user23880
Volunteer tester

Send message
Joined: 10 Oct 04
Posts: 223
Credit: 4,664
RAC: 0
Message 7093 - Posted: 29 Dec 2004, 1:19:55 UTC

We do know that some people have been asked by the researchers for particular previous models which are, if available, needed for extra analysis. If they are stored as zipped files, they don't take so much space.
__________________________________________________

ID: 7093 · Report as offensive     Reply Quote
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7094 - Posted: 29 Dec 2004, 1:27:51 UTC - in response to Message 7093.  
Last modified: 29 Dec 2004, 19:26:49 UTC

> If they are stored as zipped files, they don't take so much space.

Well, in my opinion 300 Mb/WU it's quite a lot of space


Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7094 · Report as offensive     Reply Quote
ProfileAndrew Hingston
Volunteer moderator

Send message
Joined: 17 Aug 04
Posts: 753
Credit: 9,804,700
RAC: 0
Message 7108 - Posted: 30 Dec 2004, 21:36:31 UTC - in response to Message 7094.  


> Well, in my opinion 300 Mb/WU it's quite a lot of space

Hard disk space is fairly inexpensive nowadays, but you could always save to CD/DVD if it is a problem.
ID: 7108 · Report as offensive     Reply Quote
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7109 - Posted: 30 Dec 2004, 21:43:18 UTC - in response to Message 7108.  

>
> > Well, in my opinion 300 Mb/WU it's quite a lot of space
>
> Hard disk space is fairly inexpensive nowadays, but you could always save to
> CD/DVD if it is a problem.
>

yeah, well....

Another point is that maybe CP should keep enough information about each WU crunched (which I suppose they do) so they can rerun it in case there's anything interesting there. I wouldn't trust in some zip files stored for months in a CD of a guy I don't know.

G
Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7109 · Report as offensive     Reply Quote
ProfileAndrew Hingston
Volunteer moderator

Send message
Joined: 17 Aug 04
Posts: 753
Credit: 9,804,700
RAC: 0
Message 7113 - Posted: 30 Dec 2004, 23:13:36 UTC - in response to Message 7109.  


> Another point is that maybe CP should keep enough information about each WU
> crunched (which I suppose they do) so they can rerun it in case there's
> anything interesting there. I wouldn't trust in some zip files stored for
> months in a CD of a guy I don't know.

Since September, the BOINC version has uploaded a couple of extra files, but 300MB is much more of an issue if uploaded and stored centrally than it is when sitting on a machine hard drive, especially if the client is using dial-up! It's a fair point though, and an aspect of the BOINC philosophy not fully explored so far. David Anderson's concept has local storage as a key part of the DC network, but it poses all sorts of difficulties. Storage on CD is better than nothing, though inferior to leaving it on the HD.

The CPDN project team recognise that many people will junk their old files, and BOINC does it automatically if the allocated space is full.
ID: 7113 · Report as offensive     Reply Quote
Profileold_user3222

Send message
Joined: 30 Aug 04
Posts: 20
Credit: 11,115
RAC: 0
Message 7584 - Posted: 25 Jan 2005, 11:03:17 UTC

To tell you the truth, it should be the people at CPDN the ones to design the proper ways of having these files eliminated from our files, just like all the other projects do.

<img src="http://150.214.190.154/BOINCStatistics/Signature/Signature.php?userName=Liberto&amp;cache=1&amp;auxiliar=1">
ID: 7584 · Report as offensive     Reply Quote
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7595 - Posted: 25 Jan 2005, 15:48:47 UTC - in response to Message 7584.  

Hola Liberto,

Exactly that was my point.

G
Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7595 · Report as offensive     Reply Quote
old_user1594

Send message
Joined: 26 Aug 04
Posts: 3
Credit: 45,592
RAC: 0
Message 7598 - Posted: 25 Jan 2005, 16:59:36 UTC

This bothers me greatly. I have 5 gigs devoted to Boinc, and that's more than enough for Seti, Predictor, Einstein, and LHC.... but each Climate work unit saves hundreds of megabytes on old units. Won't this quickly take up all my free space?

I don't want Climate to save all its old information on my computer, regardless of how important it is, and fill up my Boinc free space so other programs can't run!
-----

<img src="http://boinc.mundayweb.com/one/stats.php?userID=68&amp;trans=off">
ID: 7598 · Report as offensive     Reply Quote
ProfileAndrew Hingston
Volunteer moderator

Send message
Joined: 17 Aug 04
Posts: 753
Credit: 9,804,700
RAC: 0
Message 7599 - Posted: 25 Jan 2005, 17:40:02 UTC - in response to Message 7598.  


&gt; I don't want Climate to save all its old information on my computer,
&gt; regardless of how important it is, and fill up my Boinc free space so other
&gt; programs can't run!
&gt;

It won't. If you cannot spare any more space then I think that BOINC actually deletes old WUs for you, and in any case you can do it yourself. Storing these old results is voluntary, but HD storage is now so cheap, and so many people have vast amounts spare, that it hardly be said to be a great imposition. Much more of an imposition for many people would be to have 300MB uploads, especially if using dial up, and in any case keeping them means being able to view and compare them yourself.
ID: 7599 · Report as offensive     Reply Quote
old_user23880
Volunteer tester

Send message
Joined: 10 Oct 04
Posts: 223
Credit: 4,664
RAC: 0
Message 7605 - Posted: 25 Jan 2005, 22:22:20 UTC

Hola catalans i valencians i tothom!

I have 6 or 7 classic models, including some of the bigger thermohaline ones, zipped on the HD, and according to the disk defragmenter, the % of free space hasn't changed much during more than a year of running cpdn. If I run out of space, I will start using CDs as well. It would actually be safer to to store the files in both ways.

Astro was asked to upload a model a few months back. After you've done all the work, it would be depressing if the researchers needed to study one of your models in more detail and you couldn't produce it.
__________________________________________________

ID: 7605 · Report as offensive     Reply Quote
Profileold_user2099
Avatar

Send message
Joined: 27 Aug 04
Posts: 25
Credit: 214,980
RAC: 0
Message 7606 - Posted: 25 Jan 2005, 22:28:29 UTC - in response to Message 7605.  

Hola mo.v,

&gt; After you've done all the
&gt; work, it would be depressing if the researchers needed to study one of your
&gt; models in more detail and you couldn't produce it.

Well, I am sure that CPDN may have enough CPU power themselves to rerun a WU if they are interested.

G


Military intelligence is a contradiction in terms (Groucho Marx)

at
ID: 7606 · Report as offensive     Reply Quote
Profileold_user3222

Send message
Joined: 30 Aug 04
Posts: 20
Credit: 11,115
RAC: 0
Message 7617 - Posted: 26 Jan 2005, 5:03:53 UTC - in response to Message 7605.  

&gt; Hola catalans i valencians i tothom!
&gt;
&gt; I have 6 or 7 classic models, including some of the bigger thermohaline ones,
&gt; zipped on the HD, and according to the disk defragmenter, the % of free space
&gt; hasn't changed much during more than a year of running cpdn. If I run out of
&gt; space, I will start using CDs as well. It would actually be safer to to store
&gt; the files in both ways.
&gt;
&gt; Astro was asked to upload a model a few months back. After you've done all the
&gt; work, it would be depressing if the researchers needed to study one of your
&gt; models in more detail and you couldn't produce it.
&gt;

Quina sort que tens!
We do not have neither the amount of CPU's nor the HD enough free space as to keep old already crunched units. If CPDN wants to re-study any unit, they should be smart enough to be able to reproduce or extract it from their files not to be using us as a warehousing unit.
As far as I am concerned I will keep a couple, the last one completed and the ongoing one, as for the rest they will be taken out and deleted from the file.

I do the computing, very gladly - and have participation in all projects, with one single CPU, but on top of that to have to store units... nope!!
ID: 7617 · Report as offensive     Reply Quote

Questions and Answers : Wish list : Delete files from finished WU

©2024 cpdn.org