 |
|
Message boards : Number crunching : What happens if I run out of disk space?
Message board moderation
Previous · 1 · 2
| Author | Message |
|---|---|
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
0.) "Are there message lines, either in Messages, or in the stdoutdae.txt file, which say something like: Started upload of hadcm3_a009_1859_10_000258667_0_1.zip?" Yes; there are two like that in messages. There are no files named stdoutdae.txt on my machine. There are files like stdout_um.txt but they do not contain lines like that. 10-Apr-2012 09:35:40 [climateprediction.net] Started upload of hadcm3n_ydi1_1980_40_007832936_3_1.zip ... 11-Apr-2012 10:03:36 [climateprediction.net] Started upload of hadcm3n_o34p_1980_40_007833299_0_1.zip 1.) "There's NO deadline in this project." 2.) "It's been posted many times that the hadcm3 models are very competitive for the FPU, and slow each other down A LOT. Only having half the number of these as there are processor cores is a good rule of thumb. Otherwise, they go into high priority running a lot of the time." All my BOINC applications are compute limited. So they compete a lot. But none are so bad as to freeze out another processor core. I have two processor chips; Intel 3.06 GHz hyperthreaded ones with 1 Megabyte L3 caches. So it is up to you if you count that as two or four processors. Linux thinks it is four, but in some processes, one gets so busy that the other shuts down. (so that only three are running and one is idle). But this does not apply to the BOINC applications. It is usually something like Firefox when running Flash stuff. Do I care if the processes run high priority? How would I reduce the number of C.P. processes loaded into my machine by my BOINC client? I know how to adjust the number of clean energy project applications under World Community Grid, but not under C.P. 3.) "The reason that the project moved on to newer versions of library files ..." 
|
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
It is an ext3 file system. Most of the file systems are ext3 on that machine. $ cat /etc/fstab LABEL=/ / ext3 defaults 1 1 LABEL=/boot1 /boot ext3 defaults 1 2 devpts /dev/pts devpts gid=5,mode=620 0 0 tmpfs /dev/shm tmpfs defaults 0 0 LABEL=/home /home ext3 defaults 1 2 LABEL=/home/boinc /home/boinc ext3 defaults 1 2 LABEL=/homeB /homeB ext3 defaults 1 2 LABEL=/opt /opt ext3 defaults 1 2 proc /proc proc defaults 0 0 LABEL=/srv/dbms/dataA /srv/dbms/dataA ext2 defaults,noatime 1 2 LABEL=/srv/dbms/dataB /srv/dbms/dataB ext2 defaults,noatime 1 2 LABEL=/srv/dbms/dataC /srv/dbms/dataC ext2 defaults,noatime 1 2 LABEL=/srv/dbms/dataD /srv/dbms/dataD ext2 defaults,noatime 1 2 LABEL=/srv/dbms/dataE /srv/dbms/dataE ext2 defaults,noatime 1 2 LABEL=/srv/dbms/dataF1 /srv/dbms/dataF ext2 defaults,noatime 1 2 sysfs /sys sysfs defaults 0 0 LABEL=/tmp /tmp ext3 defaults 1 2 LABEL=/usr /usr ext3 defaults 1 2 LABEL=/usr/local /usr/local ext3 defaults 1 2 LABEL=/usr/src /usr/src ext3 defaults 1 2 LABEL=/var /var ext3 defaults 1 2 LABEL=SWAP-sdb8 swap swap defaults,pri=16 0 0 LABEL=SWAP-sda9 swap swap defaults,pri=16 0 0
|
|
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
Eirik I think that you may have it, there. Minimum disk space per file is something that I haven't thought about for years, let alone had to set when setting up a new computer. With HDs so big these days, I just give everything plenty of elbow room and let it get on with it. There are lots of files created for climate models; I think a couple per iteration. Every 25% of the model, these have their extension type changed, and are then zipped into a single big file to send back to the project's servers. So if each one has a much-larger-than-needed size on the HD, that would account for the excessive amount of disk space being used. So somewhere around 16-20 Gigs of space for cpdn alone. Perhaps a partition of 30 Gigs would do it for multiple projects. PS One model has just created it's 2nd zip, and it's folder is now back to 293 Megs. Including the still unsent zip. Backups: Here |
|
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
Jean It's been estimated that hyper-threading gives an equivalent of around .2-.3 of an extra processor when running climate models. So use half the number of real cores for max number of hadcm3 type models. To limit the number of models downloaded: 1) Set No new tasks for cpdn in the Projects tab. 2) Set Network activity suspended 3) Change the number of processors, either on the server, or in the Manager, to the max number wanted. (Possibly 2 in this case.) 4) Update the project so that your computer knows about it. 5) Set Allow new tasks for cpdn in the Projects tab. 6) Set Network activity always available 7) Wait until the required number of models have been downloaded. 8) Set No new tasks for cpdn in the Projects tab. 9) Change the number of processors back to what you want. 10) Update the project so that your computer knows about it. With multiple projects, extra steps will be needed. Backups: Here |
|
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
And of course, there's the other option: Suspend several climate models, so that only one runs. Then there'll be less files created. And before doing this, it'll be necessary to set No new tasks. Backups: Here |
|
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
Jean It looks like your problem is that your computer isn't zipping up the accumulated files and returning them to the server. The program that does this is hadcm3n_se_6.07_i686-pc-linux-gnu.so, which is the small executables process. From the Stderr errors list of one of your models: Unable to load library hadcm3n_se_6.07_i686-pc-linux-gnu.so You'll need to found out why it's not working and fix it, otherwise you're just using time and electricity to return a few trickles, and no big zips. Backups: Here |
 Thyme LawnSend message Joined: 5 Aug 04 Posts: 1283 Credit: 15,824,334 RAC: 0 |
Unable to load library hadcm3n_se_6.07_i686-pc-linux-gnu.so Jean-David, Check if the distribution you're using has libstdc++.so.6 (version 6 of the Linux C++ library) installed in /usr/lib. If not check if it's somewhere else (e.g. /lib). If it's missing you'll have to install it (e.g. use RPM Search, enter libstdc++.so.6 in the search box and select the relevant distribution). "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer |
|
Send message Joined: 19 Apr 08 Posts: 179 Credit: 4,306,992 RAC: 0 |
Well he certainly has libstdc++6, just an older version without GLIBCXX_3.4.9 support. If the machines are under a support contract with Red Hat, Jean should contact them. If not he can try installing an RPM on his own. If the install fails because it breaks compatibility with something, then he should try this ingenious workaround from a profoundly dedicated member. |
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
"Check if the distribution you're using has libstdc++.so.6 (version 6 of the Linux C++ library) installed in /usr/lib. If not check if it's somewhere else (e.g. /lib)." $ locate libstdc++.so.6 /usr/lib/libstdc++.so.6 /usr/lib/libstdc++.so.6.0.8 /usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.8-gdb.py /usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.8-gdb.pyc /usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.8-gdb.pyo /usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.8-gdb.py /usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.8-gdb.pyc /usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.8-gdb.pyo
|
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
rpm -qf /usr/lib/libstdc++.so.6.0.8 libstdc++-4.1.2-52.el5
|
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
Red Hat will not supply a new enough library for that. To get it, I must upgrade my entire system to REL 6, and I am not going to do that. I may upgrade to RHEL 7 when it comes out, probably a year or more from now. So I guess I have to drop out of Climate Prediction for a year or so.
|
|
Send message Joined: 5 Sep 04 Posts: 7629 Credit: 24,240,330 RAC: 0 |
|
|
Send message Joined: 19 Apr 08 Posts: 179 Credit: 4,306,992 RAC: 0 |
Jean, will libstdc++-4.3.0-8.i386.rpm from this page install? Edit: If it does you'll have to lock it in with your package manager to keep it from "updating" back to the original. |
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
It will not install: # rpm -Uvh libstdc++-4.3.0-8.i386.rpm warning: libstdc++-4.3.0-8.i386.rpm: Header V3 DSA signature: NOKEY, key ID 4f2a6fd2 error: Failed dependencies: libstdc++ < 4.2.0 is needed by (installed) compat-gcc-34-c++-3.4.6-4.1.i386 libstdc++ = 4.1.2-52.el5 is needed by (installed) libstdc++-devel-4.1.2-52.el5.i386 libstdc++ = 4.1.2-52.el5 is needed by (installed) gcc-c++-4.1.2-52.el5.i386
|
|
Dave Jackson Send message Joined: 15 May 09 Posts: 4540 Credit: 19,039,635 RAC: 18,944 |
I think I had this problem when I was running Mandriva - I had to manually install the dependencies first and that worked but it felt like a pain in the neck @ the time. Dave |
|
Send message Joined: 5 Aug 04 Posts: 1120 Credit: 17,202,915 RAC: 2,154 |
In the past, I would try to load the dependencies first. But that just produced an cascade of new dependencies. Sometimes it would take a week or more to get them all straightened out. Sometimes it was impossible and I would have to restore the whole system from a backup tape and give up. Now I will not even attempt it. It is just not worth it.
|
©2024 cpdn.org
