climateprediction.net (CPDN) home page
Thread 'wah2_eu25 work units are BIG'

Thread 'wah2_eu25 work units are BIG'

Message boards : Number crunching : wah2_eu25 work units are BIG
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 53050 - Posted: 7 Dec 2015, 23:06:33 UTC

I place the BOINC Data folder on a 14 GB ramdisk on an i7-3770 machine where I run a maximum of six wah2_eu25 work units at a time, but that was not enough memory. One of them errored out with an insufficient storage space message. Looking into it, I see that each work unit takes at least 2 GB, and I have seen up to 3.6 GB. So be forewarned, they are big.
ID: 53050 · Report as offensive     Reply Quote
Profiletullio

Send message
Joined: 6 Aug 04
Posts: 264
Credit: 965,476
RAC: 0
Message 53056 - Posted: 8 Dec 2015, 18:54:31 UTC

They are also slow, at least on my Windows 10 PC with 20 GB RAM and 2 TB disk. It is true that it hosts also VirtualBox 5.0.10 and 4 BOINC CERN projects, plus the CERN 2015 Challenge outside BOINC.
Tullio
ID: 53056 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Jan 06
Posts: 637
Credit: 26,751,529
RAC: 653
Message 53057 - Posted: 8 Dec 2015, 19:46:15 UTC - in response to Message 53056.  

They will take about 9 1/2 days on the virtual cores of my i7-3770 machines, and about 6 1/2 days on the full cores of my i5-3550. That is OK; they a presumably getting some real work done, and I run the machines 24/7 anyway. I can deal with the memory too; after this group is finished, I will increase the ramdisk to 24 GB; that should do it. But it comes as a bit of a surprise, and I think there should have been some warning.
ID: 53057 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 53058 - Posted: 8 Dec 2015, 22:09:44 UTC

I did "sort of" mention this at the bottom, in the first post of this thread.

with latest land cover model

This latest batch are submission for the MARiUS project. details of which are on the site.


And I've mentioned a lot earlier that the newer models are/will be, using the newest version of the MOSES land scheme (which is version II), and also TRIFFID

Both of these are land use schemes, and details can be found by a web search. They are also used for one of the "Linux only" models. (See the Server Status page.)

These newest modelling versions are much more detailed than those used in earlier work here, say up to 2 years ago, so they take a lot longer to calculate, and produce lots more data to return to the projects using them.

And then there's this at the end of the MaRIUS project page:

This will in particular involve the inclusion of an improved land-surface scheme into both the global and regional model, and a higher-resolution regional model at 25km.


I'm afraid that I'm often guilty of assuming that those who post here regularly, also read every new post, and then looks for more info when something new is mentioned.

Later, after I've gotten some of today out of the way, I'll make a News item, repeating a lot of this, plus anything else that I think of.

ID: 53058 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 53063 - Posted: 9 Dec 2015, 5:54:26 UTC
Last modified: 9 Dec 2015, 9:12:15 UTC

I've decided to make it a separate sticky post at the top of Number crunching.
It's called: Models are getting more detailed, and therefore bigger

Also, don't forget that failed models don't clean up after them selves. And neither do the "short" models whether Success or Fail.

edit
But it's possible that the "short" models MAY clean up on Windows.
The best way to deal with shrinking file space is to have a look for ANY left overs.
Keeping a human in the loop can help with cpdn. :)
ID: 53063 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 53065 - Posted: 9 Dec 2015, 8:22:54 UTC - in response to Message 53063.  

And neither do the "short" models whether Success or Fail.


I thought Les from reading posts on this that the short ones do clean up on windows and it was only the Linux versions which got pulled anyway that didn't?
ID: 53065 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 53066 - Posted: 9 Dec 2015, 9:07:32 UTC - in response to Message 53065.  

I'm not sure about it. You could be right.
I'll change the post a bit.

ID: 53066 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 54257 - Posted: 6 Jun 2016, 7:13:47 UTC

Are some bigger than others? http://climateapps2.oerc.ox.ac.uk/cpdnboinc/show_host_detail.php?hostid=1384830 This machine is running 4 tasks under WINE. Two of them are from batch no. 366. One is progressing at a rate of about 4% a day while the other is only managing 1% a day.

Machine info as follows.
GenuineIntel
Intel(R) Pentium(R) CPU N3540 @ 2.16GHz [Family 6 Model 55 Stepping 8]
Number of processors 4
Coprocessors ---
Operating System Microsoft Windows 8.1
Ultimate N x64 Edition, , (06.03.9600.00)
BOINC client version 7.6.22
Memory 7863.95 MB
Cache 1024 KB
Swap space 7893.95 MB
Total disk space 458.32 GB
Free Disk Space 388.98 GB
Measured floating point speed 472.1 million ops/sec
Measured integer speed 3486.25 million ops/sec

Nothing else running on machine 95% of time and during the 5% or less only a little light browsing.
ID: 54257 · Report as offensive     Reply Quote
John Eric Hopkinson

Send message
Joined: 27 Jan 05
Posts: 74
Credit: 1,047,809
RAC: 0
Message 54259 - Posted: 6 Jun 2016, 17:21:36 UTC

Dave:
How does this compare? :On a WIN10 laptop XPS14
Four wah2_eu25 WUs,
wah2_eu25_Odbv.....etc 86.316% progress @ 12d 04:14:06
three others range from 83.6% to 85.7% same elapsed time.
Question: do trickles show up during progress or only on upload of finished WU, and are there graphics associated with these tasks?
( I am not having much luck with installing CPDN on WINE Linux Mint, but have not yet given up}

Created 12 May 2016 15:34:01 UTC "(same date as restarting with CPDN)"
Total credit 15,278
Average credit 1,170.27
Cross project credit BOINCstats.com Free-DC
CPU type GenuineIntel
Intel(R) Core(TM) i7-3517U CPU @ 1.90GHz [Family 6 Model 58 Stepping 9]
Number of processors 4
Coprocessors NVIDIA GeForce GT 630M (1024MB) driver: 35362
Operating System Microsoft Windows 10
Core x64 Edition, (10.00.10586.00)
BOINC client version 7.6.22
Memory 8069.75 MB
Cache 256 KB
Swap space 9349.75 MB
Total disk space 457.85 GB
Free Disk Space 197.06 GB
Measured floating point speed 3098.05 million ops/sec
Measured integer speed 8580.18 million ops/sec
Average upload rate 64.51 KB/sec
Average download rate 126.52 KB/sec
Average turnaround time 0 days
Maximum daily WU quota per CPU 0/day
Tasks 4
Number of times client has contacted server 53
Last time contacted server 4 Jun 2016 2:59:54 UTC
% of time BOINC client is running 77.9231 %
While BOINC running, % of time host has an Internet connection 90.4781 %
While BOINC running, % of time work is allowed 99.967 %
Task duration correction factor
ID: 54259 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 54260 - Posted: 6 Jun 2016, 18:48:27 UTC - in response to Message 54259.  

Yours all seem to show broadly similar percentages for elapsed time. What seems particularly strange to me about this one is that they are both from the same batch (366) and one of them seems to be progressing three to four times as fast as the other.

Sadly no graphics on WAH2 tasks to look for clues. Will have a look at event log for checkpoint times and possible email someone who knows about what is happening at project level.
ID: 54260 · Report as offensive     Reply Quote
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 54261 - Posted: 6 Jun 2016, 19:57:18 UTC - in response to Message 54260.  

Yours all seem to show broadly similar percentages for elapsed time. What seems particularly strange to me about this one is that they are both from the same batch (366) and one of them seems to be progressing three to four times as fast as the other.

Sadly no graphics on WAH2 tasks to look for clues. Will have a look at event log for checkpoint times and possible email someone who knows about what is happening at project level.


If you are talking about the two tasks that were downloaded on May 5th, one trickled first on May 7, and the other trickled first on May 14. It looks like one didn't have to wait very long to start after downloading, but the other had to wait quite awhile for another task to finish before it could start? Unless I'm mixing up the tasks you are comparing?
ID: 54261 · Report as offensive     Reply Quote
John Eric Hopkinson

Send message
Joined: 27 Jan 05
Posts: 74
Credit: 1,047,809
RAC: 0
Message 54272 - Posted: 8 Jun 2016, 14:55:34 UTC - in response to Message 54261.  

Dave and Geophi:
The three workunits referenced below will finish in about ten hours. I have set preferences to prevent new downloads because I may be able to improve performance with some adjustments to preferences. However, I dont know how to evaluate the performance.
Can you look at these in the log file and provide some guidance.
And, will there be more workunits provided for Linux in the near future?
Thanks for the help.
ID: 54272 · Report as offensive     Reply Quote
ProfileIain Inglis
Volunteer moderator

Send message
Joined: 16 Jan 10
Posts: 1084
Credit: 7,824,485
RAC: 4,956
Message 54274 - Posted: 9 Jun 2016, 22:23:25 UTC

Just got a mex50 model - wah2_mex50_fo0v_193812_12_407_010607167_1. New one on me ...
ID: 54274 · Report as offensive     Reply Quote
John Eric Hopkinson

Send message
Joined: 27 Jan 05
Posts: 74
Credit: 1,047,809
RAC: 0
Message 54275 - Posted: 9 Jun 2016, 23:10:58 UTC - in response to Message 54274.  

Interesting:
First, because I have not seen Iain Inglis for ten years, and second because I have been waiting for downloads after 4 completions for 18 hours, with server stats showing more than 67000 units ready for blastoff.
However, I did see in Applications that Windows 10 is not listed. So maybe I have some kind of anomaly here.
Hold on, let me look at my Linux rig.
Les Bayliss, Can you add anything here?
ID: 54275 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 54276 - Posted: 10 Jun 2016, 0:09:36 UTC

Iain

Mexico has just joined up.
It's mentioned on the front page, bottom right, under
Experiments Coming Soon


ID: 54276 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 54277 - Posted: 10 Jun 2016, 0:26:19 UTC - in response to Message 54275.  

John

The only thing wrong with Windows 10, is that MicroSoft has taken over people's computers with that version.
Different problems with different versions, and with how it's set up. (And apparently you can't do anything about it with the Home version.)

Your tasks list shows that you have 4 new models there. Which probably happened 5 seconds after you posted. :)

You'll need to look at your Event log, stdoutdae.txt, and stderrdae.txt files etc, to see why they were delayed.

I'll email "them" about adding "10" to the applications list.

ID: 54277 · Report as offensive     Reply Quote
John Eric Hopkinson

Send message
Joined: 27 Jan 05
Posts: 74
Credit: 1,047,809
RAC: 0
Message 54278 - Posted: 10 Jun 2016, 3:24:46 UTC - in response to Message 54277.  

Righto mate, thanks.
Now, I found a small map a while back which showed all the wah2 regions, but now I dont know where that map is. However, Northeastern North America e.g. Quebec to Newfoundland and out to Greenland was not included. That may be a useful study area because the most recent weather has been absolutely haywire. Who decides what areas are to be studied and how is all this coordinated. This is not clear in the CPDN Home page info etc.
I will follow up on Experiments Coming Soon as you recommended.
Thanks Les.
P.S. Win10 does try to run your life if you let it, and just for relief, I have fired up an old IBM756CSE laptop which has OS2 and WIndows 3.1 loaded. Life was simpler then, but we survived, and spent less time messing with the hardware.
ID: 54278 · Report as offensive     Reply Quote
Profilegeophi
Volunteer moderator

Send message
Joined: 7 Aug 04
Posts: 2187
Credit: 64,822,615
RAC: 5,275
Message 54279 - Posted: 10 Jun 2016, 4:05:44 UTC

ID: 54279 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 54280 - Posted: 10 Jun 2016, 4:22:38 UTC - in response to Message 54278.  
Last modified: 10 Jun 2016, 4:24:54 UTC

Hi John

The regions map is in the page about the Mac versions
All new models can now be run on Macs

It's a bit small and gloomy, but if you save it, and then use Windows explorer to open it, you can expand it.

Areas to be studied are decided by a research centre interested in that area, mostly by people who are in it I guess.
How they get to know about cpdn, is probably via conferences and such.

But the big problem is money - they need to convince the people paying their wages that it's a good idea, part of which is the cost to lease space on a big server somewhere, and also pay for the large net connections to it.
This is what caused the long delay between testing the ANZ models, and them going into production.

Edit
I was a bit slow, but now you have two answers. :)
ID: 54280 · Report as offensive     Reply Quote
jrapdx

Send message
Joined: 4 Jul 15
Posts: 63
Credit: 3,223,760
RAC: 0
Message 54281 - Posted: 10 Jun 2016, 5:24:18 UTC - in response to Message 54277.  

Re: Windows 10. Yes it is a quagmire for the privacy-minded. There are a number of guides on the web re: how to configure to decrease exposure to MS snooping.

Also, some tools are available to help reduce MS tracking on Win10. This one seems to work and is fairly easy to use: see DisableWinTracking at
https://github.com/10se1ucgo/DisableWinTracking
ID: 54281 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : wah2_eu25 work units are BIG

©2024 cpdn.org