Posts by Dave Jackson

Info	Message
21) Message boards : Cafe CPDN : WCG African Rainfall Project (ARP) restart update Apr 25, 2024 Message 71833 Posted 5 Nov 2024 by Dave Jackson	CPDN does the opposite, long tasks but less data to deal with for the project. Would that be a correct assessment? I think the longer tasks for CPDN are more about it working better for the science than about data considerations though CPDN has had issues with large amounts of data. It happened with some of the IFS batches.
22) Message boards : Cafe CPDN : WCG African Rainfall Project (ARP) restart update Apr 25, 2024 Message 71830 Posted 5 Nov 2024 by Dave Jackson	I don't know how many tasks they are sending out for this but downloads and uploads are often in single figures of KB/s, I would certainly rather crunch longer tasks for them and fewer of them to reduce the amount of data going to and from crunchers. I don't know if any of the scientists involved ever look at the WCG forums. I know you, Glenn are the first to regularly come on the CPDN ones and that has I think made a massive difference to understanding among the crunchers who regularly read your posts even if they don't agree with you always. WCG would I feel benefit greatly from more direct communication between those running projects and those who crunch.
23) Message boards : Cafe CPDN : WCG African Rainfall Project (ARP) restart update Apr 25, 2024 Message 71828 Posted 5 Nov 2024 by Dave Jackson	I got a few also over half a day ago and still download issues. The short test they did over the weekend downloaded ok but just very slowly. Yep. Taking longer to download tasks than it did when I downloaded CPDN tasks on dial-up! This is where it would be nice if BOINC could pause all the downloads except for one task. That way you could get one downloaded and running a bit more quickly. Edit: The downloads for CPDN were not quite as big then as they are now!
24) Message boards : Number crunching : Almost 2025. Why doesn’t this project support multithreading? Message 71825 Posted 4 Nov 2024 by Dave Jackson	Sorry, enabling & testing the multiprocessing in the older models is not something I'm going to spend time doing. They work fine as they are. I have more pressing things to do. Fair enough. There are always going to be priorities that we won't know anything about. If climate science had the funding and resources it deserves....
25) Message boards : Number crunching : Almost 2025. Why doesn’t this project support multithreading? Message 71823 Posted 4 Nov 2024 by Dave Jackson	All the meteorological codes that CPDN run are capable of multiprocessing, even the older ones. Of course they are capable of multiprocessing. They were written to run on supercomputers. I blame my last post on having a head stuffed full of cold at the moment! I can see that total throughput of tasks might be more without multithreading. However when there are relatively small batches and the first few hundred computers grab them all, then multithreading would spread the tasks out between more computers and would get them returned more quickly.
26) Message boards : Number crunching : Connection and Download issues Oct24 Message 71820 Posted 4 Nov 2024 by Dave Jackson	I checked with Andy about this. CPDN doesn't issue a 'not needed' response if a earlier task in the workunit finishes. Experience has taught them users get annoyed by tasks being killed. So, yes, you'll need to abort it yourself If only BOINC had an option to say you were more interested in the science than in credit allowing unwanted tasks to be killed by the project for those people. On checking through the tasks, it was just three on my box that had completed by today. At least two hadn't even started so unless the person (not) running them has a very fast computer, there isn't much doubt my Ryzen9 will get in first. Edit:If I had a vote, it would be for the tasks to be deleted. It might cut down on the numbers crunching for CPDN but over time might weed out some habitual very slow returners. But I get that such decisions are way above my pay grade. I am not intending to make waves by expressing my opinion!
27) Message boards : climateprediction.net Science : Climate change in the News Message 71816 Posted 4 Nov 2024 by Dave Jackson	Piece by Friederike Otto in the Grauniad on the floods in Spain and Global North's refusal to recognise Climate Change as an issue for us as well as the Global South.
28) Message boards : Cafe CPDN : WCG African Rainfall Project (ARP) restart update Apr 25, 2024 Message 71815 Posted 4 Nov 2024 by Dave Jackson	ARPs are out there. You just can't download any of the files because of HTTP errors and download backoffs. How did I know it was going to go this way. Classic. No errors yet but they are downloading at under 100KB/s, even slower than my bored band upload rate. I think my remaining WAH2 tasks might finish before they all download and a couple are yet to start! Edit: I spoke too soon. Quite a few files have downloaded but a growing smattering of files that have partially downloaded as well. Hopefully they will download before they time out!
29) Message boards : Number crunching : Connection and Download issues Oct24 Message 71813 Posted 4 Nov 2024 by Dave Jackson	All the ones I've had like this do eventually sort themselves out. That is my experience too. If you enable http debug do you get something like "locked by file upload handler?" That happens when something has interrupted the upload of the file. I don't know what the backoff time on the server is before it allows you to resume the upload but I have had a number of occasions when it has been several hours.
30) Message boards : Number crunching : Connection and Download issues Oct24 Message 71811 Posted 4 Nov 2024 by Dave Jackson	Still have 1 task that can't upload the final _out.zip, gets as far as 1.31/4.75 MB, log says transient HHTP error. Has that last out.zip cleared? As your computers are hidden I can't check anything. (Not a request to unhide them, just an explanation.)
31) Message boards : Number crunching : Connection and Download issues Oct24 Message 71810 Posted 4 Nov 2024 by Dave Jackson	Of mine, 4 have completed. The rest I have overtaken the original machine or am very close to having done so. I am going to suspend the ones that have completed but suspect Glen will suggest deleting them. The only reason I can think of for letting them complete would be if someone wanted to compare results on different architecture machines.
32) Message boards : Number crunching : Almost 2025. Why doesn’t this project support multithreading? Message 71805 Posted 3 Nov 2024 by Dave Jackson	There are multi-threaded tasks on the way. They will be the OIFS code from ECMWF. Glenn posted a link to the program for Linux that you can run in a terminal on Linux recently and play around with the file that determines how many cores it uses. Most of the work however is still the Met Office code which is all 32bit which says something about its age in itself. Initially at least the multithreaded apps will like the other OIFS work all be Linux only. The answer to your query however is that the project hasn't had a scientist with the required programming skills to write multi-threaded apps till Glenn came on board as a volunteer scientist. I don't know whether Andy who does the sysadmin work has the skills but his other work for the project is full time and would not allow him the space to develop the multithreaded apps. This is the link to the thread with discussion about the multi-core app and the link to try it if you have access to a Linux box. Interestingly, I would say that less than a quarter of the projects I have looked at have multithreaded apps and even those that do, don't allow it on all task types. Edit: If I have missed anything important I am sure another moderator or Glenn will add to this but I can't imagine it is straightforward to modify the Met Office code to make it multithreaded even if that is possible and that is the code most of the scientists around the world are still using for their tasks.
33) Message boards : Number crunching : Connection and Download issues Oct24 Message 71802 Posted 2 Nov 2024 by Dave Jackson	Most of mine are eas tasks that have timed out on other machines. I got a chunk of these too but it looks like almost all of them will be finished by the original users way before I can finish them. I'm going to suspend them instead of spending time on them for likely no benefit. I looked at three of the machines that had been running these tasks. I am pretty sure most if not all of those I have will finish first on my machine. All three of the machines I looked at have well over 50% error rate as well so there is some doubt whether they would ever finish on the original machines.
34) Message boards : Number crunching : Connection and Download issues Oct24 Message 71800 Posted 1 Nov 2024 by Dave Jackson	Are all new work units already gone? According to the server status page they have. Most of mine are eas tasks that have timed out on other machines.
35) Message boards : Number crunching : Connection and Download issues Oct24 Message 71798 Posted 1 Nov 2024 by Dave Jackson	Now boinc in VM is working as well as in WINE. Still would like to understand why they behaved differently though.
36) Message boards : Number crunching : Connection and Download issues Oct24 Message 71791 Posted 1 Nov 2024 by Dave Jackson	Firefox on Ubuntu takes me to cpdn.org and displays the climateprediction stuff ! On my Ubuntu box if I type in cpdn.org, it takes me to www.cpdn.org and the page says welcome Dave and how much I have contributed to the project but only once I have accepted the risk and continued. On Chromium, it takes me to cpdn.org and the page Richard refers to. I wonder if the difference in how the two browsers interpret cpdn.org is somehow mirrored in the difference between how my BOINC instances in WINE and in a VM deal with things? Edit: Another trickle has gone through at 12:45 and two more tasks have downloaded and started running. Edit2: Just noticed all but one of the now 16 tasks I have are from 1021 and only one from 1028.
37) Message boards : Number crunching : Connection and Download issues Oct24 Message 71786 Posted 1 Nov 2024 by Dave Jackson	Interesting, my 8.0.2 on Win 8.1 can't connect at all but your 8.0.4 on Win 11 can. Wonder where the subdomain ignorance is located. It isn't really Win 11 but WINE on a Linux box pretending to be Win11. A further six tasks have just downloaded and started. Same BOINC version in Windows10 in a VM isn't able to get any work.
38) Message boards : Number crunching : Connection and Download issues Oct24 Message 71784 Posted 1 Nov 2024 by Dave Jackson	Zips yes, trickles no. You have a Contact time for today for your host in your account page ? Last contact I had from any host was yesterday lunchtime. 01 Nov 2024 08:32:28 1552038 22521599 wah2_nz25_10hr_209105_25_1028_012344566_0 57,899 62,420 1.0781 Credit is consistent with the number of trickle up messages the task page says have been sent too.
39) Message boards : Number crunching : Connection and Download issues Oct24 Message 71780 Posted 1 Nov 2024 by Dave Jackson	ps. Dave, the number of tasks IS changing on the server status, but it's going up not down. I should have written down the number before going to bed! That would be failed tasks being reissued. I notice my zips and trickles both seem to be going through without issue at the moment albeit, I only have one task running.
40) Message boards : Cafe CPDN : WCG African Rainfall Project (ARP) restart update Apr 25, 2024 Message 71779 Posted 1 Nov 2024 by Dave Jackson	Yesterday evening I got two ARP tasks that completed OK. Over on the BOINC fora I read these are testing ones before they properly relaunch ARP. It gave me the unusual experience downloading them of my bored band not being the bottleneck!