Thanks Glenn. It makes a big difference having someone on the inside who actually knows what is happening.

Hope you recover soon Dave! At least the weather is rubbish at the moment.

p.s. Don't assume I know what's happening, only bits of it!

Obviously, applications need data to work on, and that will change with every task. There might be scope for splitting the data up into two types: a basic climate model framework (which would be mostly unchanging), and a variable set of perturbations or parameters to define each separate task. The Einstein project has lots of experience of this system for their Gravity Wave application: users download a large initial group of recorded data files from the Hanford and Livingstone detectors, and keep it for potentially hundreds of tasks until all possible searches have been completed. Their server keeps track of what data each computer has downloaded, and tries to minimise downloads by reusing it for as long as possible, before finally deleting it and moving the machine on to a new block. I'm sure Einstein would help with the initial setup, but that's probably a discussion for another day.

I thoguht Boinc did this automatically? For example, if CPDN task A asks for files 1, 4, 7, they get downloaded. If CPDN task B next week, after task A has completed, asks for files 1, 8, 13, file 1 is already there. I guess it's not an overall cache, as 4 and 7 won't be kept. Each file must be marked as one-off or reusable. But nothing fancy, just signalling to Boinc to keep it for later.

I was using squid to cache everything downloaded through Boinc for 8 PCs. But it broke. It just stopped working, won't reinstall, won't repair, won't uninstall, nothing. I guess I could try a better made piece of software, but it was only helping out LHC really, where they don't do the above, since they don't seem to know how with everything being inside a seperate VB virtual machine. It should be possible, I thought VB allowed access to the Boinc directory on the host?

I thoguht Boinc did this automatically?

Yes, most of the hooks are there, but it's a "path less travelled". I don't think than anyone except Einstein staff - past and present - have any direct experience of using such a system. It relies on something called a "locality scheduler" (specially written for Einstein by Prof. Bruce Allen), which allocates tasks to hosts more exactly according to the data already held by the host: which in turn, brings into play concepts like "sticky" data files, which means a file which isn't automatically deleted at the end of a job.

The downside is the locality scheduler has a harder job, and takes longer to do it, than the standard scheduler. I got bitten by that a couple of years ago, when a new Beta app started expiring all my tasks on the exact second they were (re-)issued. To cut a long story short - Bernd acknowledged that it was a server coding error, but couldn't devote the time needed to discovering and fixing it - so he invited me to have a private poke in their server code repository. Fortunately (phew!), I was able to discover the logical error causing my tasks to be abandoned, and armed with that, Bernd was able to code a fix overnight. So the working locality server code at Einstein definitely differs from the BOINC standard - that, and their willingness to accept collaborative working, are why I suggested asking them for a hand-up if the idea is ever considered for deployment here.

I've seen others do it, certainly with the executable code, I assumed they were doing it with data too. In fact I'm sure LHC does. If I run a computer on LHC which hasn't been on it for a while, a newer dataset gets downloaded.

Anyway it's not too far to stretch that to data files surely? Even if it doesn't get ones suited to that file, if it's a commonish file, it'll help to keep it.

I must confess that I must have confused those 64bit applications with the other apps that need those 32bit libs.
I did think for a moment that the 64 bit ones had been statically linked.
It would be far better if those 32 bit jobbies could be re-compiled for 64 bit. It's only the fedora 32 that I have those extra libs installed. Whilst I have vbox installed on fedora 36 I failed to get those 32 bit libs installed. So I had better remember which climate w/u's are to be worked on and not download them to the wrong machine.

Anyway, I'm sure the world will keep turning.

I did think for a moment that the 64 bit ones had been statically linked.

They are statically linked. I do not know if they need to be.

[/var/lib/boinc/projects/climateprediction.net]$ file oifs_43r3_*

oifs_43r3_1.21_x86_64-pc-linux-gnu:            ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped

oifs_43r3_bl_1.11_x86_64-pc-linux-gnu:         ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped

oifs_43r3_ps_1.09_x86_64-pc-linux-gnu:         ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=cbdd0bee9e93a37522513987a51e4f872cc7a883, for GNU/Linux 3.2.0, with debug_info, not stripped

---

I did think for a moment that the 64 bit ones had been statically linked.

They are statically linked. I do not know if they need to be.

I believe the conversation about statically linked & 32bit libraries is for the MetO models (hadsm4 and friends). It doesn't apply to OpeniFS executables.

As far as handling data files is concerned, I think the simpler boinc behaves the better. There's a marginal advantage is reducing file download compared to a greater disadvantage in code complexity, which just becomes another potential piece of code no-one is maintaining. Besides which, the upload data size is considerably more than the download package size, that's really where time needs to be invested to improve the client experience.

Besides which, the upload data size is considerably more than the download package size, that's really where time needs to be invested to improve the client experience.

The OIFS tasks are the only ones where upload speed restricts the number of tasks I can run. At some point Open Reach will upgrade our estate and then things should be fine. An upload speed of 1MB/second would be more than enough to stop files backing up. I can get that via my phone on 4G, the only issue being that using 4G, after about 20 tasks my free data runs out.

The researcher for the EAS tasks has discussed the results with her professor and there were some concerns about the spin up results but they have decided everything is within range and the mainsite tasks will be released, "very soon."

Looking forward to it. Which app would that be?

---

The OIFS tasks are the only ones where upload speed restricts the number of tasks I can run. At some point Open Reach will upgrade our estate and then things should be fine. An upload speed of 1MB/second would be more than enough to stop files backing up. I can get that via my phone on 4G, the only issue being that using 4G, after about 20 tasks my free data runs out.

I've already got 6.5Mbit upload, but that gets maxed out with one 24 core Ryzen running CMS on LHC. Not sure how bad it would be here since I've never received many tasks.

What I'm waiting on is 220Mbit upload. But it's been put off 2 years due to a mess of green boxes round here. I'm on the corner of a town and connected to the next town!

For £35 a month Giffgaff will give me unlimited data. Not sure if they detect and prevent that getting tethered onto PCs. At the moment I'm on their 1GB a month £6 plan!

Edit: looks like they let you:

"If you have an ‘Always On’ allowance, then after 80 GB of data used you'll experience a reduced data speed of 384kbps from 8am to Midnight."

So Boinc would send loads at night, no problem.

"Avoid regularly tethering to 12 or more devices or regularly using unreasonably large amounts of data, including but not limited using around 650GB of data twice within a 6-month period."

Ok so 8 PCs is ok. 1300GB in 6 months, hmmm.... I could send 250GB in a day with LHC and all my computers on CMS at once. So 39TB in 6 months. They might notice.

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

[Large smile] 112 cores waiting.... Is there a way to convert "very soon" into metric units?

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

Too bad: my windows machine is a pipsqueak. ID: 1512658

And myLinux-only main machine is pretty serious. ID: 1511241

Both are connected to the Internet by a single router that can provide 75 Megabits/second. But the CPDN web site reports

Average upload rate 	235.44 KB/sec
Average download rate 	16983.62 KB/sec

Speedtewst reveals

Timestamp 	  Download   Upload 	Latency Jitter  Quality Score 	Test Server
6/7/2023 20:3:31  78.13 Mbps 63.66 Mbps 6 ms     1 ms       Excellent   ny2.speedtest.gslnetworks.com.prod.hosts.ooklaserver.net
5/5/2023 11:23:28 76.26 Mbps 89.16 Mbps 6 ms     1 ms       Excellent   speedtest.nyc.rr.com

---

Too bad: my windows machine is a pipsqueak. ID: 1512658

And myLinux-only main machine is pretty serious. ID: 1511241

Can you run a Windows emulator like I run a Linux Oracle Virtualbox?

Both are connected to the Internet by a single router that can provide 75 Megabits/second. But the CPDN web site reports

Average upload rate 	235.44 KB/sec
Average download rate 	16983.62 KB/sec

Speedtewst reveals

Timestamp 	  Download   Upload 	Latency Jitter  Quality Score 	Test Server
6/7/2023 20:3:31  78.13 Mbps 63.66 Mbps 6 ms     1 ms       Excellent   ny2.speedtest.gslnetworks.com.prod.hosts.ooklaserver.net
5/5/2023 11:23:28 76.26 Mbps 89.16 Mbps 6 ms     1 ms       Excellent   speedtest.nyc.rr.com

I have a feeling Boinc uses Mb and MB incorrectly (or was that Boinctasks?) If it's right, that number is impossible, because 16983.62KB/sec =
16.59MB/sec = 132Mbps, or 149Mbps using parity? Definitely not possible through your router getting up to 90.

Can you run a Windows emulator like I run a Linux Oracle Virtualbox?

The only emulator I ever tried was WINE and it never worked for me. Besides, technically, I would need a Windows license to do it and I do not want to.
The onlyt reasoin I have my pipsqueak Windows machine is to run TaxAct for my income tax, and Garmin Express for my GPS. But since it is there anyway, I got Boinc for it.

Both are connected to the Internet by a single router that can provide 75 Megabits/second. But the CPDN web site reports

Average upload rate 235.44 KB/sec
Average download rate 16983.62 KB/sec


Speedtest reveals

Timestamp Download Upload Latency Jitter Quality Score Test Server
6/7/2023 20:3:31 78.13 Mbps 63.66 Mbps 6 ms 1 ms Excellent ny2.speedtest.gslnetworks.com.prod.hosts.ooklaserver.net
5/5/2023 11:23:28 76.26 Mbps 89.16 Mbps 6 ms 1 ms Excellent speedtest.nyc.rr.com

I have a feeling Boinc uses Mb and MB incorrectly (or was that Boinctasks?) If it's right, that number is impossible, because 16983.62KB/sec =
16.59MB/sec = 132Mbps, or 149Mbps using parity? Definitely not possible through your router getting up to 90.

Back when someone made a similar complaint, I looked at it and it is not so simple. IIRC, the CPDN upload speed might be correct, but manifestly, their download speed is off,ny quite a bit. My ISP promises 75 megabits up and down, and their upload speed is generally faster than that. I can watch it and it may go 10% faster than the indicated speeds for very brief intervals, but surely not 3x faster. But it is not just a confusion between bits and bytes, Or Kilo and Mega. It is just plain wrong.

---

Back when someone made a similar complaint, I looked at it and it is not so simple. IIRC, the CPDN upload speed might be correct, but manifestly, their download speed is off,ny quite a bit. My ISP promises 75 megabits up and down, and their upload speed is generally faster than that. I can watch it and it may go 10% faster than the indicated speeds for very brief intervals, but surely not 3x faster. But it is not just a confusion between bits and bytes, Or Kilo and Mega. It is just plain wrong.

Maybe something is getting buffered between them and you? Back when my squid cache actually worked, Boinc was reporting speeds of 800Mbit. I have a 32Mbit line. The 800Mbit was between a garage computer and the house computer through a 40Gbit cable, when it wanted to download a data set another machine had already downloaded.

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

Will they still be just for older versions of Windows? The Applications page states that they are for Windows 2000 - Windows 8.

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

Will they still be just for older versions of Windows? The Applications page states that they are for Windows 2000 - Windows 8.

As far as the Tasks will run on all versions of Windows after Windows 2000 including Win 10 and Win 11.

Looking forward to it. Which app would that be?

Weather at Home Windows tasks.

About how much RAM per task will be required?

As far as the Tasks will run on all versions of Windows after Windows 2000 including Win 10 and Win 11.

Of course they will. Windows will almost always run older software, unlike Android and Linux.

About how much RAM per task will be required?

RAM is dirt cheap nowadays. I've got my 24 core machines up to 96GB.