Jonathan only cleared 750 Gigs of space over the weekend. He's currently looking for a cupboard with some spare shelf space to store some more. Data is stacked up everywhere. Probably have to buy some buckets for it. :)

---

Backups: Here

Buckets that size don't come cheap.

At SETI@home volunteers are donating dozens of 1TB and 2TB disks to store data.
Tullio

---

At SETI@home volunteers are donating dozens of 1TB and 2TB disks to store data.


I don�t know if this would work with this project. How do you guard against data loss.

When you say that they �donate� I assume that the drives remain in the homes of the donor. Home, non-commercial quality HD�s are not know for there overwhelming reliability. I had a 2TB external backup drive fail only a few months ago. One moment it worked, a few hours later it didn't. No warning. Also what happens if a person who has project data just suddenly stops participating.

One thing that can be said for CP is that despite all our server problems we have NEVER LOST DATA!

---

No, by donating Tulio means buying them and sending them on to Berkely.

All servers at Oxford, and there are many different departments, with server rooms and IT sections, would most likely be under a service contract.
And crunchers don't need to know all the 'behind the scenes' details and plans.

---

Backups: Here

All servers at Oxford, and there are many different departments, with server rooms and IT sections, would most likely be under a service contract.
And crunchers don't need to know all the 'behind the scenes' details and plans.

Also, there are various upload (and download) servers worldwide both for current wu and for the database of completed results.

So it's not just "distributed computing" - it's "distributed database"

Like JIM posted - no uploaded results lost in 8 years.

It's possible to build fairly reliable systems from consumer-grade discs - but takes a lot of planning, design and maintenance. Donating cheap hardware to the project might help, probably not - don't know what SETI is doing. There's lots of information on the web on how to do it - but the devil is in the details. And the work-hours of maintaining such a thing is -- done that- don't want to again - retired.

Like Les said -- I don't want to know the details -- because I've been there - and second-guessing future storage improvements and estimated total costs and all is a total brain-bender and management always complains anyhow no matter how hard you work to design and build a thing that will be obsolete before the Board of Directors signs off.

Thanks to the crew for keeping things going mostly, and for not losing any uploaded data.

---

There is a donate button http://climateprediction.net/content/donations on the main project page that could probably do with more publicity. I suspect that more people working for the project might be a higher priority than extra hard drives but as Les says, us crunchers don't need to know all the details and if we did we would probably be overwhelmed!

Dave

There is a donate button http://climateprediction.net/content/donations on the main project page that could probably do with more publicity. I suspect that more people working for the project might be a higher priority than extra hard drives but as Les says, us crunchers don't need to know all the details and if we did we would probably be overwhelmed!

Dave

Overwhelmed - no way

If the project could pay me a lousy USD 120000 per year and give me another few million for hardware I could fix all their problems (add a few consultants on the database side) I'd even come out of retirement!

Might try the "donate" button

---

Totally understand!

Dave

No, by donating Tulio means buying them and sending them on to Berkely.

Correct. It is the GPU User Group, that is those using graphic cards to accelerate their processing, that sponsors donations, orders disks and also servers, and sends them to the Space Sciences Laboratory. I think it is the only BOINC project where this happens.
Tullio

---

No, by donating Tulio means buying them and sending them on to Berkely.

Correct. It is the GPU User Group, that is those using graphic cards to accelerate their processing, that sponsors donations, orders disks and also servers, and sends them to the Space Sciences Laboratory. I think it is the only BOINC project where this happens.
Tullio

Got any more info - or link? sounds possibly useful.

---

No, by donating Tulio means buying them and sending them on to Berkely.

Correct. It is the GPU User Group, that is those using graphic cards to accelerate their processing, that sponsors donations, orders disks and also servers, and sends them to the Space Sciences Laboratory. I think it is the only BOINC project where this happens.
Tullio

Got any more info - or link? sounds possibly useful.

GPU User Group - www.gpuug.org

Users can donate towards a specific purpose, or they can buy a drive, or even donate directly to the project. The last one is done via Paypal and at the end of the month the project gets a payment less Paypal fees. UC Berkeley aren't allowed a Paypal account themselves.

I did ask Jonathan if he could tell me how much a 2Tb drive costs in the UK so I could work out a suitable donation but he hasn't provided me with any information (probably rather busy I expect). If he or someone else could tell us that and how many drives they need we could work towards that goal. The idea is to have smallish goals for specific items, something achievable.

---

BOINC blog

It's probably university policy to not discus money matters with people not working for the uni. A commercial-in-confidence type of thing.

And a hard disk isn't of much use without a server to run it.
And servers need rack space, and power.

---

Backups: Here

It's probably university policy to not discus money matters with people not working for the uni. A commercial-in-confidence type of thing.

And a hard disk isn't of much use without a server to run it.
And servers need rack space, and power.

All very true. Servers paddym and georgem were also donated. Rack space and power were not. But the SETI@home devs/admins provided them. They are also volunteers for SETI@home besides doing work for UC Berkeley.
Tullio

---

It's probably university policy to not discus money matters with people not working for the uni. A commercial-in-confidence type of thing.

And a hard disk isn't of much use without a server to run it.
And servers need rack space, and power.

I was assuming they would be replacing existing drives with something that would be newer (and theoretically more reliable) as well as possibly giving them more space, depending on what size drives they are replacing. A rough idea of how much drives cost (recommended retail price) and how many they want/need to replace would have been helpful. If that is too much information then how can they expect us to help?

---

BOINC blog

It's probably university policy to not discus money matters with people not working for the uni. A commercial-in-confidence type of thing.

And a hard disk isn't of much use without a server to run it.
And servers need rack space, and power.

I was assuming they would be replacing existing drives with something that would be newer (and theoretically more reliable) as well as possibly giving them more space, depending on what size drives they are replacing. A rough idea of how much drives cost (recommended retail price) and how many they want/need to replace would have been helpful. If that is too much information then how can they expect us to help?

Searching the web -- SAS drives in the 300 GB capacity range at 15k rpm are running a few hundred dollars each - a bit less if you buy case lots. 600GB drives in this speed and reliablity range are a bit more expensive per TB.

Consumer grade 1-2 TB drives are cheaper by far per TB but need much more expertise and connectivity and replication to make them competitive for enterprise reliability needs. And less than half the read speed and even less seek speed. So you need a database analyst and some serious testing to compare the multi-redundant SAS to the even-more-redundant cheap disks you would need to mimic the speed and reliability of the "server-grade" disks.


What I'm saying is -- speed, reliability, redundancy -- takes a lot of work to figure what's best for any particular application.

Not to mention connectivity - you want dual-port SAS drives so when one network server fails the backup system works -- another couple hundred per drive - and consumer-grade 2 TB drives don't even offer this option.


It's not about replacing a few drives with newer cheaper ones.
It's about building a reliable replacement system or 2 or 3

---

What you say is true for data that must remain readily accessible, Eirik. But it seems to me (from the outside) that CPDN's main requirement is for somewhere to put data that no-one has wanted during the last few months, and that is unlikely to be wanted for the next few months or years -- but it might be wanted sometime. Most likely, when a scientist does want it, they'll be able to give plenty of notice.

Back in the day, IBM used to sell the concept of tiered storage: on-line, near-line and off-line. The idea was that 'hot' data would stay on the on-line storage, and when people stopped accessing it it would migrate to progressively less responsive (but cheaper) storage.

Of course IBM sold fancy systems to 'migrate' unneeded data automatically. But I don't think CPDN needs that. It does need some kind of systematic archiving process, though.

I'd caution that archiving is an ongoing process, not a one-time event, and resources should be allocated and processes set up accordingly.

For non-critical data such as CPDN run results, two copies on consumer-grade storage, kept in separate file store-rooms in separate buildings in separate campuses and tested annually, should provide enough of a guarantee of future accessibility.

100 TB of non-critical offline storage is then some checksum files, a hard-back book, a label maker, 100+ 2TB disks and a USB3 dock, and two cupboards -- plus a high-school student volunteer for a few weeks each year (to stock-take and checksum the archives, replace any failed disks and archive new data). And the instructions for the student.

The data is permanently on line, and can be accessed via this page.
Each model completed, is also linked to the results pages, via a line at the bottom of each model's page.

It needs to also be remembered that there's no 'cpdn section' at Oxford uni.
The research is a research project of the Atmospheric, Oceanic and Planetary Physics department.
The 2 "programmers" are IT specialists / programmers who, with others, work for the Oxford e-Research Centre.

The Oxford e-Research Centre works with research units across the whole of Oxford University to enable the use and development of innovative computational and information technology in multidisciplinary collaborations.

As such, it can be assumed that they know about many things, including large data bases, and various storage schemes. In fact, I can vaguely recall reading the job specs for one of these positions a few years ago, which talked about these same things as part of the job requirements.

---

Backups: Here

I wonder if the BOINC volunteer storage, if they ever get it completed, would be useful here. I would post a link but the Akismet anti-spam is so paranoid I can't link to it. Suffice to say its at:

boinc dot berkeley dot edu slash trac slash wiki slash VolunteerStorage#

---

BOINC blog