Hi Richard, yes, I wondered about that, I remember talking with Andy recently. The final upload appears synchronous - the code waits until it finishes? But the trickles are asynchronous?

It's a very good idea. But I am not sure why this would cause the control code to fail? Isn't the trickle handed over to the client so it's no longer the wrapper's responsibility?

Both the trickles and the final upload are for the scientist. Both contain model output files. Think of the final upload as the final 'trickle'.

It would seem to me that the application/model that is doing all the work has done a splendid job and got all the right answers and sent them to the server but at the very last moment the wrapper has indicated a problem of its own that marks all that work and the task done as invalid.

Agreed. I've asked CPDN if there is a way of getting the server to check the upload was received OK to reclassify this as a success. It may not be easy as the uploads go to a cloud server first. Not my expertise.

I have a theory on this, which may suggest where to look.

BOINC clients communicate with projects on two completely separate levels. There are very simple-minded file copying processes, both download and upload, which simply move complete files form one place to another, with no information on or interest in what those files might contain. Those files belong to the project's scientists. And separately, BOINC clients communicate with the project's "scheduler" about the administration and inner workings of BOINC itself.

CPDN is unusual in producing intermediate returns of both types: 'uploads' go to the scientists, and 'trickles' go to the administrators. Most projects only communicate once, when the task has completely finished, and BOINC is careful to wait until the data transfer has completed, before finalising the administration.

My suspicion is that the completion of intermediary upload file transfers isn't checked so carefully by the client. Some volunteers with powerful computers but slower data lines may still have intermediate files chugging along when the rest of the task has completed, and I'm worried that the final administrative 'all done' message may be sent too early in those cases.

Next time I get some tasks, I'll trying holding some files back and studying the logs, to see what order the various 'finished' processes are handled in.

Normally, with the old Hadley models for example, the final upload file is indeed treated as an integral part of the 'wrap up and tidy up' administration package: in the monitoring tool that I use, I see the task proceed through "Running - Uploading - Ready to report" in strict order. But the intermediate upload files associated with tickles don't trigger an 'uploading' status. I'm not entirely certain that the final IFS file is treated as an 'on completion' file. If we could have another batch of tasks next week (nudge, nudge, wink, wink ..), that's one of the things I'd try to check.

Richard, I think you've hit it. Consider: the trickle files are zipped and stored in the 'tmp' folder in the projects/climateprediction.net directory (not in the slot dir). The controlling wrapper does a cautious sleep of 2 mins after it initiates the final upload and then it proceeds to delete this 'tmp' folder. If the network is down/slow, might mean the trickle zips get deleted too.

The question is whether the final boinc_upload() call blocks until all the trickles have gone?

Or, does the boinc_finish() call block until the trickles have gone? I think the code calls boinc_finish() AFTER it deletes the tmp dir, not before. This is probably where the error is but the solution might depend on knowing what boinc_finish() does with regard to the trickles. Or do we need to add code to track the upload of each trickle?

That's it though, thanks.

1) The last child process running the model completes
2) The final set of output files in the task directory below the project directory get zipped up, an intermediate 8 character file is created (begins with 'zi') in the project directory that grows and gets renamed to be the final ZIP file (very-long-name_14.zip)
3) The event log shows the upload starting and after a little while the upload being completed successfully
4) The event log shows a [slot] debug message that the very-long-name_14.zip has been removed
5) 2 more lines in the event log show [slot] debug messages that very-long-name_14.zip.gz and very-long-name_14.zip.gzt are also removed - neither of these two files are present in the project directory. There is no error message to say that these files couldn't be found to be deleted in the event log.
6) The slot files themselves are then removed, each with a debug message, about 1,950 of them.
7) The slot itself is deleted.

Is the absence of the .zip.gz and .zip.gzt files the reason an error 9 is raised ?

I can probably go and look to see if a working (not got a 9) task had those as well...

Edit: Don't think it is that. After every upload the [slot] debugs shows it trying to delete all 3 types of .zip file. Have only witnessed one final .zip file being created and not the other 2 files at the same time, but probably happens for all zips for all tasks (0 or 9).

What I find interesting is both these only seem to happen on AMD hardware. I didn't do an exhaustive trawl through the logs but I could not find a single intel machine with these fails. My suspicion is that both these errors are memory related. 'double free' corruption obviously is, the error code 9 with the 'free()..' error could also refer to a memory resident file, but quite what I am not sure. Both codes were compiled on Intel with the latest Intel compiler. Whether there's additional compiler options required I don't know. I have not been able to reproduce these errors on my little AMD box.

It's possible AMD chips are triggering memory bugs in the code depending on what else happens to be in memory at the same time (hence the seemingly random nature of the fail). Hard to say exactly at the moment but it could also been something system/hardware related specific to Ryzens. I have never seen the model fail like this before on the processors I've worked with in the past (none of which were AMD unfortunately). I am tempted to turn down the optimization and see what happens....

In case you care, my machine is Intel, not AMD. And it runs Linux. And it has never failed with the oifs_43r3_ps_1.01_x86_64-pc-linux-gnu stuff, the oifs_43r3_ps_1.05_x86_64-pc-linux-gnu stuff, nor the 43r3_bl_1.07 ones either.

Read all about it here:

Computer 1511241
Computer information
IP address 	Show IP address
Domain name 	localhost.localdomain
Local Standard Time 	UTC -5 hours
Created 	14 Nov 2020, 15:37:02 UTC
Total credit 	6,420,965
Average credit 	808.68
Cross project credit 	BOINCstats.com Free-DC
CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16
Coprocessors 	---
Virtualization 	None
Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.6 (Ootpa) [4.18.0-372.26.1.el8_6.x86_64|libc 2.28]
BOINC version 	7.20.2
Memory 	62.28 GB
Cache 	16896 KB
Swap space 	15.62 GB
Total disk space 	488.04 GB
Free Disk Space 	478.01 GB
Measured floating point speed 	6.05 billion ops/sec
Measured integer speed 	24.32 billion ops/sec
Average upload rate 	3768.45 KB/sec
Average download rate 	4430.72 KB/sec
Average turnaround time 	0.63 days
Application details 	Show
Tasks 	392
Number of times client has contacted server 	3664
Last time contacted server 	18 Dec 2022, 19:24:40 UTC
Fraction of time BOINC is running 	99.97%
While BOINC is running, fraction of time computing is allowed 	100.00%
While is BOINC running, fraction of time GPU computing is allowed 	100.00%
Task duration correction factor 	1

---

Or, does the boinc_finish() call block until the trickles have gone? I think the code calls boinc_finish() AFTER it deletes the tmp dir, not before. This is probably where the error is but the solution might depend on knowing what boinc_finish() does with regard to the trickles. Or do we need to add code to track the upload of each trickle?

If code needs adding, it should be added to BOINC generically, not just CPDN. I know how to ask for that, but we need to be sure of our ground first.

This bit...?

          fprintf(stderr,"Uploading the final file: %s\n",upload_file_name.c_str());
          sleep_until(system_clock::now() + seconds(20));
          boinc_upload_file(upload_file_name);
          retval = boinc_upload_status(upload_file_name);
          if (!retval) {
             fprintf(stderr,"Finished the upload of the final file\n");
          }
	       
	  // Produce trickle
          process_trickle(current_cpu_time,wu_name.c_str(),result_base_name,slot_path,current_iter);
       }
       boinc_end_critical_section();

[other skipped code]

    // Now task has finished, remove the temp folder
    std::remove(temp_path.c_str());

    sleep_until(system_clock::now() + seconds(120));

    // if finished normally
    if (process_status == 1){
      boinc_end_critical_section();
      boinc_finish(0);

Never see trickles so no idea how long they can take.
Also 2 boinc_end_critical_section's are performed ? [No idea what it does yet]
Understand I'm looking at old code.

Glenn, meant to say thanks for your earlier detailed explanation.

Hi, I can answer some of this. As it's getting technical maybe the moderators might want to move it to a separate thread?

With the help of others here, we know that 'double free' corruption seems to be symptomatic of the model itself failing. The fail with error code 9 happens after the model has finished as you say, and sometimes with the 'free()...' message as well.

What I find interesting is both these only seem to happen on AMD hardware. I didn't do an exhaustive trawl through the logs but I could not find a single intel machine with these fails. My suspicion is that both these errors are memory related. 'double free' corruption obviously is, the error code 9 with the 'free()..' error could also refer to a memory resident file, but quite what I am not sure. Both codes were compiled on Intel with the latest Intel compiler. Whether there's additional compiler options required I don't know. I have not been able to reproduce these errors on my little AMD box.

It's possible AMD chips are triggering memory bugs in the code depending on what else happens to be in memory at the same time (hence the seemingly random nature of the fail). Hard to say exactly at the moment but it could also been something system/hardware related specific to Ryzens. I have never seen the model fail like this before on the processors I've worked with in the past (none of which were AMD unfortunately). I am tempted to turn down the optimization and see what happens....

I did a little bit of searching and found 3 tasks that failed with errors you described on intel processors. I think it might be too early to attribute these errors as ryzen specific.

https://www.cpdn.org/result.php?resultid=22245369
Exit status 5 (0x00000005) Unknown error code
double free or corruption (out)

https://www.cpdn.org/result.php?resultid=22248281
Exit status 9 (0x00000009) Unknown error code
double free or corruption (out)

https://www.cpdn.org/result.php?resultid=22245144
Exit status 9 (0x00000009) Unknown error code
double free or corruption (out)

I did a little bit of searching and found 3 tasks that failed with errors you described on intel processors. I think it might be too early to attribute these errors as ryzen specific.

https://www.cpdn.org/result.php?resultid=22245369
Exit status 5 (0x00000005) Unknown error code
double free or corruption (out)

https://www.cpdn.org/result.php?resultid=22248281
Exit status 9 (0x00000009) Unknown error code
double free or corruption (out)

https://www.cpdn.org/result.php?resultid=22245144
Exit status 9 (0x00000009) Unknown error code
double free or corruption (out)

I looked at those and noticed they were all running old versions of the boinc client. I do not see why that would cause these problems though. It could cause problems some have downloading new work. Some users of Rosetta are having this. Even worse for me because my client is up-to-date and still does not work. But I am having no trouble with CPDN other than lack of work units.

---

Have now read the BasicApi wiki about boinc_end_critical_section :)

One more thing that doesn't seem right (to me !), there is code to delete the files, from the project sub-directory, that get added to the zip.
I'm assuming only the files listed in the stderr log are being added to the zip file ?
There is also a progress file kept in there (something like progress_task-id.xml), if that isn't added to the zip and therefore not deleted in that process does that mean when the directory is removed it creates an error ?
std::remove is expecting an empty directory, but I've no idea if it would still delete the directory and generate a 9 error, the directory does not remain for a 9 task.
I can see the progress file being created but never deleted, unless it is included in the zip file.
The file isn't in the sub-directory all the time.
The only explanation I can think of is that it is zipped and hence deleted but not shown in the stderr log as being added.

Or have I missed something !

Mr. P Hucker wrote:

I take it all these problems mean you haven't even considered starting on virtualbox to get windows machines running it? I've got 126 cores sat waiting....

Once they get around to implement this, keep an eye on two resource constraints:

a) RAM footprint per task. I don't know if vboxwrapper supports "memory ballooning" by now or still needs to have a static amount of RAM allocated for the whole duration (which needs to be larger than the peak RAM demand of the scientific application).

b) Rate of upload data generated per task. I recall from our earlier discussion that you have got a 7 Mbit/s uplink, while I happen to have 8 Mbit/s. With the OIFS batches issued so far, ~30 tasks running at once would saturate my uplink with uploads. (Fewer in case of service interruptions or in case of issues with CPDN's upload server.) You have got almost the same upload bandwidth, and several faster/ some slower cores than I got. Hence, if previous OIFS would be ported 1:1 to Windows, you could run about ~25 tasks at once (total across all your computers if they are all on the same link), before running into an upload bottleneck, as a rough estimation.

Both resource requirements are of a new magnitude compared to those of the work types which CPDN issued in the past.

It's possible AMD chips are triggering memory bugs in the code depending on what else happens to be in memory at the same time (hence the seemingly random nature of the fail). Hard to say exactly at the moment but it could also been something system/hardware related specific to Ryzens. I have never seen the model fail like this before on the processors I've worked with in the past (none of which were AMD unfortunately). I am tempted to turn down the optimization and see what happens....

I did a little bit of searching and found 3 tasks that failed with errors you described on intel processors. I think it might be too early to attribute these errors as ryzen specific.
https://www.cpdn.org/result.php?resultid=22245369
Exit status 5 (0x00000005) Unknown error code
double free or corruption (out)

Ah thanks. That's useful, and the first time I've seen a error code 5 (I/O error) but consistent with something file related. Frustrating there is no traceback from the model - which is why I didn't think it was the model in the first place.

I can spend time looking at this but it's the law of diminishing returns. The OpenIFS batch error rate is now 10%, HadSM4's is about 5%. It would be nice to get it lower but I also need to move the higher resolution work on which I've not been able to start yet. This is quite important to attract scientists to the platform.

A bit academic as the windows ones won't be coming till a few more of the problems have been ironed out and thoroughly tested. The drive for it will come if they start needing much bigger numbers processed by crunchers at a time.

Just had another task fail with status 9 but also had "free(): invalid pointer" at the end of the stderr file...

Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
15:13:12 (1519724): called boinc_finish(0)
free(): invalid pointer 

At the end of the run there were 2 trickle files present in the project directory, the earliest one from 14:43. The latter one appeared very near the end of the run.

-rw-r--r-- 1 boinc boinc 209 Dec 19 15:11 trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671462672.xml
-rw-r--r-- 1 boinc boinc 25518410 Dec 19 15:10 oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip
-rw-r--r-- 1 boinc boinc 209 Dec 19 14:43 trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671461035.xml

Eventually the task completed in Boinc Manager with a computation error and the 2 trickles were still sitting in the directory, the zip file had gone.
The task didn't report back to the server so after a few minutes I manually Updated and the task disappeared from the Manager.
Looking in the project folder I saw that the 2 trickle files had also gone.
The event log showed that the trickles and job files (jf_*) had been removed, it also said it removed all the zip files though they weren't present, none of the.gz or .gzt files were present (for zip or job files).
Note: trickle names in event log end with ".sent" but that suffix was not seeen in the project directory, the .xml files were deleted !

Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671461035.xml.sent
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/trickle_up_oifs_43r3_ps_1351_2021050100_123_946_12164440_2_1671462672.xml.sent
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_1c36b977f0a0e944c1bdda1a7ee5396c.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_475c23e3be91c1b31667a48915dcc36c.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/jf_ab0d2288d1c874c01a43253fbe058b4e.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_0.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_1.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_2.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_3.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_4.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_5.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_6.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_7.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_8.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_9.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_10.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_11.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_12.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_13.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_14.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_15.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_16.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_17.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_18.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_19.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_20.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_21.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_22.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_23.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_24.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_25.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_26.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_27.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_28.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_29.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_30.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_31.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_32.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_33.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_34.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_35.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_36.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_37.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_38.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_39.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_40.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_41.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_42.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_43.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_44.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_45.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_46.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_47.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_48.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_49.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_50.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_51.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_52.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_53.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_54.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_55.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_56.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_57.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_58.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_59.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_60.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_61.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_62.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_63.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_64.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_65.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_66.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_67.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_68.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_69.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_70.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_71.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_72.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_73.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_74.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_75.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_76.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_77.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_78.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_79.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_80.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_81.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_82.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_83.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_84.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_85.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_86.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_87.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_88.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_89.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_90.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_91.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_92.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_93.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_94.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_95.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_96.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_97.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_98.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_99.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_100.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_101.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_102.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_103.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_104.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_105.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_106.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_107.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_108.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_109.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_110.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_111.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_112.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_113.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_114.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_115.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_116.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_117.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_118.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_119.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_120.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_121.zip.gzt
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip.gz
Mon 19 Dec 2022 15:15:07 GMT |  | [slot] removed file projects/climateprediction.net/oifs_43r3_ps_1351_2021050100_123_946_12164440_2_r1535625001_122.zip.gzt

A bit academic as the windows ones won't be coming till a few more of the problems have been ironed out and thoroughly tested. The drive for it will come if they start needing much bigger numbers processed by crunchers at a time.

Hopefully "error rate is now 10%" and "I also need to move the higher resolution work on which I've not been able to start yet" indicate more stuff coming.

PDW, thanks. I am not familiar with how the trickles are configured and work. That's for CPDN to deal with, I'm mostly concerned with the model interface. I would hope the client moves them to a safe place in case the transfer is stalled, allowing the task to finish. The progress XML file is just an internal file for the controlling code so it knows how far into the run it's got. It gets deleted by the client when the slot dir is removed.

Anyway, I see from the internal CPDN website the rest of the 39,000 PS tasks should be out this week. No idea what's happened to the rest of the BL tasks - seems to have slowed down.

It only remains for me to wish everyone a good Christmas and prosperous New Year. I'm off now until 2023.

Thanks for all the helpful messages, it has been useful.

A bit academic as the windows ones won't be coming till a few more of the problems have been ironed out and thoroughly tested. The drive for it will come if they start needing much bigger numbers processed by crunchers at a time.

Is it true the server is having difficulty as it is? It's gonna need an upgrade when the Virtualbox stuff comes out.

A bit academic as the windows ones won't be coming till a few more of the problems have been ironed out and thoroughly tested. The drive for it will come if they start needing much bigger numbers processed by crunchers at a time.

Is it true the server is having difficulty as it is? It's gonna need an upgrade when the Virtualbox stuff comes out.

I'm not sure where you read that, but the server has been moved onto a newer, managed cloud service with better resources. Previously it was on an 'unmanaged' cloud. There were some configuration differences which took a little time for CPDN to iron out but it's working much better than the older one. There is no need for an 'upgrade' to the server for vbox applications in terms of the resources it needs - it'll be just another 'app' as far as it's concerned.

the server has been moved onto a newer, managed cloud service with better resources.

It sure has. I am now running four Oifs _ps tasks at a time and was watching the Transfers tab as things went up. Often I got 3 to 4 megabytes/second upload speeds each when two tasks were uploading at once, and I notice one that got over 7 megabytes/second when uploading alone. Once in a while, I get one considerably slower. My Internet connection at my end is 75 megabits/second or 9.375 Megabytes/second.

These trickles normally go up in about 6 seconds, but the slowest one I remember took almost two minutes.

---

New day, new task !
This one had 90 minutes to run but would finish when I was out, it also had a trickle file that had been sitting there for 20 minutes or so.
I suspended the task, the event log shows this and when I resumed it when I got back in (the trickle file had disappeared).

Expecting the final trickle file to also take a while to go I clicked on Suspend task as soon as that trickle file appeared after the last zip file.
The status of the task changed to suspended but I'm aware of what the critical_section code does on these occasions.
After a few minutes, probably the 2 minutes sleep in the wrapper, the status of the task changed to Completed even though the final trickle file was still present in the project folder.
A few more minutes passed before I clicked on Update.
The task disappeared from the list of tasks, the trickle file got deleted, the task is showing as completed in my list of tasks on the server !