I can upgrade one computer to boinc 7.20.5 using this ppa: https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/boinc
I think that version is a development release. I guess it could cause other issues.

Gianfranco's versions of BOINC rarely cause problems. I have been running the 7.21.0 compiled from source from Git-Hub and have yet to have problems with it. Every few weeks I do it afresh from the nightly build. (Occasionally I have had problems getting it to compile but that is another matter!)

I can upgrade one computer to boinc 7.20.5 using this ppa: https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/boinc
I think that version is a development release. I guess it could cause other issues.

I'm running the same v7.20.5, from the same source. This is a very minor change - some security fixes for the latest Apple Mac OS, and a small bugfix for Linux and Windows for an error introduced during that Mac change.

Otherwise, it's exactly the same as the full release v7.20.2

biodoc & cletus: could you please look in your /var/log/messages file around the time the task crashed. I am interested to see if there is any report in there that a process was killed due to a memory issue. There may not be one but some systems do log process kills due to out of memory issues for example.

Thanks.

I also had a task that failed with that error, however the model did not finish:

https://www.cpdn.org/result.php?resultid=22250347

Exit status 9 (0x00000009) Unknown error code
...
12:03:35 STEP 973 H= 243:15 +CPU= 10.376
12:03:45 STEP 974 H= 243:30 +CPU= 10.186
12:03:56 STEP 975 H= 243:45 +CPU= 10.185
double free or corruption (out)
12:04:06 STEP 976 H= 244:00 +CPU= 10.574

</stderr_txt>

The same computer successfully completed 11 other jobs from the latest oifs batch. They were being run 6 at a time, with around 20GB free ram available.
I'm certainly fine with test jobs being sent to it, if you want to.

syslog of https://www.cpdn.org/result.php?resultid=22250486. This one crashed in the middle of a run. No useful information.

Dec 14 19:48:21 x32-linux3 boinc[1692]: 14-Dec-2022 19:48:21 [climateprediction.net] Started upload of oifs_43r3_bl_a054_2016092300_15_949_12166578_0_r1730349614_14.zip
Dec 14 19:48:34 x32-linux3 boinc[1692]: 14-Dec-2022 19:48:34 [climateprediction.net] Finished upload of oifs_43r3_bl_a054_2016092300_15_949_12166578_0_r1730349614_14.zip
Dec 14 19:48:37 x32-linux3 boinc[1692]: 14-Dec-2022 19:48:37 [climateprediction.net] Computation for task oifs_43r3_bl_a054_2016092300_15_949_12166578_0 finished

syslog of https://www.cpdn.org/result.php?resultid=22250622. Looks like most of the output files were missing.

Dec 15 07:38:08 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:08 [climateprediction.net] Started upload of oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_42.zip
Dec 15 07:38:16 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:16 [climateprediction.net] Finished upload of oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_42.zip
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Computation for task oifs_43r3_ps_1325_2021050100_123_946_12164414_2 finished
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_43.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_44.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_45.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_46.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_47.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_48.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_49.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_50.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_51.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_52.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_53.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_54.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_55.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_56.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_57.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_58.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_59.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_60.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_61.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_62.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_63.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_64.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_65.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_66.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_67.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_68.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_69.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_70.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_71.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_72.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_73.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_74.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_75.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_76.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_77.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_78.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_79.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_80.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_81.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_82.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_83.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_84.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_85.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_86.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_87.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_88.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_89.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_90.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_91.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_92.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_93.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_94.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_95.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_96.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_97.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_98.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_99.zip for task oifs_43r3_ps
_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_100.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_101.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_102.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_103.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_104.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_105.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_106.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_107.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_108.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_109.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_110.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_111.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_112.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_113.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_114.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_115.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_116.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_117.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_118.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_119.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_120.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_121.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:38:18 x32-linux3 boinc[1692]: 15-Dec-2022 07:38:18 [climateprediction.net] Output file oifs_43r3_ps_1325_2021050100_123_946_12164414_2_r1007344185_122.zip for task oifs_43r3_p
s_1325_2021050100_123_946_12164414_2 absent
Dec 15 07:40:11 x32-linux3 boinc[1692]: 15-Dec-2022 07:40:11 [climateprediction.net] Started upload of oifs_43r3_bl_a004_2016092300_15_949_12166398_1_r1266389906_8.zip
Dec 15 07:40:24 x32-linux3 boinc[1692]: 15-Dec-2022 07:40:24 [climateprediction.net] Finished upload of oifs_43r3_bl_a004_2016092300_15_949_12166398_1_r1266389906_8.zip
Dec 15 07:42:08 x32-linux3 systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Dec 15 07:42:08 x32-linux3 systemd[1]: Started Process Core Dump (PID 24512/UID 0).
Dec 15 07:42:10 x32-linux3 systemd-coredump[24513]: Core file was truncated to 2147483648 bytes.
Dec 15 07:42:11 x32-linux3 systemd-coredump[24513]: Process 23225 (oifs_43r3_model) of user 129 dumped core.#012#012Stack trace of thread 23225:#012#0  0x0000000001dc903b n/a (/var/lib/
boinc-client/slots/0/oifs_43r3_model.exe (deleted) + 0x19c903b)
Dec 15 07:42:11 x32-linux3 systemd[1]: systemd-coredump@0-24512-0.service: Succeeded.

Glen,
I looked in syslog, kern.log and the systemd journal, but did not see anything unusual while the job was running or when it ended.

The boinc log messages for when the job failed were:

Dec 14 12:04:05 hal boinc[2320]: 14-Dec-2022 12:04:05 [climateprediction.net] Finished upload of oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_9.zip
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Computation for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 finished
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Output file oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_10.zip for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 absent
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Output file oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_11.zip for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 absent
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Output file oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_12.zip for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 absent
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Output file oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_13.zip for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 absent
Dec 14 12:04:07 hal boinc[2320]: 14-Dec-2022 12:04:07 [climateprediction.net] Output file oifs_43r3_bl_a019_2016092300_15_949_12166439_0_r1529103669_14.zip for task oifs_43r3_bl_a019_2016092300_15_949_12166439_0 absent

---

Thanks Richard. That gives some reassurance about mixing versions.

To follow up on that:

Just at the moment, one of my machines is currently running one of your IFS_ps tasks built with API version 7.20.1, alongside a HadSM4 N144 built with API 7.9.0. They're getting along just fine.

It's a brutally simple but effective system. The compiler puts the text string API_VERSION_whatever into the compiled library, and the deployment script searches for that text in the finished executable, and copies it to the XML control file. That way, the BOINC client knows how to talk to the app via the correct library calls.

I upgraded the boinc client 7.20.5 on the computer with the 2 errors to see if it's more reliable.

Thanks Richard. That gives some reassurance about mixing versions.

To follow up on that:It's a brutally simple but effective system. The compiler puts the text string API_VERSION_whatever into the compiled library, and the deployment script searches for that text in the finished executable, and copies it to the XML control file. That way, the BOINC client knows how to talk to the app via the correct library calls.

Ok. If you do hear any word on the grapevine that 7.20 is, shall we say, less trustworthy than earlier versions, let me know ;)

syslog of https://www.cpdn.org/result.php?resultid=22250622. Looks like most of the output files were missing.... (snipped)

biodoc & cletus, thanks for looking. This is exactly what I was hoping for, in biodoc's syslog (at the bottom), we have:

Dec 15 07:42:11 x32-linux3 systemd-coredump[24513]: Process 23225 (oifs_43r3_model) of user 129 dumped core.#012#012Stack trace of thread 23225:#012#0  0x0000000001dc903b n/a (/var/lib/boinc-client/slots/0/oifs_43r3_model.exe (deleted) + 0x19c903b)

This tells me it's the model process that's failing, and not the controlling wrapper code. Which is very useful because up to now we've been assuming it was the controlling wrapper.

I've never seen the model fail like this on my machines, nor on the machines attached to CPDN's development test site. I wonder if it's hardware related, as this failed on biodocs's 5950X. I only have small AMD box to test on and develop on intel.

The missing files was a bug that was corrected before the latest batch of the oifs_43r3_bl app went out.

Thanks again.

I've never seen the model fail like this on my machines, nor on the machines attached to CPDN's development test site. I wonder if it's hardware related, as this failed on biodocs's 5950X. I only have small AMD box to test on and develop on intel.

Thanks again.

Actually that computer is a 3950X which is also AMD so your point is taken.

I've never seen the model fail like this on my machines, nor on the machines attached to CPDN's development test site. I wonder if it's hardware related, as this failed on biodocs's 5950X. I only have small AMD box to test on and develop on intel.

Actually that computer is a 3950X which is also AMD so your point is taken.

I stand corrected. What does puzzle me is why there was no stack trace in the log returned with the task. Something else to look into.

There may also be two errors here because one of your logs showed the model finishing normally and only at the very end did we see the double free corruption. If you do see any more 'double free' errors, please do check in the syslog for any core dump messages, that would be very helpful.

Overall this batch of 250 tasks ran considerably better than previous batches, with only a 8% error rate. I believe there are about 6500 tasks of the oifs_43r3_bl app & 39000 tasks of the oifs_43r3_ps app ready to go once CPDN are happy with this test batch. Hopefully soon.

Overall this batch of 250 tasks ran considerably better than previous batches, with only a 8% error rate. I believe there are about 6500 tasks of the oifs_43r3_bl app & 39000 tasks of the oifs_43r3_ps app ready to go once CPDN are happy with this test batch. Hopefully soon.

My machine is ready to go, running 3 of each all at the same time, once I get them. So far, all the tasks I have run have completed successfully. This may be too many to run at once from a performance standpoint, but we will see.

Good time to send them out, too, since Rosetta's download server has be down for days and I ran out of work from them two days ago.

---

Good time to send them out, too, since Rosetta's download server has be down for days and I ran out of work from them two days ago.

they have 6.5 million queued on the server, and I got some 1, 2, 3, and 8 hours ago, and the odd one every day for the last week (but they had not much to send)

Uploads are not working very well for me currently (and haven't last week either):

I am currently running three "OpenIFS 43r3 Perturbed Surface v1.05" tasks in parallel. (Those are replica tasks from workunits with earlier error results, a.k.a. resends.) From the result file output of these few tasks alone, I am seeing very frequent "transient HTTP error" events. Upload server is upload11.cpdn.org.

Secondary problem: When the client retries failed uploads, it receives "Error reported by file upload server: [file name] locked by file_upload_handler PID=12345" on the first several retries.

Secondary problem: When the client retries failed uploads, it receives "Error reported by file upload server: [file name] locked by file_upload_handler PID=12345" on the first several retries.

The locked by file_upload_handler is I think that the process managing the initial try hasn't let go of it yet. Are the files getting through eventually? If the problem persists we can chase on Monday when I am guessing Andy will be in.

Dave Jackson wrote:

The locked by file_upload_handler is I think that the process managing the initial try hasn't let go of it yet. Are the files getting through eventually? If the problem persists we can chase on Monday when I am guessing Andy will be in.

Yes, files get through eventually. It takes them a good while, but the overall backlog is not increasing in the long run.

(Edit, some files upload without errors on first try. But rather many don't.)

Yes, files get through eventually. It takes them a good while, but the overall backlog is not increasing in the long run.

I saw that when uploads started to work again and then later all seemed OK. Some other users saw no problems. I had put it down to the servers getting hammered when Andy sorted things out but what you report makes me not so sure. I guess we will just need to keep an eye on it.

FWIW, the rest of the few replica tasks in my work buffers completed and uploaded throughout yesterday, all results are marked valid. Though the uploads were dragged out by the described temporary transfer failures, which recurred yesterday as well.

Any indications if the Mac models have been fixed and are going to be re-released soon? Just wondering as re-configuring the PC between WSL2 and VBox Mac is not as simple, thus preferably would run one for a while before switching to the other.

Any indications if the Mac models have been fixed and are going to be re-released soon? Just wondering as re-configuring the PC between WSL2 and VBox Mac is not as simple, thus preferably would run one for a while before switching to the other.

"Soon if all goes well." Is what I have seen but whose definition of soon? At least another day before the next OIFS tasks is likely.

Edit: If I was to bet on it I would say the OIFS will make it first but I really am guessing.

£dit2: 1,000 OIFS perturbed surface tasks which have almost gone. #950