Message boards : Number crunching : Stuck upload issue
Message board moderation
Previous · 1 · 2 · 3 · 4 · 5 · Next
Author | Message |
---|---|
Send message Joined: 4 Nov 06 Posts: 5 Credit: 3,764,885 RAC: 296 |
6 days later, I've still got the exact same files in exactly the same spot trying to upload... wah2_cam25_a0ib_200405_18_691_011370208_0_r706661527_12.zip - 30.03% (31.56/105.10 MB) wah2_cam25_a0ib_200405_18_691_011370208_0_r706661527_13.zip - 9.93% (10.45/105.17 MB) wah2_cam25_a0jx_200405_18_691_011370266_0_r76671198_10.zip - 52.51% (55.16/105.13 MB) wah2_cam25_a0ea_200405_18_691_011370063_0_r1526698435_14.zip - 31.20% (32.83/105.20 MB) wah2_cam25_a0ht_200405_18_691_011370190_0_r177040747_1.zip - 21.42% (22.56/105.30 MB) Not sure if I should try messing with it... I really just kinda want to abort/clear these tasks at this point, as it's taking longer to upload than it did to compute in the first place :( |
Send message Joined: 22 Feb 06 Posts: 491 Credit: 30,975,898 RAC: 14,500 |
This is the message from that link just now: <data_server_reply> <status>1</status> <message>no command</message> </data_server_reply> so no change at the moment. |
Send message Joined: 15 May 09 Posts: 4538 Credit: 19,006,502 RAC: 21,456 |
Will add another message to the batch's card. |
Send message Joined: 31 Aug 04 Posts: 10 Credit: 2,538,005 RAC: 0 |
I have two stuck uploads, one of them for over a week, the other for a couple of days: 21/01/2018 10:41:08 | climateprediction.net | Started upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip 21/01/2018 10:41:32 | | Project communication failed: attempting access to reference site 21/01/2018 10:41:32 | climateprediction.net | Temporarily failed upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip: transient HTTP error 21/01/2018 10:41:32 | climateprediction.net | Backing off 03:59:04 on upload of wah2_cam25_a047_200405_18_689_011368595_0_r2068114942_7.zip 21/01/2018 10:41:35 | | Internet access OK - project servers may be temporarily down. 21/01/2018 10:42:07 | climateprediction.net | Started upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip 21/01/2018 10:42:31 | | Project communication failed: attempting access to reference site 21/01/2018 10:42:31 | climateprediction.net | Temporarily failed upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip: transient HTTP error 21/01/2018 10:42:31 | climateprediction.net | Backing off 04:12:33 on upload of wah2_cam25_a03s_200405_18_689_011368580_0_r616315434_17.zip 21/01/2018 10:42:32 | | Internet access OK - project servers may be temporarily down. Anything I can do about this on my side or does this need to be resolved on the server side ? Tom |
Send message Joined: 15 May 09 Posts: 4538 Credit: 19,006,502 RAC: 21,456 |
Sometimes exiting BOINC manager and restarting it can shift stuck uploads but I suspect this is a server side problem.I messaged the project about it on Friday. |
Send message Joined: 16 Jan 10 Posts: 1084 Credit: 7,808,726 RAC: 5,192 |
Sometimes exiting BOINC manager and restarting it can shift stuck uploads but I suspect this is a server side problem.I messaged the project about it on Friday. ... my CAM upload has been stuck since 7 January: that CAM upload server needs a good kick with a steel-toecapped Dr Martens. I had the same error with a couple of the 145 x ~50 MB Zips for global batch #696, but they cleared after a day or so. It would have been an aggravating waste of time to calculate and upload most - but not all - of over 7 GB data. So whatever was done to the "global" server needs to be done to the CAM server, sharpish. Edit: As it is, my CAM model has calculated and uploaded 17/18 x 100 MB - i.e. ~2 GB - which I assume is useless to the project without the missing Zip. I'm tempted to abort it, so at least someone else could re-run it. |
Send message Joined: 15 May 09 Posts: 4538 Credit: 19,006,502 RAC: 21,456 |
So whatever was done to the "global" server needs to be done to the CAM server, sharpish. I suggested a while ago that the uploads could be redirected to Oxford as was done with the batch destined for a Korean server. I have repeated that suggestion with a slightly stronger wording. Edit:I have been told they don't have the capacity at Oxford to take those two batches at present but will investigate to see if there is a server somewhere else that can take the uploads. |
Send message Joined: 7 Aug 04 Posts: 2187 Credit: 64,822,615 RAC: 5,275 |
These stuck uploads have been happening occasionally for more than a few years now. But it seems to be the models with really big upload files that have the most problems. A server reboot seems to release the stuck uploads. The first time I remember a recurrent problem was with the ANZ runs back in March of 2014. |
Send message Joined: 11 Jan 18 Posts: 5 Credit: 129,961 RAC: 0 |
Hi. Am new to the project and so am not sure whether the upload issue that I'm experiencing, is out of the ordinary or not. The tasks reach 100% according to the transfer tab in boinc manager but then the project goes into backoff, before then starting the counter again at 0% Have emboldened the terms I'm not used to seeing in the event log below. Thanks
31/01/2018 16:50:47 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 16:50:47 | climateprediction.net | Sending scheduler request: To send trickle-up message. 31/01/2018 16:50:47 | climateprediction.net | Not requesting tasks: "no new tasks" requested via Manager 31/01/2018 16:50:49 | climateprediction.net | Scheduler request completed 31/01/2018 16:52:08 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded 31/01/2018 16:52:08 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error 31/01/2018 16:52:08 | climateprediction.net | Backing off 00:02:04 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 16:52:09 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded 31/01/2018 16:52:09 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error 31/01/2018 16:52:09 | climateprediction.net | Backing off 00:02:28 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 16:54:13 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 16:54:38 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 16:55:07 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded 31/01/2018 16:55:07 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error 31/01/2018 16:55:07 | climateprediction.net | Backing off 00:06:38 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 16:55:37 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded 31/01/2018 16:55:37 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error 31/01/2018 16:55:37 | climateprediction.net | Backing off 00:04:53 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 17:24:21 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 17:24:21 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 17:26:22 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded 31/01/2018 17:26:22 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error 31/01/2018 17:26:22 | climateprediction.net | Backing off 00:08:57 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 17:26:28 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded 31/01/2018 17:26:28 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error 31/01/2018 17:26:28 | climateprediction.net | Backing off 00:11:09 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 17:35:20 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 17:37:26 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded 31/01/2018 17:37:26 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error 31/01/2018 17:37:26 | climateprediction.net | Backing off 00:25:42 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 17:49:14 | climateprediction.net | Started upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 17:50:56 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: Disk quota exceeded 31/01/2018 17:50:56 | climateprediction.net | Temporarily failed upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip: transient upload error 31/01/2018 17:50:56 | climateprediction.net | Backing off 00:24:50 on upload of wah2_sas50_q2jw_201612_13_707_011450425_1_r573430588_8.zip 31/01/2018 18:12:22 | climateprediction.net | Started upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip 31/01/2018 18:14:40 | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: Disk quota exceeded 31/01/2018 18:14:40 | climateprediction.net | Temporarily failed upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip: transient upload error 31/01/2018 18:14:40 | climateprediction.net | Backing off 00:43:48 on upload of wah2_sas50_qbkv_201612_13_708_011462124_0_r1129882916_5.zip |
Send message Joined: 5 Aug 04 Posts: 1496 Credit: 95,522,203 RAC: 0 |
I'm experiencing similar problems over the last half-day. I'm not yet aware of the problem(s). Hang in with us. These things get sorted out sooner or later. (Servers are located around the globe and problems could be anywhere and of any type. [Curiously, I'm experiencing problems with EAS, SAS, EU, and NAM tasks -- currently, one task is uploading on each machine, albeit very slowly.]) Welcome to the project and the boards. "We have met the enemy and he is us." -- Pogo Greetings from coastal Washington state, the scenic US Pacific Northwest. |
Send message Joined: 18 Jul 13 Posts: 438 Credit: 25,620,508 RAC: 4,981 |
I'm having the same issue. It's has been like this for 12hours (8:00 CET) Mine all are SAS50 batch 708, 707 |
Send message Joined: 15 May 09 Posts: 4538 Credit: 19,006,502 RAC: 21,456 |
I noticed this morning, the site was down for a couple of hours. This may or may not be relevant. I have had one upload from 706 and one from 708 go through after the site came back on stream. They went through at the normal speed for me which is total upload speed of about 100Mbit/second but that is I suspect a limitation of my connection rather than anything to do with the project. |
Send message Joined: 20 Jul 05 Posts: 25 Credit: 414,873 RAC: 406 |
I noticed this morning, the site was down for a couple of hours. This may or may not be relevant. I have had one upload from 706 and one from 708 go through after the site came back on stream. They went through at the normal speed for me which is total upload speed of about 100Mbit/second but that is I suspect a limitation of my connection rather than anything to do with the project. Dave I am curious to know how long take you to upload 45.97 meg looking at my manager takes just over 12 minutes but currently the project is in back off mode |
Send message Joined: 13 Jan 07 Posts: 195 Credit: 10,581,566 RAC: 0 |
I am having the same problem but they all eventually upload with the exception of a restart.zip for a cam25 model, batch 691 which is stuck at 27.63 percent. |
Send message Joined: 15 May 09 Posts: 4538 Credit: 19,006,502 RAC: 21,456 |
Dave I am curious to know how long take you to upload 45.97 meg looking at my manager takes just over 12 minutes but currently the project is in back off mode Mine took 18minutes but there were two uploads going at once so I guess you can half that. Worth noting that I don't get any faster than that when I run a speed test on my connection. |
Send message Joined: 31 Aug 04 Posts: 29 Credit: 3,971,150 RAC: 884 |
I'm now getting 1/31/2018 11:03:42 PM | climateprediction.net | [error] Error reported by file upload server: can't write file wah2_sas50_q3b2_201612_13_707_011451403_0_r448480803_11.zip: Disk quota exceeded <sigh> if it's not one project with capacity problems, it's another |
Send message Joined: 5 Aug 04 Posts: 1283 Credit: 15,824,334 RAC: 0 |
The project team are aware of the upload problem for WAH2 SAS50 batch 706, 707 and 708 files. Upload server upload2.cpdn.org has reached its disk quota and they are making more space available. "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer |
Send message Joined: 11 Jan 18 Posts: 5 Credit: 129,961 RAC: 0 |
Selfishly speaking, I'm glad it's not just me ;) Thanks for the responses. And to those working hard on our behalf. |
Send message Joined: 1 Sep 04 Posts: 161 Credit: 81,522,141 RAC: 1,164 |
I have had a stuck upload from batch 691 for so many weeks I have lost track. |
Send message Joined: 31 Aug 04 Posts: 391 Credit: 219,896,461 RAC: 649 |
The project team are aware of the upload problem for WAH2 SAS50 batch 706, 707 and 708 files. Upload server upload2.cpdn.org has reached its disk quota and they are making more space available. Thanks for update. Also thanks for knowing upload server managers have no clue from project as to what space provision needed. Us contributors would like to think the infrastructure is well-managed. Obviously not I keep on crunching - knowing how underpaid and overworked the support staff are |
©2024 cpdn.org