 |
|
Message boards : Number crunching : Work available and being requested but none downloaded
Message board moderation
| Author | Message |
|---|---|
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
OK, I’m now officially confused! For the past week my main rig :- https://www.cpdn.org/show_host_detail.php?hostid=1489273 has been running a single cpdn WU (plus 7 or 6 WCG and 5 or 6 Rosetta) and requesting more each day but, despite showing at least 2000 WUs available, has not been receiving any. My resource allocation is set to WCG=100, Rosetta=80, CPDN=60. Tonight it finished that WU, trickle fed the results and then reported completion. As this did not elicit a response I explicitly requested more work but, despite still showing more than 2000 Linux work units available, I “got 0 new tasks, no tasks sent”. The buffer was not full as 12 minutes later WCG requested work and download 7 new tasks. What have I done wrong to be shunned this way :-) |
|
Send message Joined: 15 Jan 06 Posts: 637 Credit: 26,751,529 RAC: 653 |
They had an outage today. It is working again for me. |
 Dave JacksonSend message Joined: 15 May 09 Posts: 4540 Credit: 19,039,635 RAC: 18,944 |
If still not getting work, go to the event log options and tick work fetch debug, then paste the lines following the attempt to fetch work. Probably worth clicking on the line in the event log asking for work and setting the event log to, "Show only this project" This may give you the answer by itself in which case it would be good if you told us that. Do leave an hour after previous attempts to get work as trying earlier restarts the one hour back off period. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
If still not getting work, go to the event log options and tick work fetch debug, then paste the lines following the attempt to fetch work. Probably worth clicking on the line in the event log asking for work and setting the event log to, "Show only this project" This may give you the answer by itself in which case it would be good if you told us that. As you request so it shall be done :- Wed 29 Apr 2020 08:36:42 BST | | Re-reading cc_config.xml Wed 29 Apr 2020 08:36:42 BST | | Config: GUI RPC allowed from any host Wed 29 Apr 2020 08:36:42 BST | | Config: GUI RPCs allowed from: Wed 29 Apr 2020 08:36:42 BST | | log flags: file_xfer, sched_ops, task, work_fetch_debug Wed 29 Apr 2020 08:36:42 BST | | [work_fetch] Request work fetch: Core client configuration Wed 29 Apr 2020 08:36:46 BST | | choose_project(): 1588145806.618128 Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] target work buffer: 17280.00 + 17280.00 sec Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 08:36:46 BST | climateprediction.net | [work_fetch] REC 885.418 prio -0.299 can request work Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] shortfall 25907.81 nidle 0.00 saturated 27934.29 busy 0.00 Wed 29 Apr 2020 08:36:46 BST | climateprediction.net | [work_fetch] share 0.250 Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 08:36:46 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 08:36:46 BST | climateprediction.net | can fetch CPU Wed 29 Apr 2020 08:36:46 BST | | [work_fetch] No project chosen for work fetch Wed 29 Apr 2020 08:36:54 BST | | Re-reading cc_config.xml Wed 29 Apr 2020 08:36:54 BST | | Config: GUI RPC allowed from any host Wed 29 Apr 2020 08:36:54 BST | | Config: GUI RPCs allowed from: Wed 29 Apr 2020 08:36:54 BST | | log flags: file_xfer, sched_ops, task, work_fetch_debug Wed 29 Apr 2020 08:36:54 BST | | [work_fetch] Request work fetch: Core client configuration Wed 29 Apr 2020 08:36:56 BST | | choose_project(): 1588145816.662131 Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] target work buffer: 17280.00 + 17280.00 sec Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 08:36:56 BST | climateprediction.net | [work_fetch] REC 885.375 prio -0.299 can request work Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] shortfall 26006.11 nidle 0.00 saturated 27934.29 busy 0.00 Wed 29 Apr 2020 08:36:56 BST | climateprediction.net | [work_fetch] share 0.250 Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 08:36:56 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 08:36:56 BST | climateprediction.net | can fetch CPU Wed 29 Apr 2020 08:36:56 BST | | [work_fetch] No project chosen for work fetch Wed 29 Apr 2020 08:37:37 BST | climateprediction.net | update requested by user Wed 29 Apr 2020 08:37:37 BST | | [work_fetch] Request work fetch: project updated by user Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | piggyback_work_request() Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] target work buffer: 17280.00 + 17280.00 sec Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] REC 885.375 prio -0.299 can request work Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] shortfall 25064.93 nidle 0.00 saturated 27934.29 busy 0.00 Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] share 0.250 Wed 29 Apr 2020 08:37:42 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | piggyback: resource CPU Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] set_request() for CPU: ninst 12 nused_total 0.00 nidle_now 0.00 fetch share 0.25 req_inst 3.00 req_secs 25064.93 Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] request: CPU (25064.93 sec, 3.00 inst) Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | Sending scheduler request: Requested by user. Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | Requesting new tasks for CPU Wed 29 Apr 2020 08:37:48 BST | climateprediction.net | Scheduler request completed: got 0 new tasks Wed 29 Apr 2020 08:37:48 BST | climateprediction.net | No tasks sent Wed 29 Apr 2020 08:37:48 BST | climateprediction.net | Project requested delay of 3636 seconds Wed 29 Apr 2020 08:37:48 BST | | [work_fetch] Request work fetch: RPC complete Wed 29 Apr 2020 08:37:53 BST | | choose_project(): 1588145873.252965 Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] target work buffer: 17280.00 + 17280.00 sec Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 08:37:53 BST | climateprediction.net | [work_fetch] REC 885.375 prio -0.211 can't request work: scheduler RPC backoff (3630.96 sec) Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] shortfall 25171.40 nidle 0.00 saturated 27934.29 busy 0.00 Wed 29 Apr 2020 08:37:53 BST | climateprediction.net | [work_fetch] share 0.000 Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 08:37:53 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 08:37:53 BST | climateprediction.net | skip: scheduler RPC backoff Wed 29 Apr 2020 08:37:53 BST | | [work_fetch] No project chosen for work fetch Wed 29 Apr 2020 08:38:25 BST | | Re-reading cc_config.xml Wed 29 Apr 2020 08:38:25 BST | | Config: GUI RPC allowed from any host Wed 29 Apr 2020 08:38:25 BST | | Config: GUI RPCs allowed from: Wed 29 Apr 2020 08:38:25 BST | | log flags: file_xfer, sched_ops, task |
|
Dave Jackson Send message Joined: 15 May 09 Posts: 4540 Credit: 19,039,635 RAC: 18,944 |
Not spotting the cause immediately. Will study some more. Worth turning the flag off again for the event log now you have the information so as not to clog it up. Also waiting till some work finishes on my box here to see what happens when I request work. |
|
Send message Joined: 1 Jan 07 Posts: 1061 Credit: 36,719,896 RAC: 7,946 |
Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] request: CPU (25064.93 sec, 3.00 inst)Well, he's asking for a perfectly reasonable amount of work, but the server isn't sending it. At most projects, I'd say 'check your project preferences - make sure you're allowing the type of work currently available'. But this project has those options turned off. |
|
Dave Jackson Send message Joined: 15 May 09 Posts: 4540 Credit: 19,039,635 RAC: 18,944 |
Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] request: CPU (25064.93 sec, 3.00 inst)Well, he's asking for a perfectly reasonable amount of work, but the server isn't sending it. And, I have temporarily allowed all four cores on my laptop and it is currently downloading a task. Now winter has gone, running on all four cores it gets a bit warm and noisy so once downloads are finished and I know it is running OK I will suspend the new task until another is finished. Any memory or disk space problems I would expect a message to say so which means I am a bit stumped. Edit also checked Maximum daily WU quota per CPU 0/day. wasn't -1/day which is when Andy stops them getting new work till they have told us 32 bit libs have been installed. (I would have expected some message in the log if that were the case but can't remember from old posts here what it is.) |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
Wed 29 Apr 2020 08:37:42 BST | climateprediction.net | [work_fetch] request: CPU (25064.93 sec, 3.00 inst)Well, he's asking for a perfectly reasonable amount of work, but the server isn't sending it. The system normally runs 2 CPDN tasks and would happily run many more if I didn't suspend the excess. Memory should be OK, currently running 6 Rosetta tasks taking about 5GB and 6 WCG taking about 0.5GB out of 15GB available. Disk has 30+GB free for Boinc. |
|
Send message Joined: 15 Jan 06 Posts: 637 Credit: 26,751,529 RAC: 653 |
Sometimes CPDN doesn't send work with a short buffer (e.g., 0.1 + 0.5 days). If that is your case, try the following: Set No New Work to all the projects except CPDN Set the buffer to 1.0 + 2.0 days Hit Update See what happens. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
Sometimes CPDN doesn't send work with a short buffer (e.g., 0.1 + 0.5 days). That is probably it, the buffer is set to 0.2 + 0.2 but dumbklutz here has just messed up following simple instructions and downloaded 21 Rosetta and 56 WCG tasks by mistake. I'll try again in an hour when the timeout's cleared :-) |
|
Send message Joined: 15 Jan 06 Posts: 637 Credit: 26,751,529 RAC: 653 |
I know, you learn the hard way in this business. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
OK, I've now taken it up to 2.2 + 2.2 and still no joy. Sadly I didn't have log work fetch on, I'll up the request to 3.2 + 2.2 with logging in an hour and report back. The reason I've not put it higher quicker is fear of getting 12 WUs at once as has happened before. As I don't run more than 2 at the same time that would take me a couple of months to clear which I'm loath to do. |
|
Send message Joined: 15 Jan 06 Posts: 637 Credit: 26,751,529 RAC: 653 |
You could always detach from CPDN, reboot and then reattach. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
No, with 3.2 + 2.2 it just requested more tasks and still failed :- Wed 29 Apr 2020 22:17:58 BST | | Re-reading cc_config.xml Wed 29 Apr 2020 22:17:58 BST | | Config: GUI RPC allowed from any host Wed 29 Apr 2020 22:17:58 BST | | Config: GUI RPCs allowed from: Wed 29 Apr 2020 22:17:58 BST | | log flags: file_xfer, sched_ops, task, work_fetch_debug Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] Request work fetch: Core client configuration Wed 29 Apr 2020 22:17:58 BST | | choose_project(): 1588195078.734215 Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] target work buffer: 276480.00 + 190080.00 sec Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 22:17:58 BST | climateprediction.net | [work_fetch] REC 726.617 prio 0.000 can't request work: "no new tasks" requested via Manager Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] shortfall 4315045.60 nidle 0.00 saturated 103410.02 busy 40249.55 Wed 29 Apr 2020 22:17:58 BST | climateprediction.net | [work_fetch] share 0.000 Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 22:17:58 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 22:17:58 BST | climateprediction.net | skip: "no new tasks" requested via Manager Wed 29 Apr 2020 22:17:58 BST | | [work_fetch] No project chosen for work fetch Wed 29 Apr 2020 22:18:02 BST | climateprediction.net | work fetch resumed by user Wed 29 Apr 2020 22:18:02 BST | | [work_fetch] Request work fetch: project work fetch resumed by user Wed 29 Apr 2020 22:18:03 BST | | choose_project(): 1588195083.758160 Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] target work buffer: 276480.00 + 190080.00 sec Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | [work_fetch] REC 726.603 prio -1.000 can request work Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] shortfall 4315078.42 nidle 0.00 saturated 103400.22 busy 40249.55 Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | [work_fetch] share 1.000 Wed 29 Apr 2020 22:18:03 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | can fetch CPU Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | CPU needs work - buffer low Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | checking CPU Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | [work_fetch] set_request() for CPU: ninst 12 nused_total 0.00 nidle_now 0.00 fetch share 1.00 req_inst 12.00 req_secs 4315078.42 Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | CPU set_request: 4315078.415430 Wed 29 Apr 2020 22:18:03 BST | climateprediction.net | [work_fetch] request: CPU (4315078.42 sec, 12.00 inst) Wed 29 Apr 2020 22:18:04 BST | climateprediction.net | Sending scheduler request: To fetch work. Wed 29 Apr 2020 22:18:04 BST | climateprediction.net | Requesting new tasks for CPU Wed 29 Apr 2020 22:18:06 BST | climateprediction.net | Scheduler request completed: got 0 new tasks Wed 29 Apr 2020 22:18:06 BST | climateprediction.net | No tasks sent Wed 29 Apr 2020 22:18:06 BST | climateprediction.net | Project requested delay of 3636 seconds Wed 29 Apr 2020 22:18:06 BST | climateprediction.net | [work_fetch] backing off CPU 3242 sec Wed 29 Apr 2020 22:18:06 BST | | [work_fetch] Request work fetch: RPC complete Wed 29 Apr 2020 22:18:11 BST | | choose_project(): 1588195091.181629 Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] ------- start work fetch state ------- Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] target work buffer: 276480.00 + 190080.00 sec Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] --- project states --- Wed 29 Apr 2020 22:18:11 BST | climateprediction.net | [work_fetch] REC 726.603 prio 0.000 can't request work: scheduler RPC backoff (3630.96 sec) Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] --- state for CPU --- Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] shortfall 4315167.74 nidle 0.00 saturated 103390.40 busy 40240.66 Wed 29 Apr 2020 22:18:11 BST | climateprediction.net | [work_fetch] share 0.000 project is backed off (resource backoff: 3236.67, inc 2400.00) Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] ------- end work fetch state ------- Wed 29 Apr 2020 22:18:11 BST | climateprediction.net | choose_project: scanning Wed 29 Apr 2020 22:18:11 BST | climateprediction.net | skip: scheduler RPC backoff Wed 29 Apr 2020 22:18:11 BST | | [work_fetch] No project chosen for work fetch Wed 29 Apr 2020 22:18:24 BST | climateprediction.net | work fetch suspended by user Wed 29 Apr 2020 22:18:34 BST | | Re-reading cc_config.xml Wed 29 Apr 2020 22:18:34 BST | | Config: GUI RPC allowed from any host Wed 29 Apr 2020 22:18:34 BST | | Config: GUI RPCs allowed from: Wed 29 Apr 2020 22:18:34 BST | | log flags: file_xfer, sched_ops, task I'll set it back to normal and try detach / attach. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
Woah, that's caused chaos! From steadily running 7/6 WCG and 5/6 Rosetta I now have 9 Rosetta waiting to run with 9 WCG and 3 Rosetta running. I'll leave it until the morning when it's (hopefully) sorted that out before taking any further action. |
|
Alan K Send message Joined: 22 Feb 06 Posts: 491 Credit: 31,042,229 RAC: 14,816 |
You would probably need to set the work buffer higher as the N216 models typically take 10 to 14days to run. Try setting the buffer to 10 days work but reduce your number of processors to 25% to reduce the work capabillty depending on how many cores your processors have. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
You would probably need to set the work buffer higher as the N216 models typically take 10 to 14days to run. Try setting the buffer to 10 days work but reduce your number of processors to 25% to reduce the work capabillty depending on how many cores your processors have. At 0.2 + 0.2 my client requested 3 WUs which is already more than I need, I think at this point we need to work out why the server did not honour that request - I cannot be playing games every 10 days trying to get work that should be trickle feeding anyway. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
I'm going to have to wait several days before I can usefully try again, in the current state it won't request work because the buffer is full of WCG and Rosetta WUs. A partial copy of the work log :- Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] ------- start work fetch state ------- Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] target work buffer: 17280.00 + 17280.00 sec Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] --- project states --- Thu 30 Apr 2020 10:41:05 BST | climateprediction.net | [work_fetch] REC 607.737 prio -1.000 can request work Thu 30 Apr 2020 10:41:05 BST | Rosetta@home | [work_fetch] REC 4978.827 prio -6.791 can't request work: "no new tasks" requested via Manager Thu 30 Apr 2020 10:41:05 BST | World Community Grid | [work_fetch] REC 6234.763 prio -6.627 can't request work: "no new tasks" requested via Manager (25.47 sec) Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] --- state for CPU --- Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] shortfall 0.00 nidle 0.00 saturated 55844.45 busy 0.00 Thu 30 Apr 2020 10:41:05 BST | climateprediction.net | [work_fetch] share 1.000 Thu 30 Apr 2020 10:41:05 BST | Rosetta@home | [work_fetch] share 0.000 Thu 30 Apr 2020 10:41:05 BST | World Community Grid | [work_fetch] share 0.000 Thu 30 Apr 2020 10:41:05 BST | | [work_fetch] ------- end work fetch state ------- Thu 30 Apr 2020 10:41:05 BST | climateprediction.net | piggyback: resource CPU Thu 30 Apr 2020 10:41:05 BST | climateprediction.net | piggyback: don't need CPU Thu 30 Apr 2020 10:41:05 BST | climateprediction.net | [work_fetch] request: CPU (0.00 sec, 0.00 inst) Thu 30 Apr 2020 10:41:06 BST | climateprediction.net | Sending scheduler request: Requested by user. Thu 30 Apr 2020 10:41:06 BST | climateprediction.net | Not requesting tasks: don't need (job cache full) Thu 30 Apr 2020 10:41:07 BST | climateprediction.net | Scheduler request completed Question, what is the "piggyback: don't need CPU" saying, is that just echoing the job cache full decision? |
|
Dave Jackson Send message Joined: 15 May 09 Posts: 4540 Credit: 19,039,635 RAC: 18,944 |
I'm going to have to wait several days before I can usefully try again, in the current state it won't request work because the buffer is full of WCG and Rosetta WUs. Not if you suspend those projects while the request is made. |
|
Send message Joined: 28 Jul 19 Posts: 150 Credit: 12,830,559 RAC: 228 |
I'm going to have to wait several days before I can usefully try again, in the current state it won't request work because the buffer is full of WCG and Rosetta WUs. Ah, thank you - I’ll try that. |
©2024 cpdn.org
