# cores limitation

Message boards : Number crunching : # cores limitation

To post messages, you must log in.

AuthorMessage
Profile [AF>Amis des Lapins] Phil1966

Send message
Joined: 24 Jan 17
Posts: 6
Credit: 247,909,594
RAC: 6
   
Message 30 - Posted: 29 Jan 2017, 7:15:31 UTC

Hello,
Can someone tell me what's wrong with the below app_config ?
I try to limit the cores use to 4 (out of 8)
Thank You !

<app_config>

   <app>
      <app_name>amicable_2_64</app_name> 
      <max_concurrent>1</max_concurrent>
      </app>

<app_version>
<app_name>amicable_2_64</app_name> 
<avg_ncpus>4.000000</avg_ncpus> 
</app_version>

</app_config>
ID: 30 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 31 - Posted: 29 Jan 2017, 11:01:19 UTC - in response to Message 30.  

Did you try to shut down BOINC client and manager completely, then set "avg_ncpus" in both client_state.xml and client_state_prev.xml?
ID: 31 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Crystal Pellet

Send message
Joined: 25 Jan 17
Posts: 2
Credit: 690,212
RAC: 0
   
Message 32 - Posted: 29 Jan 2017, 12:32:39 UTC - in response to Message 30.  

This one is working. Adjust the numbers 15 twice and don't forget a new line after </app_config>

<app_config>
 <app>
  <name>amicable_2_64</name>
  <max_concurrent>1</max_concurrent>
 </app>
 <app_version>
  <app_name>amicable_2_64</app_name>
  <plan_class>mt</plan_class>
  <cmdline>--nthreads 15</cmdline>
  <avg_ncpus>15.000000</avg_ncpus>
 </app_version>
</app_config>
ID: 32 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Amis des Lapins] Phil1966

Send message
Joined: 24 Jan 17
Posts: 6
Credit: 247,909,594
RAC: 6
   
Message 33 - Posted: 29 Jan 2017, 13:02:42 UTC - in response to Message 32.  
Last modified: 29 Jan 2017, 13:05:31 UTC

Hello to both of you :)

AND Thank You ! :)

Looking forward,

Phil1966
ID: 33 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tern

Send message
Joined: 17 Feb 17
Posts: 27
Credit: 69,545,002
RAC: 11
   
Message 126 - Posted: 17 Feb 2017, 1:33:23 UTC

It would really be nice to be able to set this on the Preferences page - many people are NOT going to be able to edit app_config files and get it right...

Also would be good to explain back on the "join" page that this project uses ALL your CPUs, because in spite of reading everything I could find about the project, and even some of the forum postings, I was surprised when "Use: 11C" showed up on 21 tasks! I can't dedicate the whole box to this project, sorry! Cut that to a more reasonable 5, one at a time...
ID: 126 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Bryan
Avatar

Send message
Joined: 23 Jan 17
Posts: 17
Credit: 278,854,007
RAC: 0
   
Message 127 - Posted: 17 Feb 2017, 4:57:25 UTC - in response to Message 126.  
Last modified: 17 Feb 2017, 5:13:29 UTC

Never mind
ID: 127 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 128 - Posted: 17 Feb 2017, 5:49:25 UTC - in response to Message 126.  
Last modified: 17 Feb 2017, 5:59:19 UTC

It would really be nice to be able to set this on the Preferences page - many people are NOT going to be able to edit app_config files and get it right...

Also would be good to explain back on the "join" page that this project uses ALL your CPUs, because in spite of reading everything I could find about the project, and even some of the forum postings, I was surprised when "Use: 11C" showed up on 21 tasks! I can't dedicate the whole box to this project, sorry! Cut that to a more reasonable 5, one at a time...

BOINC doesn't have this functionality for a number of reasons:
1) Preferences page affects all or nothing: if you set BOINC client to use preferences from one project, it will use them for all projects it runs.
2) Users with many computers must be able to set up them independently of each other.

That's why most of advanced settings can be changed only by config files. I'll create a FAQ on the main page with some examples. But I'll also look at the client code to see if I can modify the server to tell the client how many threads to use.
ID: 128 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Bryan
Avatar

Send message
Joined: 23 Jan 17
Posts: 17
Credit: 278,854,007
RAC: 0
   
Message 129 - Posted: 17 Feb 2017, 13:00:49 UTC - in response to Message 128.  

But I'll also look at the client code to see if I can modify the server to tell the client how many threads to use.


Adding a app_config files isn't rocket science. A good example with step-by-step instructions should allow anyone to change it. For those who aren't already familiar with the technique it gives them a chance to learn something.

If you do make a modification to preferences, please have a selection that is ALL or 100%. There is no other setting that will work for those with more than 4 machines (using "locations") that have different thread counts.
ID: 129 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 130 - Posted: 17 Feb 2017, 13:20:47 UTC - in response to Message 129.  

The idea is to add a setting here: https://sech.me/boinc/Amicable/hosts_user.php, so you can set it for each individual host. The default value is use all cores of course. But I'm not sure if it's possible at all with current BOINC client/server code.
ID: 130 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tern

Send message
Joined: 17 Feb 17
Posts: 27
Credit: 69,545,002
RAC: 11
   
Message 131 - Posted: 17 Feb 2017, 13:38:22 UTC - in response to Message 128.  
Last modified: 17 Feb 2017, 13:40:25 UTC


BOINC doesn't have this functionality for a number of reasons:
1) Preferences page affects all or nothing: if you set BOINC client to use preferences from one project, it will use them for all projects it runs.
2) Users with many computers must be able to set up them independently of each other.

That's why most of advanced settings can be changed only by config files. I'll create a FAQ on the main page with some examples. But I'll also look at the client code to see if I can modify the server to tell the client how many threads to use.


Several things incorrect here... "set client to use" - not settable to a project, it uses most recently changed for computing preferences. "for all projects" - true for computing preferences, but not project preferences. "independently" - computing preferences are overridden (if desired) at each host. Only project preferences are relevant for the question at hand, though.

lhcathome-dev does it. See here: <https://lhcathomedev.cern.ch/lhcathome-dev/prefs.php?subset=project>
Einstein@home does it (for GPU utilization). See here: <https://einsteinathome.org/account/prefs/project>

The number of CPUs used by mt tasks is rightfully part of the project preferences (plan class), not the BOINC preferences, so is project-specific. Multiple computers are handled through the venue (work/school/home/etc, an unlimited number of them if you do as PrimeGrid has). (And no, unfortunately, the hosts page would be very difficult to use for this, though it would be nice.)

Again, the main thing is not 'how to deal with mt tasks after joining', but 'why did a new user not know they WERE mt tasks BEFORE joining' - that points out a lack of communication in the project description and signup process, and is the problem. Also, "Bug Tracker" is not a normal place to look for project information in the forum - 'Number Crunching' is the standard heading once a project is "production".

Communication is something I harp on from projects, because it is key to success. You can have all the problems in the world and still get volunteers if you communicate clearly and frequently. (Einstein, Rosetta.) You can lose people very quickly even with no problems at all, if you never clearly communicate with them. (lhcathome-dev, Sztaki, too many others to remember.)

At least you aren't using VirtualBox without telling new users about it...
ID: 131 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 132 - Posted: 17 Feb 2017, 13:56:38 UTC - in response to Message 131.  
Last modified: 17 Feb 2017, 13:56:48 UTC

So you're talking about this page: https://sech.me/boinc/Amicable/prefs.php?subset=project and separate preferences per each venue. That's also an option. I'm finishing GPU version now, so I don't have time to fix it this week or next week. But when GPU version is out, I'll add all needed settings for CPU and GPU.
ID: 132 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 133 - Posted: 17 Feb 2017, 14:03:07 UTC - in response to Message 132.  

I actually found something in existing BOINC code, so I enabled the option "Max # CPUs". It's now on project preferences page: https://sech.me/boinc/Amicable/prefs.php?subset=project. I have no idea if it works or not.
ID: 133 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 136 - Posted: 17 Feb 2017, 14:32:54 UTC - in response to Message 133.  

Tried it, didn't work. Multithreaded applications ignore this option.
ID: 136 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tern

Send message
Joined: 17 Feb 17
Posts: 27
Credit: 69,545,002
RAC: 11
   
Message 137 - Posted: 17 Feb 2017, 14:34:58 UTC - in response to Message 136.  
Last modified: 17 Feb 2017, 14:35:35 UTC

That SHOULD work - but only after the next work fetch. It won't affect tasks already downloaded to the client.

Depending on how you set plan class in your server code...
ID: 137 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 138 - Posted: 17 Feb 2017, 14:35:38 UTC - in response to Message 137.  
Last modified: 17 Feb 2017, 14:37:12 UTC

Yes, I waited until it fetched next work unit, and it still used all cores.

P.S. I guess I know why. I use custom built binaries and anonymous platform for testing purposes.
ID: 138 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tern

Send message
Joined: 17 Feb 17
Posts: 27
Credit: 69,545,002
RAC: 11
   
Message 141 - Posted: 17 Feb 2017, 14:47:38 UTC - in response to Message 138.  
Last modified: 17 Feb 2017, 14:51:37 UTC

Something to work on for later. Right now I'd worry about putting "This app uses ALL your CPU cores, it is multi-threaded. If you want to limit this app, you will need to create an app_config file. We are working on allowing this to be set in the project preferences." on the signup page!

(Oh - once work fetch occurs, existing and new WUs use the 'new' plan class even though the existing WUs still SAY they have the old value...)

And a trivial issue; I've got a bunch of WU's "in progress" that are "ghosts" - due to my messing with the app_config file and getting it wrong a couple of times. They aren't on my machine and will time out on deadline. Luckily your deadlines are reasonably short.
ID: 141 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 142 - Posted: 17 Feb 2017, 14:51:22 UTC - in response to Message 141.  

"Max # CPUs" in project preferences actually works. My other PC which doesn't use anonymous platform successfully picked up new CPU limit.
ID: 142 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tern

Send message
Joined: 17 Feb 17
Posts: 27
Credit: 69,545,002
RAC: 11
   
Message 143 - Posted: 17 Feb 2017, 14:52:00 UTC - in response to Message 142.  
Last modified: 17 Feb 2017, 14:53:37 UTC

GREAT! I can get rid of that app_config file and not worry about it!

I'd still put something on the signup or home page about being mt and that the user should set the number of CPUs in the preferences...
ID: 143 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 146 - Posted: 17 Feb 2017, 14:55:39 UTC - in response to Message 143.  
Last modified: 17 Feb 2017, 15:07:50 UTC

Yes, I'll add a warning on the sign-up page.

P.S. Added.
ID: 146 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : # cores limitation


©2024 Sergei Chernykh