Countermeasures for Increased CPU Time

Message boards : Number crunching : Countermeasures for Increased CPU Time

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 994 - Posted: 27 Nov 2018, 3:30:25 UTC

Greetings All,

I am sure that many of you have noticed that the ratio of CPU time to run time for each task has been steadily increasing. The reason for this is that each task requires all of the prime numbers in an ever-growing range to be calculated, and this calculation can only be performed by the CPU.

As we approach the end of the project the CPU time for each task will continue to increase, and once it reaches the run time for a given task, you will notice that your GPU utilization begins to decrease. If you would like to keep GPU utilization high, you will need to run more than 1 task per GPU at some point. For owners of any Nvidia Turing (RTX 20-series) or Volta cards, the utilization issue has existed for a week or more already, and we are fast approaching the time when GTX 1080ti and other high performance cards will be affected.

Eight days ago I switched to 2 concurrent tasks on my RTX 2070 and utilization is now hovering around 98%, but slowly beginning to decrease. Once it reaches ~85% I will begin running 3 concurrent tasks.

To determine GPU utilization you can use Hardware Info 64 (https://www.hwinfo.com/download/), GPU-Z (https://www.techpowerup.com/gpuz/) or any other software you prefer.

Below I have put some directions together for how to run multiple tasks on a GPU and some general guidelines for how many concurrent tasks are likely required to get full utilization on some common GPUs. Keep in mind that this number will be affected by your CPU speed, CPU cache size, RAM speed, number of cores you are using, whether hyperthreading or SMT is enabled on your CPU, etc. They are very general, but should be in the right ballpark.

To run two tasks concurrently, make a file called app_config.xml (make a text file, then change the extension to .xml from .txt) in your Amicable Numbers data directory (on Windows the default location is C:\ProgramData\BOINC\projects\sech.me_boinc_amicable\). Put the following text in it and save the file;

<app_config>
<app>
<name>amicable_10_20</name>
<gpu_versions>
<gpu_usage>0.5</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
</app_config>

Make sure that the file is a true .xml file and not just a .txt file with “app_config.xml” as the name. To make sure this is the case, show file name extensions in the folder (instructions here; http://kb.winzip.com/kb/entry/26/)

Once that is saved, select the Options menu in BOINC and click "Read Config Files" and two units should start running at the same time. Each unit will get a full CPU core and half of the GPU.

To run more than two tasks, keep decreasing the <gpu_usage> value. To run 3 tasks, use 0.33, 4 tasks will use 0.25 and 5 tasks will use 0.2, etc. Everything else should stay the same. This will tell BOINC to dedicate 1 CPU core and a defined fraction of the GPU to a given task.

Currently, the following number of concurrent tasks is likely required to have full GPU utilization. Again, this is just a general guideline and will vary significantly from computer to computer; checking GPU utilization will be the best way of knowing how many tasks are required. I know very little about AMD cards, so if someone has AMD guidelines, please feel free to post.

  • Titan V/ RTX 2080 ti - 3-4 tasks
  • RTX 2080 - 3 tasks
  • RTX 2070 - 2 tasks
  • GTX 1080 ti - 1 task
  • GTX 1080/GTX 1070 ti - 1 task
  • GTX 1070 - 1 task
  • GTX 1060 - 1 task
  • GTX 1050 ti - 1 task
  • GTX 1050 - 1 task
  • GT 1030 - 1 task

ID: 994 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Matt Kowal
Avatar

Send message
Joined: 12 Feb 17
Posts: 14
Credit: 177,829,581
RAC: 0
   
Message 995 - Posted: 27 Nov 2018, 5:22:56 UTC - in response to Message 994.  
Last modified: 27 Nov 2018, 5:23:59 UTC

Thank you for sharing this. I am running 2 tasks on my (aging!) 7970 as GPU utilization had dropped to ~93%.
ID: 995 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Azmodes

Send message
Joined: 5 Mar 17
Posts: 20
Credit: 3,515,133,859
RAC: 82,867
    
Message 997 - Posted: 27 Nov 2018, 9:49:15 UTC
Last modified: 27 Nov 2018, 10:01:32 UTC

There is no way to tell BOINC to use different app_configs for different cards on the same host, right? For instance, one of my machines has an RTX 2070 and two GTX 1060 3GB and I'm getting screenlag with two tasks at the same time (screen connected to one of the GTXs), so it would be ideal to have the RTX crunch two and the rest only one.

Is it really necessary to assign a full CPU core to each task even with the decrease in usage more tasks entail?
ID: 997 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jozef J

Send message
Joined: 24 Jan 17
Posts: 20
Credit: 1,193,014,322
RAC: 0
   
Message 998 - Posted: 27 Nov 2018, 15:43:06 UTC
Last modified: 27 Nov 2018, 15:46:30 UTC

Hi, after weeks, when i start run two task on 2080Ti ,compare to previous (my last post here)
i see increasing gpu memory use to 3900-4000 mb And gpu load fall litle bit to 85-90% always on kernel 23,
Soo task are continually increase memory alocation and proably need more cpu power,,?? theory..
I use my 2080Ti on main pc win 10.. where also i do some gpu task(gaming,render,..etc) if somebody watch on my run time tasks, most stable time you can see is on night.. but if you will use 2080ti like only boinc machine" with linux ,results and day Rac can be better 15%+. and there you can try run 4 task on one gpu ,,, but it will be lagy. cpu power(ipc) can help too
maybe put one core per task is good idea, somebody try)
It will be amazing in jaunary/feb ... when memory use will increase we will back to max four task on 2080Ti(3000mb per task ) ..? and also day rac will decrease , on older gf cards will decreasing even pronounced...?
ID: 998 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Matt Kowal
Avatar

Send message
Joined: 12 Feb 17
Posts: 14
Credit: 177,829,581
RAC: 0
   
Message 999 - Posted: 27 Nov 2018, 19:39:44 UTC - in response to Message 997.  

Is it really necessary to assign a full CPU core to each task even with the decrease in usage more tasks entail?


It likely depends on the GPU/CPU, but for my AMD 7970/i5-2500k I reserve only 0.5 CPU for each task.
ID: 999 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 1000 - Posted: 27 Nov 2018, 22:11:10 UTC

Hi All,

Thank you for your comments. Your utilization issue is very interesting Matt and highlights a potential difference between AMD and Nvidia cards. My Nvidia cards do not show a decrease in utilization until CPU time is 90% of Run Time or higher. Right now your CPU time is about 1/10th of the Run Time when running two concurrent tasks, so about 1/5 of a CPU core used per GPU. On Nvidia cards this would not cause a decrease in utilization, however AMD appears to handle these tasks differently. This may mean that AMD cards will require even more concurrent tasks to maintain full utilization.

Azmodes and Jozef's cases are somewhat different in that the utilization will decrease when 1 CPU core is no longer capable of feeding prime numbers to 1 GPU. When that happens it will be necessary to run multiple tasks per GPU and to dedicate more than 0.5 CPU cores to each one. The goal of adding more tasks is to increase the CPU cores to GPUs ratio. Running 2 concurrent tasks with 0.5 CPU cores each is the same as running 1 task with 1 CPU core. Running 3 concurrent tasks with 0.5 CPU cores each is a 1.5:1 ratio, so utilization will be improved. With that said though, BOINC handles fractional CPU usage strangely, so even if you specify 0.5 cores, it may end up using more than that, so it may give the appearance that 2 tasks with 0.5 cores is an improvement over 1 task with 1 core, but that is illusory.

Ultimately; my suggestion (as it is what I do) is to always dedicate 1 CPU core per task and wait until this is insufficient to maintain full GPU utilization and then just add another task. This will make sure that you are not running a greater number of tasks than necessary as, like Jozef pointed out, memory usage will increase with each added task.

I checked Azmodes' computer and noticed that at 9:25 UTC today the config file was updated. Looks like you've gone from 1 task every ~220s to 2 tasks every ~330s, which is a 50% increased throughput! Nice!!

Jozef; your computer still looks like it is CPU bound, as the CPU Time and Run Time of each task is identical. If you want to get full utilization, it is probably time for 3 tasks ;)
ID: 1000 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 1001 - Posted: 27 Nov 2018, 22:19:07 UTC - in response to Message 997.  

Hi Azmodes,

As far as I know there is no way to tell BOINC to send different numbers of tasks to different GPUs, but this may be possible with the cc_config file, which you can use to specify which GPUs are used for specific jobs or at all. I am not familiar with the cc_config files though. Here is the link to the cc_config page for BOINC though in case you want to try things out; [url]https://boinc.berkeley.edu/wiki/Client_configuration [/url]

Good luck!
ID: 1001 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 1002 - Posted: 27 Nov 2018, 22:27:05 UTC - in response to Message 998.  

Hi Jozef,

As we get closer to the end of the project I think that it may be possible for Sergei to reduce the memory requirements (but I am no expert so he would have to confirm it). Right now, the prime tables up to ~1.414*10^10 are being loaded into memory, however the calculations may not require the entire prime table for this range anymore and this table is the majority of the VRAM requirement as far as I know.

Happy crunching!
ID: 1002 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sergei Chernykh
Project administrator
Project developer

Send message
Joined: 5 Jan 17
Posts: 534
Credit: 72,451,573
RAC: 0
   
Message 1003 - Posted: 27 Nov 2018, 23:28:10 UTC

i see increasing gpu memory use to 3900-4000 mb And gpu load fall litle bit to 85-90% always on kernel 23

Don't use kernel size 23 with multiple tasks, kernel size=21 will do just fine. Memory usage shouldn't be increasing at this stage of the search, it mainly depends on kernel size you select.
ID: 1003 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Azmodes

Send message
Joined: 5 Mar 17
Posts: 20
Credit: 3,515,133,859
RAC: 82,867
    
Message 1004 - Posted: 28 Nov 2018, 0:22:08 UTC - in response to Message 1001.  

Hi Azmodes,

As far as I know there is no way to tell BOINC to send different numbers of tasks to different GPUs, but this may be possible with the cc_config file, which you can use to specify which GPUs are used for specific jobs or at all. I am not familiar with the cc_config files though. Here is the link to the cc_config page for BOINC though in case you want to try things out; [url]https://boinc.berkeley.edu/wiki/Client_configuration [/url]

Good luck!

I'm aware of that link, but thanks all the same. The workaround I did earlier today is simply to exclude the GTX from AN and use it for another project with no lag (Collatz).
ID: 1004 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Speedy

Send message
Joined: 6 Jun 17
Posts: 83
Credit: 201,069,888
RAC: 7,119
   
Message 1018 - Posted: 13 Dec 2018, 4:18:08 UTC - in response to Message 995.  

Thank you for sharing this. I am running 2 tasks on my (aging!) 7970 as GPU utilization had dropped to ~93%.

Matt I had a look at your tasks. In terms of run-time am I correct in saying that you are returning 2 results roughly every 45 minutes on your 7970?
ID: 1018 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jozef J

Send message
Joined: 24 Jan 17
Posts: 20
Credit: 1,193,014,322
RAC: 0
   
Message 1019 - Posted: 13 Dec 2018, 12:26:43 UTC

Hi, for exemple i run now six task , also i try seven but system was bit laggy .
six task 0.16 about 9700mb average gpu ram usage totall..
but today if you have more gpu ram on your gf. card ,you can run more tasks
ID: 1019 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 1020 - Posted: 13 Dec 2018, 16:57:26 UTC - in response to Message 1019.  

You can use a smaller kernel size to reduce VRAM usage and run more tasks if you are not at 100%. Try 10 tasks with kernel size 22 and 1 full CPU core dedicated to each task and this should clear up the bottleneck if your card isn't at 100%. You can try kernel size 21 as well if too much memory is used.

The critical thing is to dedicate 1 CPU core to each task and don't run other stuff on the CPU.
ID: 1020 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Matt Kowal
Avatar

Send message
Joined: 12 Feb 17
Posts: 14
Credit: 177,829,581
RAC: 0
   
Message 1021 - Posted: 13 Dec 2018, 20:43:08 UTC - in response to Message 1018.  

Thank you for sharing this. I am running 2 tasks on my (aging!) 7970 as GPU utilization had dropped to ~93%.

Matt I had a look at your tasks. In terms of run-time am I correct in saying that you are returning 2 results roughly every 45 minutes on your 7970?


That is correct. GPU is a 7970 @ 985Mhz/1425mhz

I run 3x Numberfields@home tasks and the other core feeds 2x amicable numbers tasks on the 7970 and 1x Primegrid PPS Sieve task on a GTX 1060 3gb.
Utilization is ~90% on a 2500k
ID: 1021 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Speedy

Send message
Joined: 6 Jun 17
Posts: 83
Credit: 201,069,888
RAC: 7,119
   
Message 1022 - Posted: 13 Dec 2018, 22:02:17 UTC - in response to Message 1019.  

Hi, for exemple i run now six task , also i try seven but system was bit laggy .
six task 0.16 about 9700mb average gpu ram usage totall..
but today if you have more gpu ram on your gf. card ,you can run more tasks

I am not sure how you can run 6 tasks on GPU as none of your GPU's have got 9700 MB of GPU RAM
ID: 1022 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Speedy

Send message
Joined: 6 Jun 17
Posts: 83
Credit: 201,069,888
RAC: 7,119
   
Message 1023 - Posted: 13 Dec 2018, 22:05:35 UTC - in response to Message 1021.  

Thank you for sharing this. I am running 2 tasks on my (aging!) 7970 as GPU utilization had dropped to ~93%.

Matt I had a look at your tasks. In terms of run-time am I correct in saying that you are returning 2 results roughly every 45 minutes on your 7970?


That is correct. GPU is a 7970 @ 985Mhz/1425mhz

Thank you for this information
ID: 1023 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kellen

Send message
Joined: 14 Nov 17
Posts: 70
Credit: 1,000,005,236
RAC: 0
   
Message 1024 - Posted: 13 Dec 2018, 22:52:00 UTC - in response to Message 1022.  

The GTX 1080 ti and RTX 2080 ti both have 11GB RAM, but only 4GB shows up in BOINC data ;)
ID: 1024 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Speedy

Send message
Joined: 6 Jun 17
Posts: 83
Credit: 201,069,888
RAC: 7,119
   
Message 1025 - Posted: 14 Dec 2018, 2:51:32 UTC - in response to Message 1024.  

The GTX 1080 ti and RTX 2080 ti both have 11GB RAM, but only 4GB shows up in BOINC data ;)

Fair point I forgot about that. Same as the 1070 TI at his 8 GB but only 4 GB shows
ID: 1025 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Vitalii Koshura

Send message
Joined: 28 Aug 17
Posts: 2
Credit: 53,711,973
RAC: 0
   
Message 1026 - Posted: 15 Dec 2018, 13:54:39 UTC - in response to Message 1025.  

The GTX 1080 ti and RTX 2080 ti both have 11GB RAM, but only 4GB shows up in BOINC data ;)

Fair point I forgot about that. Same as the 1070 TI at his 8 GB but only 4 GB shows


There is an existing issue: https://github.com/BOINC/boinc/issues/1773
ID: 1026 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile The Sarge

Send message
Joined: 5 Nov 17
Posts: 8
Credit: 1,071,354,589
RAC: 0
   
Message 1027 - Posted: 15 Dec 2018, 15:46:28 UTC - in response to Message 994.  

Good Day. Created the app_config.xml file as described, saved it as an xml file (using Word), verified name/extension was correct, placed in the amicable sub-directory of the Boinc projects parent directory, had the Boinc manager read the config file, and received this notice/message:

Missing <app_config> in app_config.xml

Also checked and saw only once instance of amicable tasks running, not two as I wanted, and verified utilization was still in the 65%-75% range

Any suggestions?

Thank you for any assistance or insights offered.
ID: 1027 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · Next

Message boards : Number crunching : Countermeasures for Increased CPU Time


©2024 Sergei Chernykh