I am using OpenMP to parallelize a for loop in my code by adding "#pragma omp parallel for" before the for loop. The code runs without producing any errors, but I have noticed that the core utilization is not constant throughout the execution. At the beginning, I get 100% utilization on all threads, but as the execution continues, this number drops to ~70% for all threads. I monitored this behavior using htop (I am running on a Linux machine). At the beginning, all of the threads show green bars with 100% on the right. After sometimes, the green starts to decrease and some red starts to creep up like this:
I have looked and saw that the red shows up whenever kernel threads are used. However, I still do not understand why it shows up in the first place. This behavior only occurs when the for loop is parallelized. When I take out the #pragma statement, the code runs at 100% throughout. My for loop has the following format:
double xe1, ye1, ze1, te1, e1;
double xe2, ye2, ze2, te2, e2;
double xi1, yi1, zi1, ti1;
double xi2, yi2, zi2, ti2;
bool calculate_signal = true;
time_t loop_start, loop_end;
int max = omp_get_max_threads();
omp_set_dynamic(0);
omp_set_num_threads(max);
#pragma omp parallel for
for (int k = 0; k < index; k++) {
time(&loop_start);
AvalancheMicroscopic aval;
aval.SetSensor(&sensor);
aval.EnableSignalCalculation(calculate_signal);
aval.AvalancheElectron(electrons.at(k*4 + 0),electrons.at(k*4 + 1),electrons.at(k*4 + 2),electrons.at(k*4 + 3), 0.1, 0,0,0);
int np = aval.GetNumberOfElectronEndpoints();
DriftLineRKF drift;
drift.SetSensor(&sensor);
drift.EnableSignalCalculation(calculate_signal);
for (int j = np; j--;) {
aval.GetElectronEndpoint(j, xe1, ye1, ze1, te1, e1,
xe2, ye2, ze2, te2, e2, status);
drift.DriftIon(xe1, ye1, ze1, te1);
}
time(&loop_end);
std::cout << "Time for " << np << " electrons: " << double(loop_end - loop_start) << std::endl;
}
This code is written using a toolkit called Garfield++ (https://garfieldpp.web.cern.ch/garfieldpp/; https://gitlab.cern.ch/garfield/garfieldpp/-/tree/master/). Here, electrons() is a 1d vector that holds coordinates. The index variable that can range from 1 to 7000 depending on the user. I have ran with index = 1000 and index = 200 and gotten the same behavior. Also, np can range from 1 to ~2000 and is something that is determined by the outer loop, so user has no control over this.
I have tried schedule(dynamic), but that didn't seem to help. Any insight into what caused this kind of behavior and how to fix it is greatly appreciated. Thank you.
Edit 1: The code is edited to provide more details.
Edit 2: I ran the code again to find the average time it takes for each loop. The answer, as expected, is not very straightforward (at least not for me). The code were run twice: one without parallelization (#pragma statement is commented out) and one with parallelization. Both executions were run by using the same electrons vector. Below, I have printed the time (in unit of seconds) each loop takes. The amount of time it takes to finished the inner loop is on the right side of the semicolon, while the number on the left of the semicolon np. I calculated the time by using time_t. I edited the above code to show what I did. Also, another thing to note is that the first case (w/o parallel) took 6065 sec to run while the second case took 2183 sec.
Using 1 core Using 10 cores
Time for 1 electrons: 7 Time for 1 electrons: 11
Time for 1565 electrons: 91 Time for 1 electrons: 0
Time for 1 electrons: 8 Time for 1 electrons: 10
Time for 1 electrons: 7 Time for 92 electrons: 12
Time for 3229 electrons: 162 Time for 1 electrons: 12
Time for 1 electrons: 7 Time for 1 electrons: 12
Time for 1 electrons: 7 Time for 397 electrons: 21
Time for 1 electrons: 6 Time for 563 electrons: 41
Time for 1 electrons: 8 Time for 902 electrons: 79
Time for 1 electrons: 7 Time for 572 electrons: 18
Time for 1 electrons: 7 Time for 1 electrons: 14
Time for 2028 electrons: 103 Time for 1207 electrons: 80
Time for 898 electrons: 57 Time for 391 electrons: 4
Time for 1 electrons: 7 Time for 1392 electrons: 1
Time for 1 electrons: 8 Time for 297 electrons: 9
Time for 1 electrons: 7 Time for 1439 electrons: 39
Time for 1 electrons: 8 Time for 1 electrons: 14
Time for 1739 electrons: 91 Time for 1 electrons: 13
Time for 1977 electrons: 99 Time for 1 electrons: 14
Time for 3053 electrons: 152 Time for 393 electrons: 21
Time for 166 electrons: 20 Time for 543 electrons: 31
Time for 1 electrons: 8 Time for 1767 electrons: 38
Time for 1 electrons: 7 Time for 744 electrons: 5
Time for 573 electrons: 39 Time for 436 electrons: 44
Time for 31 electrons: 16 Time for 2444 electrons: 19
Time for 774 electrons: 51 Time for 2469 electrons: 4
Time for 680 electrons: 44 Time for 368 electrons: 30
Time for 514 electrons: 36 Time for 895 electrons: 29
Time for 1281 electrons: 69 Time for 2223 electrons: 6
Time for 2847 electrons: 148 Time for 1 electrons: 8
Time for 1004 electrons: 57 Time for 1394 electrons: 9
Time for 1422 electrons: 76 Time for 1 electrons: 4
Time for 762 electrons: 48 Time for 3283 electrons: 9
Time for 1817 electrons: 96 Time for 1 electrons: 3
Time for 2133 electrons: 110 Time for 1 electrons: 11
Time for 895 electrons: 52 Time for 1010 electrons: 5
Time for 939 electrons: 55 Time for 1 electrons: 12
Time for 743 electrons: 44 Time for 484 electrons: 5
Time for 1121 electrons: 63 Time for 112 electrons: 19
Time for 2482 electrons: 126 Time for 733 electrons: 0
Time for 1449 electrons: 79 Time for 1 electrons: 14
Time for 2037 electrons: 106 Time for 143 electrons: 9
Time for 4227 electrons: 204 Time for 1 electrons: 11
Time for 611 electrons: 39 Time for 394 electrons: 36
Time for 738 electrons: 44 Time for 1489 electrons: 3
Time for 1416 electrons: 75 Time for 540 electrons: 66
Time for 519 electrons: 39 Time for 1 electrons: 13
Time for 676 electrons: 45 Time for 1835 electrons: 27
Time for 497 electrons: 36 Time for 837 electrons: 5
Time for 303 electrons: 27 Time for 559 electrons: 18
Time for 2695 electrons: 144 Time for 828 electrons: 8
Time for 120 electrons: 18 Time for 811 electrons: 28
Time for 142 electrons: 19 Time for 136 electrons: 49
Time for 809 electrons: 48 Time for 1881 electrons: 9
Time for 27 electrons: 14 Time for 1 electrons: 12
Time for 343 electrons: 26 Time for 816 electrons: 13
Time for 168 electrons: 19 Time for 651 electrons: 2
Time for 78 electrons: 15 Time for 1 electrons: 11
Time for 883 electrons: 50 Time for 878 electrons: 18
Time for 563 electrons: 36 Time for 361 electrons: 14
Time for 418 electrons: 30 Time for 2581 electrons: 26
Time for 808 electrons: 46 Time for 155 electrons: 10
Time for 1977 electrons: 96 Time for 518 electrons: 50
Time for 610 electrons: 38 Time for 1241 electrons: 8
Time for 326 electrons: 26 Time for 722 electrons: 4
Time for 84 electrons: 16 Time for 223 electrons: 6
Time for 3116 electrons: 145 Time for 362 electrons: 10
Time for 979 electrons: 53 Time for 865 electrons: 15
Time for 259 electrons: 24 Time for 1 electrons: 10
Time for 761 electrons: 44 Time for 791 electrons: 3
Time for 2690 electrons: 128 Time for 1 electrons: 11
Time for 795 electrons: 48 Time for 1 electrons: 10
Time for 1 electrons: 8 Time for 826 electrons: 83
Time for 1762 electrons: 80 Time for 2428 electrons: 2
Time for 2410 electrons: 124 Time for 1159 electrons: 17
Time for 454 electrons: 33 Time for 149 electrons: 10
Time for 1 electrons: 7 Time for 1355 electrons: 4
Time for 2912 electrons: 146 Time for 457 electrons: 25
Time for 1624 electrons: 85 Time for 523 electrons: 26
Time for 857 electrons: 54 Time for 561 electrons: 16
Time for 4517 electrons: 221 Time for 235 electrons: 43
Time for 2083 electrons: 113 Time for 708 electrons: 4
Time for 1268 electrons: 75 Time for 1388 electrons: 51
Time for 7548 electrons: 358 Time for 369 electrons: 3
Time for 557 electrons: 40 Time for 403 electrons: 7
Time for 1 electrons: 7 Time for 1234 electrons: 13
Time for 1 electrons: 7 Time for 3203 electrons: 13
Time for 1 electrons: 8 Time for 262 electrons: 27
Time for 1167 electrons: 65 Time for 444 electrons: 43
Time for 1410 electrons: 77 Time for 686 electrons: 7
Time for 1 electrons: 7 Time for 7654 electrons: 26
Time for 4490 electrons: 412 Time for 731 electrons: 23
Time for 3093 electrons: 146 Time for 1336 electrons: 77
Time for 1170 electrons: 67 Time for 2117 electrons: 111
Time for 1 electrons: 7 Time for 1750 electrons: 96
Time for 1 electrons: 8 Time for 1089 electrons: 58
Time for 1 electrons: 7 Time for 3823 electrons: 185
Time for 1 electrons: 7 Time for 996 electrons: 54
Time for 713 electrons: 45 Time for 1 electrons: 7
Time for 973 electrons: 57 Time for 1 electrons: 8