How to control parallel tasks in Linux to avoid too much context switch

user_2288882 0
Viewed 618

Now I'm using Linux to perform the following task:

while read parameter
do
    ./program_a $parameter $parameter.log 2>&1 &
done < parameter_file

Each parameter refers to the name of the file to be processed. Each file contains a different number of lines to process.

For example:
Parameter file contains:

File_A
File_B
File_C

File_A contains 1k lines, File_B contains 10k lines and File_C contains 1000k lines, which means that in the above script program_a simultaneously processes 1000 lines, 10k lines and 1000k lines respectively. The processing time for each task is almost linearly dependent on the number of lines and each task is independent.

I have 6 cores CPU with 12 threads. Because processing time could vary so that after running tasks for File_A and File_B, only one core will process the task for File_C. This is wasting resources.

I want to split each file to 1k lines and run them simultaneously. But for this example there will be 1011 tasks running (1k for each task). I think this will lead to a serious overly context switch problem. Maybe I can tune to number in each line to solve this problem, but I don't think this is a good solution.

My thought is to limit the tasks running will be always 6 tasks which means always using maximum number of cores to run and reduce context switches to as few as possible. But I don't know how to modify my script to achieve this goal. Anyone can give me some advice?

shell multithreading unix linux bash
4 Answers

I assume program_a can read a single file.

Then this should work using GNU Parallel:

parallel --pipepart --block 10k --cat program_a :::: File_A File_B File_C

Adjust the 10k to be the size of your 1000 lines.

It does much of the same as @Marcus Rickert's answer, but hides the complexity from you and cleans up temporary files.

If program_a can read from a fifo, this should be faster:

parallel --pipepart --block 10k --fifo program_a :::: File_A File_B File_C

If program_a can read from stdin, it will be shorter:

parallel --pipepart --block 10k program_a :::: File_A File_B File_C

If you really must have excactly 1000 arguments try:

cat File_A File_B File_C | parallel --pipe -L1000 -N1 --cat program_a

or:

cat File_A File_B File_C | parallel --pipe -L1000 -N1 program_a
user_3630280
Related