Why doesn't xargs in parallel mode work with grep?

Viewed 145
cat gg

gives

192

and

cat tmpfilelist

gives

android/app/src/main/res/drawable-mdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png
android/app/src/main/res/drawable-mdpi/src_assets_images_ic_biggames.png  
android/app/src/main/res/drawable-xhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png  
android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png   
android/app/src/main/res/drawable-xxhdpi  /src_assets_images_alerts_iconnotificationsmartpluggreen.png  
android/app/src/main/res/drawable-xxhdpi/src_assets_images_ic_biggames.png  
gg  
ios/WebRTC.framework/Headers/RTCCallbackLogger.h  
ios/WebRTC.framework/Headers/RTCFileLogger.h  
ios/WebRTC.framework/Headers/RTCLogging.h  

I run xargs with parralel mode on - this does not find the required text "192":

cat tmpfilelist | \xargs   -P0 -t  -I {} \bash  -c "\grep -C 2 -H -I -r 192 {}" |& \grep -C 3 "192 gg"
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png'  
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png'  
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdp/src_assets_images_ic_biggames.png'  
bash -c '\grep -C 2 -H -I -r 192 gg'
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCCallbackLogger.h'  
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCFileLogger.h'  
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCLogging.h'  

When I disable parralel mode it succesfully finds the text "192" in file "gg":

cat tmpfilelist | \xargs   -t  -I {} \bash  -c "\grep -C 2 -H -I -r 192 {}" |& \grep -C 3 "192 gg"
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png'  
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png'  
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdp/src_assets_images_ic_biggames.png'  
bash -c '\grep -C 2 -H -I -r 192 gg'  
gg:192  
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCCallbackLogger.h'  
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCFileLogger.h'  

Any reason why parallel mode would break grep? Or have I made a mistake somwhere?

Many thanks

3 Answers

The issue is indeed due to the parallel execution. The issue is that the output is not deterministic when the you run it in parallel. I create a similar test

bash-5.0# ls -alh
total 32K
drwxr-xr-x    9 root     root         288 May 20 17:00 .
drwxr-xr-x    1 root     root        4.0K May 20 17:11 ..
-rw-r--r--    1 root     root           3 May 20 17:00 a
-rw-r--r--    1 root     root           3 May 20 17:00 b
-rw-r--r--    1 root     root           3 May 20 17:00 c
-rw-r--r--    1 root     root           4 May 20 17:00 d
-rw-r--r--    1 root     root           3 May 20 17:00 e
-rw-r--r--    1 root     root           3 May 20 17:00 f
-rw-r--r--    1 root     root          11 May 20 17:00 files

bash-5.0# tail -n +1 *
==> a <==
192
==> b <==
193
==> c <==
192
==> d <==
195

==> e <==
200
==> f <==
198
==> files <==
a
b
c
d
e
f

Now if we run your two commands without the last grep the outputs are like below

Without parallel

bash-5.0# cat files | \xargs   -t  -I {} \bash  -c "\grep -C 2 -H -I -r 192 {}"
bash -c \grep -C 2 -H -I -r 192 a
a:192
bash -c \grep -C 2 -H -I -r 192 b
bash -c \grep -C 2 -H -I -r 192 c
c:192
bash -c \grep -C 2 -H -I -r 192 d
bash -c \grep -C 2 -H -I -r 192 e
bash -c \grep -C 2 -H -I -r 192 f

With Parallel

bash-5.0# cat files | \xargs  -P4 -t  -I {} \bash  -c "\grep -C 2 -H -I -r 192 {}"
bash -c \grep -C 2 -H -I -r 192 a
bash -c \grep -C 2 -H -I -r 192 b
bash -c \grep -C 2 -H -I -r 192 c
bash -c \grep -C 2 -H -I -r 192 d
bash -c \grep -C 2 -H -I -r 192 e
a:192
bash -c \grep -C 2 -H -I -r 192 f
c:192

Hope you can see how it is affecting the output and orders of your line. So your issue is that when you do grep -C 3 192 gg, you are supposed to get 3 lines before printing of 192 gg and 3 after, which you actually do get.

But gg:192 is printed sometime later because of each command sending their output in parallel on the same output terminal

Not an answer - but pointing to possible problem.

The code runs two filters in pipeline

  • grep -C 2 -H -I -r 192 FILENAMe
  • grep -C 3 "192 gg"

The output from the first line will follow the format 'FILENAME:DATA'. In the example above

gg:192

The second filter is not going to find the pattern '192 gg'.

So the question may be how does the non-parallel produces any output ?

Short answer

Your final grep -C 3 requests 3 lines of context around the 192 gg. In the parallel case, this might or might not be enough to find the line containing gg:192.

Details

Before the final grep, your output will consist of a number of lines like:

bash -c \grep -C 2 -H -I -r 192 <filename>

which are echoed to stderr just before xargs launches each bash -c ... command, and the line

gg:192

is echoed to stdout when the relevant bash -c ... command (i.e. the one involving the file gg) finds the match.

In the parallel case, the whole output that is piped into your the final grep (once stderr and stdout are combined using |&) might look something like this, where I have replaced the other filenames for brevity:

bash -c \grep -C 2 -H -I -r 192 some_file
bash -c \grep -C 2 -H -I -r 192 some_other_file
bash -c \grep -C 2 -H -I -r 192 another_file
bash -c \grep -C 2 -H -I -r 192 gg
bash -c \grep -C 2 -H -I -r 192 and_another_file
bash -c \grep -C 2 -H -I -r 192 yet_another_file
gg:192
bash -c \grep -C 2 -H -I -r 192 and_yet_another_file

In this example, the bash -c ... commands involving and_another_file and yet_another_file were launched after the one involving gg was launched, but before the one involving gg wrote its output (or at least, before any stdio buffers associated with that output were flushed), so they appear between the lines containing 192 gg and gg:192.

The number of such intervening lines between the lines containing 192 gg and the gg:192 (in this example, 2) will depend on timings and the number of other parallel tasks being launched after the one involving gg. This will vary, and for example if you inserted a sleep statement (e.g. ... \bash -c "sleep 1; \grep -C ...) then there would tend to be more such lines. In any case, you are then piping it to a grep -C 3 to extract 3 lines of context. If it so happens that there are fewer than 3 intervening lines, then this grep -C will find the line containing gg:192, but if there are 3 or more, then it will be outside the requested amount of context and will not be in the final output.

In the serial case, however, the gg:192 line is guaranteed to always be immediately after the 192 gg line, as follows:

bash -c \grep -C 2 -H -I -r 192 some_file
bash -c \grep -C 2 -H -I -r 192 some_other_file
bash -c \grep -C 2 -H -I -r 192 another_file
bash -c \grep -C 2 -H -I -r 192 gg
gg:192
bash -c \grep -C 2 -H -I -r 192 and_another_file
bash -c \grep -C 2 -H -I -r 192 yet_another_file
bash -c \grep -C 2 -H -I -r 192 and_yet_another_file

and so it is always within the 3 lines of context.

Related