What's the correct usage of sed with parallel --jobs option?

user_16388827 0
Viewed 207

parallel -a input --colsep ' ' --jobs 100 -I {} sed -i 's/{1}/{2}/g' file

input is a file delimited by space, where the first column is pattern and the second column is replacement.

The problem is that after I ran the command, not all patterns were replaced in file. Then I ran the same command again, more patterns were replaced, but still not all. However, if I change --jobs 100 to --jobs 1, it will work as expected (but much slower).

Is there any parameter necessary missing in my command?

gnu-parallel sed
2 Answers

Sounds more like you have a race condition. If you have several sed processes writing to the file, one will win, and the other(s) will lose.

Having multiple processes process the same file is hugely suboptimal anyway; just generate a single sed script and then run it once. Or if you really want to parallelize, split the input file into smaller pieces, run the generated sed script on each in parallel, and then concatenate them back when you are done.

Parallel processing helps when your task is CPU bound, but this one is I/O bound; you are simply creating congestion by having several processes fight over the access to bytes from the disk, and then in this case also fighting over write access back to the same file.

There are many examples of how to generate a sed script; here's a quick and dirty one which will however not work on some platforms where sed -f - does not read the script from standard input.

sed 's%^\([^ ]*\) \([^ ]*\)$%s/\1/\2/g%' input |
sed -f - file >temp  # or sed -f - -i file

I omitted the -i option so that you can check that this does what you want before plunging ahead and deploying it in production. The commented-out version is what you would use once you are satisfied that this really does what you want.

There is still the question of replacement precedence. If you have s/a/b/ and s/b/c/ then do you want effectively s/a/c/, or the opposite? If you have s/abc/x/ and s/abcdef/y/, should abcdef always become y, or is xdef what you expect? A common hack is to sort the replacements by length so that the longer ones always get executed before the shorter ones; then at least you know what to expect.

user_8741880

Let us assume that input is big and file is huge.

You really do not want to read file more than once.

First you need to convert input into a single big sed script.

cat input | parallel --colsep ' ' echo s/{1}/{2}/g >bigsed

As @tripleee says, you may need to sort this, so the longest source string is first.

Then you need to split file into one chunk per CPU thread, run the script on each chunk and finally append the replaced chunks back in order:

parallel --pipepart -a file -k sed -f bigsed > replaced

You will need that /tmp has enough free space to contain replaced or set $TMPDIR to a dir that is.

user_3630280
Related