I have an xargs command:
xargs -a es_doc_positions.txt -P 64 -I {} populate_es_index.py --start {} --end {}
Is it possible to take the input {} and apply simple arithmetic to it? I want to supply the end argument something like:
$(({} + 100000 - 1))
I haven't had any luck with what I've tried so far.
xargs does not invoke a shell unless you tell it to. Thus, it cannot perform shell arithmetic on its own.
Tell xargs to invoke a shell, and perform your arithmetic inside that shell:
xargs -a es_doc_positions.txt -P 64 -n 16 sh -c '
for arg do
./populate_es_index.py --start "$arg" --end "$(( arg + 100000 - 1 ))"
done
' _
The trailing underscore is a placeholder for $0, so values passed by xargs into the shell are assigned to $1 and later.
The -n argument controls how many instances each sh invocation runs. Lower values will result in fewer unused CPU cycles when you get near the end of your batch of inputs (when fewer than P*n values remain to be processed); higher-value will reduce CPU overhead in starting up shell instances.
Note that it's very intentional that this solution does not use -I. If you wanted to amend it to do so, you'd want to put the placeholder as a final argument, after the _, not as a substring inside the code; using a placeholder as a substring introduces shell injection vulnerabilities, and also prevents a single shell instance from processing multiple inputs.