Passing multiple parameters to GNU parallel

Question

I have a sample script called sample.sh which takes three inputs X,Y and Z

>> cat sample.sh

#! /bin/bash

X=$1
Y=$2
Z=#3

file=X$1_Y$2_Z$3
echo `hostname` `date` >> ./$file

Now I can give parameters in the following way:

parallel ./sample.sh {1} {2} {3} ::: 1.0000 1.1000 ::: 2.0000 2.1000 ::: 3.0000 3.1000

Or I could do:

parallel ./sample.sh {1} {2} {3} :::: xlist ylist zlist

where xlist, ylist and zlist are files which contain the parameter list.

But what if I want to have one file called parameter.dat?

>>> cat parameter.dat
#xlist
1.0000 1.1000
#ylist
2.0000 2.1000
#zlist
3.0000 3.1000

I can use awk to read parameter.dat and produce temporary files called xlist, ylist and so on... But is there a better way using gnu-parallel itself?

Ultimately what I am looking for is to simply add more lines of xlist,ylist and zlist to parameter.dat and use the last instance of xlist, ylist or zlist to run sample.sh with, so that I keep a record of the parameter runs I have already done in parameter.dat itself. I am looking for an elegant way to do this.

Edit: My current solution is:

#! /bin/bash
tail -1 < parameter.dat | head -1 | awk '{$1=$1};1' | tr ' ' '
' > zlist
tail -3 < parameter.dat | head -1 | awk '{$1=$1};1' | tr ' ' '
' > ylist
tail -5 < parameter.dat | head -1 | awk '{$1=$1};1' | tr ' ' '
' > xlist

parallel ./sample.sh {1} {2} {3} :::: xlist ylist zlist

rm xlist ylist zlist

user_363028 · Answer

There is no built-in way of doing what you want, and your solution is not too bad.

If you control parameter.dat and it is not too big (128 KB) I would probably do:

$ cat parameter.dat
::: x valueX1 ValueX2
::: y valueY1 ValueY2
::: z ValueZ1 ValueZ2 ValueZ3

# There is on purpose no " around $() and the ::: is in parameter.dat
$ parallel --header : ./sample.sh $(cat parameter.dat)

--header : is used to ignore the first value of each line. It also means you can use {x} {y} and {z} in the command template.

This is easy to add another parameter, and you do not need to clean up tmp-files.

You are, however, restricted: Your values cannot contain space and some of the characters that have special meaning in shell (e.g. ? *). Other characters (e.g. $ ' " `) are fine.

Passing multiple parameters to GNU parallel

1 Answers