GNU parallel combinatorics, usage of list of argument multiple times

user_8632988 0
Viewed 769

I would like to use following for generating unique jobs, where {1} and {2} are unique tuples:

parallel echo {1} {2} ::: A B C D ::: A B C D

For instance in python (itertools) provides such a combinatoric generator:

permutations('ABCD', 2)

AB AC AD BA BC BD CA CB CD DA DB DC

Is there a way to implement it directly via bash? Or GNU parallel itself? Maybe skip redundant jobs somehow? But then, how can I check which parameter combinations were already used.

parallel echo {= 'if($_==3) { skip() }' =} ::: {1..5}
combinations gnu parallel-processing linux bash
3 Answers

If the values are unique:

parallel echo {= 'if($arg[1] eq $arg[2]) { skip() }' =} ::: A B C D ::: A B C D

Or more generally: 

parallel echo \
  '{= my %seen; for my $a (@arg) { $seen{$a}++ and skip() } =}' \
  ::: A B C D ::: A B C D ::: A B C D

If you want to treat AB as BA then this only runs one of the combinations:

parallel echo \
  '{= for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } } =}' \
  ::: A B C D ::: A B C D ::: A B C D

If you use these alot, remember you can use --rpl to make your own replacement strings by putting this in ~/.parallel/config

--rpl '{unique} my %seen; for my $a (@arg) { $seen{$a}++ and skip() }'
--rpl '{choose_k} for my $t (2..$#arg) { if($arg[$t-1] ge $arg[$t]) { skip() } }'

And then run:

parallel echo {unique} ::: A B C D ::: A B C D ::: A B C D
parallel echo {choose_k} ::: A B C D ::: A B C D ::: A B C D
user_3630280

One ugly solution: use Python to generate the sequence and the --link option (or --xapply):

$ parallel --xapply echo {1} {2} ::: $(python  -c "from itertools import permutations ; print(' ::: '.join([' '.join(_) for _ in zip(*list(permutations('ABCD',2)))]))")
A B
A C
A D
B A
B C
B D
C A
C B
C D
D A
D B
D C
user_17636140

You can use parallel twice if you don't mind:

# filter the first's parallel output for accepted combinations and pipe into a 2nd parallel
parallel echo {1} {2} ::: a b c ::: a b c | awk '{ if ($1 != $2) {print $0}}' | parallel echo this is the actual {1} {2}

# no one-liner for better maintenance
parallel echo {1} {2} ::: a b c ::: a b c | awk '{ if ($1 != $2) {print $0}}' > myargs
parallel --arg-file myargs echo {1} {2}
rm myargs

# Output is in both cases
a b
a c
b a
b c
c a
c b
user_29234060
Related