Correctly using --shard in GNU parallel

user_17398234 0
Viewed 32

I'd like to use the --shard flag in GNU parallel to do some concurrency control.

My input file (input.csv) looks like this:

node-foo,op1
node-bar,op2
node-baz,op3
node-foo,op4
node-blah,op5
...

I'm using parallel to send ops to nodes. Node names may appear more than once in the first column. I'd like to use --shard to ensure that at most one op is sent to a node at a time.

My command looks something like this:
parallel --shard 1 send_op.sh --node={1} --op={2} ::: input.csv

Unfortunately, this doesn't work. parallel tells me "--shard requires --jobs to be higher than the number of arguments. Increase --jobs."

This doesn't make sense to me. Doesn't it defeat the point of --shard if --jobs is set to be higher than the number of input lines (the number of ops in my example)? How can I achieve my desired concurrency property?

gnu-parallel parallel-processing
1 Answers

I'd like to use --shard to ensure that at most one op is sent to a node at a time.

This is not what --shard does. --shard is a specialized version of --pipe, and you are not using pipe-mode.

So what can you do instead?

sem makes sure only a single jobs with that id runs in parallel. So you might be able to:

parallel --colsep , sem --id {1} send_op.sh --node={1} --op={2} :::: input.csv

This solution has some issues:

  • Order matters - a lot. If all node-A are grouped together, these will take up jobslots in parallel and thus slow down the processing. But if you add --shuf to parallel the order will be shuffled and thus this would be less of an issue.
  • sem takes 150 ms to start. So if your jobs are short lived (< 1 s total wall clock time) you may see this as slowing down your processing.
user_3630280
Related