How to keep only one header row in output processing test using python and gnu-parallel?

user_892534 0
Viewed 201

The Problem

Often I need to process a directory of several CSV files and produce a single output file. Frequently, I rely on GNU parallel to run these tasks concurrently. However, I need a way to discard the first row (header) for all but the first job that returns output.

To make this concrete, imagine a directory of several CSV files like this...

x,y
1,1.2
2,5.3
3,6.0

Then, there's some (Python) script, call it calc.py, that cleans the data or does calculations...

import csv
import math
import sys

rdr = csv.DictReader(sys.stdin)
wtr = csv.DictWriter(sys.stdout, fieldnames=['x', 'y', 'siny'])
wtr.writeheader()

for row in rdr:
    row['siny'] = math.sin(float(row['y']))
    wtr.writerow(row)

We can then process the data files in parallel with GNU parallel...

parallel --lb python calc.py '<' {} ::: $(ls -1 *.csv)

This, however, will produce multiple header rows. For example...

x,y,siny
1,1.2,0.9320390859672263
2,5.3,-0.8322674422239013
3,6.0,-0.27941549819892586
x,y,siny
4,7.2,0.7936678638491531
5,2.2,0.8084964038195901
6,0.9,0.7833269096274833

I am looking for a simple way (ideally an option to parallel) to only keep the first header line in the output. The stuff related to headers in the manual seems to be about inputs.

Possible Solutions

I see a few options, but I don't love any of them...

  1. Don't have calc.py output the header and instead echo a header before running parallel. The disadvantage is that the header must be known or we need to peek at the header by running something like python calc.py data1.csv | head -n 1 before running parallel.
  2. Save the output of each job to a separate file, then concatenate them ex post (e.g. with xsv, tail, sed, etc.), removing the header from all but the first. This has the disadvantage of having to manage additional files on disk and clean them up afterwards.
  3. Write another program that does this and pipe the results of parallel to that.
    • Seems CPU intensive to compare each line of output against the first line, and we know few records will match.
    • Assumes no valid data records match the header row.

What's the best way to solve this? Is there an option that tells parallel to ignore all but one header row from each job's output?

gnu-parallel
2 Answers

How about adjusting option 1:

Make the program take two arguments: file jobnumber

if jobnumber == 1:
  output header

To guarantee that job 1 is printed first, use --keep-order:

parallel --keep-order python calc.py {#} '<' {} ::: *.csv

GNU Parallel will cache the output from the running jobs in /tmp to serialize the output, which may or may not be slower than --lb.

In general you can do something like:

parallel -k python 'calc.py < {} {= uq; $_ = seq()==1 ? "" : "| tail +2" =}' ::: *.csv

uq is available since 20190722.

You will be running a tail, so it can be slightly slower. On my system tail delivers 0.5 GB/s per core.

user_3630280

AFAIK, GNU Parallel do not provide any such option to write header only once in output

But a simple approach can be to use awk to output header row only once like

parallel --lb python calc.py '<' {} ::: $(ls -1 *.csv) | awk 'NR==1 { header=$0; print $0; next; } $0 != header'

Explanation

NR is line number in awk. When line number is 1 it will store it in variable header and only print once. If any row exactly matched with header afterwards, it will be ignored

Pros

  • You do not need to know the header beforehand
  • You do not need to save output into separate files
  • No need to change your code

Cons

  • Assumes no data row exactly matches header
  • In case there is no header present it will trim first line and all sames lines from output
  • There is slight overhead of string matching
user_44420940
Related