jq how to merge two JSON streams into a single JSON stream

user_2002387 0
Viewed 77

How to merge two json streams/lists into a single stream of jsons like the following.

file 1:

{"key1": 1}
{"key1": 2}
{"key1": 3}

file 2:

{"key2": -1}
{"key2": -2}
{"key2": -3}

Expected output:

{"key1": 1, "key2": -1}
{"key1": 2, "key2": -2}
{"key1": 3, "key2": -3}
jq merge stream json
2 Answers

You can transpose and add:

$ jq -c -n --slurpfile a file1.json --slurpfile b file2.json '[$a, $b] | transpose[] | add'
{"key1":1,"key2":-1}
{"key1":2,"key2":-2}
{"key1":3,"key2":-3}
user_193551810

Here is a "streaming" solution that is more efficient (both memory-wise and otherwise) than one that requires that both files be read in their entirety before producing any output:

< file1.json jq -nc --slurpfile file2 file2.json '
  # For each item $s in the stream s,
  # emit [$s, $t] where $t is the corresponding item in the input array
  def zip(s):
    . as $in
    | foreach s as $s (-1; .+1; [$s, $in[.]]);

  $file2 | zip(inputs) | add
'

If your jq does not support the --slurpfile command-line option, then use --argfile instead.

As a one-liner

foreach inputs as $s (-1; .+1; $s + $file2[.])
user_9973580
Related