Aggregate json arrays from multiple files using jq, grouping by key

user_3182644 0
Viewed 470

I would like to aggregate two or more files into a single json, and aggregate arrays under a same key.

file1.json

{
  "shapes": [
    {
      "id": "1",
      "name": "circle"
    },
    {
      "id": "2",
      "name": "square"
    }
  ]
}

file2.json

{
  "shapes": [
    {
      "id": "3",
      "name": "triangle"
    }
  ]
}

Expected result :

{
  "shapes": [
    {
      "id": "1",
      "name": "circle"
    },
    {
      "id": "2",
      "name": "square"
    },
    {
      "id": "3",
      "name": "triangle"
    }
  ]
}

I can do this with the following jq command :

jq -s '{shapes: map(.shapes)|add }' file*.json

But this requires me to know the shapes attribute and hardcode it. Is there a simple way I can get the same result without ever using the key name explicitly?

jq json
3 Answers

Here is a solution that also solves a more general problem: first, it handles arbitrarily many input files; and second, it forms the "sum" by key, for every key, on the assumption that every top-level key is array-valued.

The generic function:

  # the values at each key are assumed to be arrays
  def aggregate(stream): 
    reduce stream as $o ({}; 
      reduce ($o|keys_unsorted[]) as $k (.; 
        .[$k] += $o[$k] ));

To avoid "slurping", we will use inputs:

aggregate(inputs)

The invocation must therefore use the -n command-line option:

jq -n -f program.jq *.json
user_9973580

Here is a solution that’s suitable when each top-level object has only one key, and that is both efficient and conceptually simple. It assumes jq is invoked with the -n option.

reduce inputs as $in (null;
   ($in|keys_unsorted[0]) as $k | { ($k): (.[$k] + $in[$k]) })

or slightly more compactly:

reduce inputs as $in (null; ($in|keys_unsorted[0]) as $k | .[$k] += $in[$k] )
user_9973580

Try the following code. This can handle any number of files. All inputs are assumed to be json objects with all values inside as arrays. All such arrays are aggregated after grouping by keys. It outputs an object which has keys associated with corresponding aggregated arrays.

jq -s 'map(to_entries)|add|group_by(.key)|
    map( { "key": (.[0].key), "value": (map(.value)|add)})|
    from_entries' file1.json file2.json

For your sample input this gives:

{
  "shapes": [
    {
      "id": "1",
      "name": "circle"
    },
    {
      "id": "2",
      "name": "square"
    },
    {
      "id": "3",
      "name": "triangle"
    }
  ]
}
user_19215460
Related