Sending concatenated files to JQ with wrapped as JSON array

Viewed 32

I have a curious problem sending concatenated files to JQ. I'm using Windows 10, but the problem is surfacing with Bash (via Git Bash) as well.

I have several input JSON files in the form input-1.json, input-2.json, input-3.json, etc. The contents of each is a nested object that looks like this:

{
  "blah": "blah",
  "foo": [{
      "bar" {…}
    }, {
      "bar" {…}
    }]
}

I want to concatenate the files and extract all the values of "bar" across all the files into one array. (Note that "foo" is an array of objects, each with "bar" containing its own object.) In other words, I want to end up with (placed on one line for readability here):

[{…}, {…}, {…}, {…}, {…}]

Note that these are the foo…bars across all the files!

I start with this:

type input-*.json | jq ".foo[].bar" > output.json

That gives me

{…}, {…}, {…}, {…}, {…}

Close! Now I just have to wrap them in […], right? So I do this:

type input-*.json | jq "[.foo[].bar]" > output.json

Uh, oh; it gives me the following:

[{…}, {…}, {…}], [{…}, {…}]

But why? At the point of .foo[].bar, JQ just sees a stream of objects, right? How does JQ "remember" that some of those objects came from different inputs?

Note that How to convert a JSON object stream into an array with jq and jq: output array of json objects seem to be similar questions. They say to use JQ's --slurp mode. But won't that prevent streaming, i.e. if the output of all the files is really huge, won't it this have the potential of running out of memory by loading the entire input into memory?

Besides (and this is the crux of my confusion), if jq ".foo[].bar" gives me exactly the {…}, {…}, {…}], [{…}, {…} array contents I want, how does JQ "remember" that some of those objects came from different inputs? You'll see that JQ isn't wrapping each {…} in an array, but actually wrapping several objects based upon which input-*.json file it came from. Why?

1 Answers

One way would be to "slurp" the files, like so:

jq -s 'map( .foo[].bar )' input-*.json

If you want to save memory, you could go with:

jq -n '[inputs | .foo[].bar]' input-*.json

If your shell does not support filename globbing, then you would have to specify the file names in some other way, e.g. as in the Q.

Related