Output paths to all keys named "id" where the type of value is "string"

user_4656035 0
Viewed 103

Given a huge (15GB) deeply nested (12+ object layers) JSON file how can I find the paths to all the keys named id whose values are type string?

A massively simplified example file:

{
  "a": [
    {
      "id": 3,
      "foo": "red"
    }
  ],
  "b": [
    {
      "id": "7",
      "bar": "orange",
      "baz": {
        "id": 13
      },
      "bax": {
        "id": "12"
      }
    }
  ]
}

Looking for a less ugly solution where I don't run out of RAM and have to punt to grep at the end (sigh). (I failed to figure out how to chain to_entries into this usefully. If that's even something I should be trying to do.)

Ugly solution 1:

$ cat huge.json | jq 'path(..|select(type=="string")) | join(".")' | grep -E '\.id"$'
"b.0.id"
"b.0.bax.id"

Ugly solution 2:

$ cat huge.json | jq --stream -c | grep -E '"id"],"'
[["b",0,"id"],"7"]
[["b",0,"bax","id"],"12"]
jq select path json
3 Answers

Something like this should do that.

jq --stream 'select(.[0][-1] == "id" and (.[1] | strings)) | .[0]' file

And by the way, your first ugly solution can be simplified to this:

jq 'path(.. .id? | strings)' file
user_102486780

Stream the input in as you started with your second solution, but add some filtering. You do not want want to read the entire contents into memory. And also... UUOC.

$ jq --stream '
select(.[0][-1] == "id" and (.[1]|type) == "string")[0]
  | join(".")
' huge.json
user_3902780

Thank you both oguz and Jeff! Beautiful! This runs in 6.5 minutes (on my old laptop), never uses more than 21MB of RAM, and gives me exactly what I need. <3

$ jq --stream -c 'select(.[0][-1] == "id" and (.[1]|type) == "string")' huge.json
user_46560350
Related