Best way to iterate over JSON array with variable keys with jq/bash

user_1652219 0
Viewed 298

I have a JSON-file like the one below:

{
    "Results": [
        {
            "Severity": "LOW",
            "InstalledVersion": "1.21.1ubuntu2",
            "FixedVersion": "1.21.1ubuntu2.1"
        },
        {
            "Severity": "LOW",
            "InstalledVersion": "1.21.1ubuntu2"
        },
        {
            "Severity": "HIGH",
            "VulnerabilityID": "CVE-2016-2781"
        }
    ]
}

I would like to create the following JSON-file:

[
    {
        "Severity": "LOW",
        "InstalledVersion": "1.21.1ubuntu2",
        "FixedVersion": "1.21.1ubuntu2.1",
        "VulnerabilityID": ""
    },
    {
        "Severity": "LOW",
        "InstalledVersion": "1.21.1ubuntu2",
        "FixedVersion": "",
        "VulnerabilityID": ""
    },
    {
        "Severity": "HIGH",
        "InstalledVersion": "",
        "FixedVersion": "",
        "VulnerabilityID": "CVE-2016-2781"
    }
]

I have tried iterating over the first JSON-file, but when I do so in the following way, I get random parsing errors (with the real JSON-file I am working with). So I think there is something off about the piping that I am doing.

jq -c '.Results[]' my.json | while read i; do
    echo $i;
done

In the JSON-file I am working with jq -c '.Results[]' works perfectly, but when I do jq operations on the resulting $i while iterating, I suddently get several parsing errors.

Is there better way to iterate over key-value-pairs in a JSON-file? Ofcause I could just install Python/R and do the typical drill, but I would really like for this to work in a shell script.

jq arrays bash json
4 Answers

You can use + to add two objects together by combining their fields; if both objects have the same one the value of the right hand one is used. So if you make your existing object the RHS:

 jq '[ .Results[] | {Severity:"", InstalledVersion:"", FixedVersion:"", VulnerabilityID:""} + . ]' input.json   
[
  {
    "Severity": "LOW",
    "InstalledVersion": "1.21.1ubuntu2",
    "FixedVersion": "1.21.1ubuntu2.1",
    "VulnerabilityID": ""
  },
  {
    "Severity": "LOW",
    "InstalledVersion": "1.21.1ubuntu2",
    "FixedVersion": "",
    "VulnerabilityID": ""
  },
  {
    "Severity": "HIGH",
    "InstalledVersion": "",
    "FixedVersion": "",
    "VulnerabilityID": "CVE-2016-2781"
  }
]
user_99521960

One more variation

jq '[
    .Results[]
    | .FixedVersion //= ""
    | .VulnerabilityID //= ""
    | .InstalledVersion //= ""
]' file.json
user_75520

Another way of adding the fallbacks to a empty string:

[ .Results[] | . + { VulnerabilityID: (.VulnerabilityID // ""), FixedVersion: (.FixedVersion // ""), InstalledVersion: (.InstalledVersion // "") } ]

[
  {
    "Severity": "LOW",
    "InstalledVersion": "1.21.1ubuntu2",
    "FixedVersion": "1.21.1ubuntu2.1",
    "VulnerabilityID": ""
  },
  {
    "Severity": "LOW",
    "InstalledVersion": "1.21.1ubuntu2",
    "VulnerabilityID": "",
    "FixedVersion": ""
  },
  {
    "Severity": "HIGH",
    "VulnerabilityID": "CVE-2016-2781",
    "FixedVersion": "",
    "InstalledVersion": ""
  }
]

Online demo

user_56255470

A more generic approach :

jq '.Results |
    (map(keys)|add|unique|map({key:.,value:""})|from_entries) as $default |
    map($default + .)
   ' input.json

Explanations :

1) map(keys)|add|unique
   gets all the keys. If it's a subset you are interested, you can substitute it with :
   ["FixedVersion", "VulnerabilityID", "InstalledVersion"]
2) map({key:.,value:""})|from_entries) as $default
   constructs a default object with all (or selected) keys.
3) $default + .
   adds a key from default object if the key does not exist.
user_21256710
Related