Filter JSON of nested array by regex and sum using jq

user_5586435 0
Viewed 493

I have a JSON of nested array that I would like to filter and then sum the size field values for all partitions of a particular topic/prefix. So, for topicA, I want to add the 70, 45 and 100 for topicA-0, topicA-1, topicA-2 sizes to get 215.

{
  "version": 1,
  "brokers": [
    {
      "broker": 1,
      "logDirs": [
        {
          "logDir": "/var/lib/kafka/data",
          "error": null,
          "partitions": [
            {
              "partition": "topicA-0",
              "size": 70,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicB-0",
              "size": 40,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicC-0",
              "size": 20,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicD-0",
              "size": 30,
              "offsetLag": 0,
              "isFuture": false
            }
          ]
        }
      ]
    },
    {
      "broker": 2,
      "logDirs": [
        {
          "logDir": "/var/lib/kafka/data",
          "error": null,
          "partitions": [
            {
              "partition": "topicA-1",
              "size": 45,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicB-1",
              "size": 50,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicC-1",
              "size": 10,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicD-1",
              "size": 20,
              "offsetLag": 0,
              "isFuture": false
            }
          ]
        }
      ]
    },
    {
      "broker": 3,
      "logDirs": [
        {
          "logDir": "/var/lib/kafka/data",
          "error": null,
          "partitions": [
            {
              "partition": "topicA-2",
              "size": 100,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicB-2",
              "size": 60,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicC-2",
              "size": 80,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "topicD-2",
              "size": 5,
              "offsetLag": 0,
              "isFuture": false
            }
          ]
        }
      ]
    }
  ]
}

How can I achieve this in jq? I tried jq -c 'select( any(.brokers[].logDirs[].partitions[].partition; test("topicA-")) | .size? | numbers ] | add)' but didn't work. Thanks in advance

jq bash json
2 Answers

Select partitions whose name start with topicA-, populate an array with their sizes and feed it to add.

[.brokers[].logDirs[].partitions[] | select(.partition|startswith("topicA-")) .size] | add

demo at jqplay.org

But if you want to calculate the sum for all topics without specifying a name, and produce a report in JSON format, you can do something like this:

reduce .brokers[].logDirs[].partitions[] as $p ({};
  .[($p.partition/"-")[0]] += $p.size
)

demo at jqplay.org

user_102486780

OK, here's a pure jq solution:

jq  '(
         [
             .brokers[].logDirs[].partitions[]
             | .partition[0:(.partition | index("-"))]
         ]
         | unique | .[]
     ) as $topic
     | [.brokers[].logDirs[].partitions[]
     | select(.partition | startswith($topic))
     | .size] | $topic, add' < file.json

I'm not sure it's readable, I'd probably stop when jq outputs topics and sizes and do the maths in Perl:

jq -r '.brokers[].logDirs[].partitions[]
      | .partition as $p
      | $p[0:($p | index("-"))], .size' < file.json |
perl -lnwE '$h{$_} += <> ; END {say "$_: $h{$_}" for keys %h}'
user_10306750
Related