How to count the number of records of a JSON object flow file in NiFi?

user_7098625 0
Viewed 2524

I am calling an API which returns a JSON object as the body of the response as a flow file. I want to count the total records in the flow file.

I tried record processors like CalculateRecordStats needs a record reader and that in-turn needs a schema. But I don't have a schema. So Is there a better way to do it?

{
"result":[
    {
        "key1": "value1",
        "key2": "value2"
    },
    {
        "key1": "value1",
        "key2": "value2"
    }
]
}

This is the sample JSON the api response body returns. The records are the items of the list which is the value of the "result" key

In the above sample there are 2 records (2 items in the list). So that is what I want to calculate.

apache-nifi
2 Answers

Thank you Andy for your response. I found another answer from Hortonworks Community Discussion which I felt was very simplistic.

It was to use a "EvaluateJSONPath" processor and set a new flowfile-attribute 

resultCount = $.result.length()

This attribute ${resultCount} could be later validated for correctness.

user_70986250

There is a Jolt specification for calculating the size of an array but I'm not sure if that operation is available in the version of the Jolt library used by NiFi currently. 

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "keyLength": "=size(@(1,key))"
    }
  }
]

Using the record readers, you should be able to create a JsonTreeReader or JsonPathReader and set the Schema Access Strategy to infer the schema in realtime for this purpose.

user_704650
Related