Parse enormous json file, output objects to their own files

sword_of_omens 0
Viewed 49

I have json files of arbitrary size, some small, some enormous (>40GB). I'm trying to use jq to stream objects from an object_array to their own files for later processing.

object_array has N objects.

The json has the following structure:

{
    "top_level_name": "Some Name Here",
    "top_level_type": "Some Type",
    "last_updated_on": "2022-07-09",
    "version": "1.0.0",
    "object_array": [
        {
            "arrangement": "abcd",
            "name": "another name",
            "type": "efgh",
            "type_ver": "2021",
            "code": "12345",
            "desc": "some description",
            "rate": [
                {
                    "groups": [
                        {
                            "IDs": [
                                "123654890",
                                "012365485"
                            ],
                            "id_type": {
                                "type": "xyz",
                                "value": "8527419630"
                            }
                        }
                    ],
                    "prices": [
                        {
                            "price_type": "priceType",
                            "rate": "00.00",
                            "date": "2023-01-01",
                            "svc_code": [
                                "89"
                            ],
                            "class": "some-class",
                            "modifier": [
                                "78"
                            ],
                            "additional_information": "null"
                        }
                    ]
                }
            ]
        }
    ]
}

The current path I'm on has me trying jq -cn --stream 'fromstream(1|truncate_stream(inputs))' test.json| awk '{print > "onif00" NR ".json"}' but my results vary from a single file with all the data or a ton of files each with some piece of the data e.g. [ followed by another file with { etc.

Specifically, I'd like to capture each object like the one below, from object_array, and place it in its own file

{
    "arrangement": "abcd",
    "name": "another name",
    "type": "efgh",
    "type_ver": "2021",
    "code": "12345",
    "desc": "some description",
    "rate": [{
        "groups": [{
            "IDs": [
                "123654890",
                "012365485"
            ],
            "id_type": {
                "type": "xyz",
                "value": "8527419630"
            }
        }],
        "prices": [{
            "price_type": "priceType",
            "rate": "00.00",
            "date": "2023-01-01",
            "svc_code": [
                "89"
            ],
            "class": "some-class",
            "modifier": [
                "78"
            ],
            "additional_information": "null"
        }]
    }]
}
jq json
2 Answers

As in the Q, I'd avoid calling jq more than once. I also like using awk, so if for example you want .name as part of the file name, I'd go with s.t. like:

< my.json jq -cnr --stream '
    fromstream(2|truncate_stream(inputs | select(.[0][0] == "object_array")) )
    | .name, .
' | awk 'fn=="" {fn=$1; next} {print > "onif00_" fn ".json"; fn=""; }'

This has also been tested using gojq, the Go implementation of jq.

See @Jeff_Mercado's comment elsewhere on this page re adapting this for versions of jq which support --stream but for which fromstream is buggy.

peak0

Note: Apart from jq, this is a pure bash solution. For an awk solution, see @peak's answer.

In jq, select from the stream those parts that belong to the object_array, then truncate by 2 (field name and array index). In the shell, read the lines provided by jq -c and redirect them to a file.

< enormous.json jq -cn --stream '
  fromstream(2|truncate_stream(inputs| select(.[0][0] == "object_array")))
' | while read -r json;
    do fname="onif00$((++NR))"; > "$fname.json" cat <<< "$json";
    done

To also extract data to be used for naming the files, interleavingly output them as well. To allow for any string, including special characters such as newlines (although you shouldn't use newlines in file names), change the line delimiter to the NUL byte (using -j and "\u0000" in jq, and -d '' with read). For instance, taking the .name value from the sample input and appending .json results in a file called another name.json.

< enormous.json jq -jn --stream '
  fromstream(2|truncate_stream(inputs| select(.[0][0] == "object_array")))
  | (.name, @json) + "\u0000"
' | while read -d '' fname; read -d '' json;
    do > "$fname.json" cat <<< "$json";
    done

Note: Apparently, for jq versions prior to 1.6 there was a bug in truncate_stream/1 that prevented it from taking inputs other than 1. One remedy would be to redefine truncate_stream yourself by prepending your filter with

def truncate_stream(stream):
  . as $n | null | stream | . as $input
  | if (.[0]|length) > $n then setpath([0];$input[0][$n:]) else empty end;

This was taken directly from the 1.6 source code, and it works perfectly with jq 1.5, for example.

pmf0
Related