Read Huge JSON File & Send Data on Kafka Topic Using SpringBoot App

Viewed 53

I have json file like below & the file size is more than 50GB and this file have 300M+ records in "in_network" collection.

I want to iterate this file without storing full data in memory and want to ready single JSON item in each iteration from "in_network" collection.

How can i pull data from in network collection without having memory failure?

SAMPLE JSON

{
    "id":"2234-2342-2342342",
    "name":"sample-network-file",
    "in_network":[
        {
          "a": "asd",
          "b": "asd",
           "c": [100,100,100,100,100,100,100],
           "d":{
            "aa": "asd",
            "bb": {
                "xy": "asd",
                "npx":[1,2,3,4,5]
            }
           }
        },
         {
           "a": "asd",
           "b": "asd",
           "c": [100, 100, 100, 100, 100, 100, 100],
           "d": {
             "aa": "asd",
             "bb": {
               "xy": "asd",
               "npx": [1, 2, 3, 4, 5]
             }
           }
         }
    ]
}

I Am Trying Like This

public  static void Read() throws IOException {
        String filePath="data.json";

        InputStream inputStream = Files.newInputStream(Path.of(filePath));
        JsonReader reader = new JsonReader(new InputStreamReader(inputStream));

        reader.beginObject();
        while (reader.hasNext()) {
            String name = reader.nextName();
            if (name.equals("in_network")) {
                reader.beginArray();
                while (reader.hasNext()) {
                    InNetwork inNetworkItem = new Gson().fromJson(reader, InNetwork.class);
                    //SendDataOnKafkaTopic(inNetworkItem)
                    System.out.println(inNetworkItem);
                }
                reader.endArray();
            }else {
                reader.skipValue();
            }
        }
        reader.endObject();
        reader.close();
    }
0 Answers
Related