I have json file like below & the file size is more than 50GB and this file have 300M+ records in "in_network" collection.
I want to iterate this file without storing full data in memory and want to ready single JSON item in each iteration from "in_network" collection.
How can i pull data from in network collection without having memory failure?
SAMPLE JSON
{
"id":"2234-2342-2342342",
"name":"sample-network-file",
"in_network":[
{
"a": "asd",
"b": "asd",
"c": [100,100,100,100,100,100,100],
"d":{
"aa": "asd",
"bb": {
"xy": "asd",
"npx":[1,2,3,4,5]
}
}
},
{
"a": "asd",
"b": "asd",
"c": [100, 100, 100, 100, 100, 100, 100],
"d": {
"aa": "asd",
"bb": {
"xy": "asd",
"npx": [1, 2, 3, 4, 5]
}
}
}
]
}
I Am Trying Like This
public static void Read() throws IOException {
String filePath="data.json";
InputStream inputStream = Files.newInputStream(Path.of(filePath));
JsonReader reader = new JsonReader(new InputStreamReader(inputStream));
reader.beginObject();
while (reader.hasNext()) {
String name = reader.nextName();
if (name.equals("in_network")) {
reader.beginArray();
while (reader.hasNext()) {
InNetwork inNetworkItem = new Gson().fromJson(reader, InNetwork.class);
//SendDataOnKafkaTopic(inNetworkItem)
System.out.println(inNetworkItem);
}
reader.endArray();
}else {
reader.skipValue();
}
}
reader.endObject();
reader.close();
}