Read line by line and insert data ( 16 GB JSON) too slow ( MySQL + JAVA spring boot )

Viewed 98

I have B2B2C business and I have few suppliers who keep update my data. Every week I need to update around 30GB JSON ( one file ). I would like to know about faster way than readline of BufferedReader the file type is ZStandard(zstd). For now its inserting 500Mb every 4 hours. is that make any sense that is so slow ?? the web app and database are deploying on unix server ( Tomcat apache 9.06)

my code :

    try {
        BufferedReader br;
        br = new BufferedReader(new FileReader("/opt/tomcat/" + fileName));

        String line = br.readLine();

        ObjectMapper om = new ObjectMapper();

        while (line != null) {
            

            if (!rootRepository.existsByAddress(om.readValue(line, Data.class).getAddress())) {
                rootRepository.save(om.readValue(line, Data.class));
            }

            line = br.readLine();
        }

        br.close();

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
    return "Completed";


}`
0 Answers
Related