How to read json files that are larger than 2gb with node js?

Viewed 19

My goal is to index an entire food product database from https://world.openfoodfacts.org/data . I have downloaded entire database in mongodb BSON format, I have loaded it to mongodb compass and exported only essential fields:

  • id
  • product_name
  • nutrients

I have been left with a json file that is about 3GB in size.

The reason I want to read the file is to transform and change naming conventions of nutrients object that fits my database style.

I have tried to do the following:

fs.readFile(
  "/Users/chx/Downloads/dump/off/products.json",
  "utf8",
  (err, data) => {
    if (err) {
      console.error(err);
      return;
    }
    console.log(data[0]);
  }
);

I get the following error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2805202321) is greater than 2 GiB

How can I read and transform my data array?

0 Answers
Related