My goal is to index an entire food product database from https://world.openfoodfacts.org/data . I have downloaded entire database in mongodb BSON format, I have loaded it to mongodb compass and exported only essential fields:
- id
- product_name
- nutrients
I have been left with a json file that is about 3GB in size.
The reason I want to read the file is to transform and change naming conventions of nutrients object that fits my database style.
I have tried to do the following:
fs.readFile(
"/Users/chx/Downloads/dump/off/products.json",
"utf8",
(err, data) => {
if (err) {
console.error(err);
return;
}
console.log(data[0]);
}
);
I get the following error:
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2805202321) is greater than 2 GiB
How can I read and transform my data array?