Javascript Read large files failed

user_7710522 0
Viewed 4911

A JSON file is 6 GB. When reading it with the following code,

var fs = require('fs');
var contents = fs.readFileSync('large_file.txt').toString();

It had the following error:

buffer.js:182
    throw err;
    ^

RangeError: "size" argument must not be larger than 2147483647
    at Function.Buffer.allocUnsafe (buffer.js:209:3)
    at tryCreateBuffer (fs.js:530:21)
    at Object.fs.readFileSync (fs.js:569:14)
    at Object.<anonymous> (/home/readHugeFile.js:4:19)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)

Could somebody help, please?

io file javascript
2 Answers

u can read the file using line reader node js package and at every 50000 lines you can make small files squenceally then process those file and clear them out for your purpose if you have some task to read data from each line for a bigger file.line reader can do the job as it use stream in backend. the line reader dont wait for you if you directly read and process data like update in mongodb etc. i did it and it worked even for a 10gb file .

user_165374850
Related