I have built a script in Laravel that reads a JSON file line by line and imports the contents into my database.
However, when running the script, I get an out of memory error after inserting about 80K records.
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
PHP Fatal error: Out of memory (allocated 421527552) (tried to allocate 12288 bytes) in /home/vagrant/Code/sandbox/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php on line 1758
mmap() failed: [12] Cannot allocate memory
PHP Fatal error: Out of memory (allocated 421527552) (tried to allocate 32768 bytes) in /home/vagrant/Code/sandbox/vendor/symfony/debug/Exception/FatalErrorException.php on line 1
I have built a sort of makeshift queue to only commit the collected items every 100, but this made no difference.
This is what the part of my code that does the inserts looks like:
public function callback($json) {
if($json) {
$this->queue[] = [
'type' => serialize($json['type']),
'properties' => serialize($json['properties']),
'geometry' => serialize($json['geometry'])
];
if ( count($this->queue) == $this->queueLength ) {
DB::table('features')->insert( $this->queue );
$this->queue = [];
}
}
}
It's the actual inserts (DB::table('features')->insert( $this->queue );) that are causing the error, if I leave those out I can perfectly iterate over all lines and echo them out without any performance issues.
I guess I could allocate more memory, but doubt this would be a solution because I'm trying to insert 3 million records and it's currently already failing after 80K with 512Mb memory allocated. Furthermore, I actually want to run this script on a low budget server.
The time it takes for this script to run is not of any concern, so if I could somehow slow the insertion of records down that would be a solution I could settle for.