Laravel : (1 m) Bulk data upload to the mysql with optimization

Viewed 30

i am using laravel and i need to upload bulk (~ 1 million) of data (in .xlsx file) to the mysql using laravel....

currently i am uploading data near 200k and it is uploading to the mysql with phpoffice-phpspreadsheet package....

Here is my code to upload the data....

if($extension == 'xlsx')
{
    info('Entered');
    try{
        $spreadsheet = IOFactory::load($tempPath);                    
        $sheet        = $spreadsheet->getActiveSheet();
        $row_limit    = $sheet->getHighestDataRow();
        $column_limit = $sheet->getHighestDataColumn();
        $row_range    = range( 2, $row_limit );
        $column_range = range( 'AH', $column_limit );
        $startcount = 2;
        $data = array();
        $counter = 0;
        foreach ( $row_range as $row ) {
            $counter++;
            if($counter%5 == 0)
            {
                info('Next 5');
            }
            $data = [
                'id' =>$sheet->getCell( 'A' . $row )->getValue(),
                'ACNo' => $sheet->getCell( 'B' . $row )->getValue(),
                'PartNo' => $sheet->getCell( 'C' . $row )->getValue(),
                'SectionNo' => $sheet->getCell( 'D' . $row )->getValue(),
                'SlNo' => $sheet->getCell( 'E' . $row )->getValue(),
                'HouseNo' =>$sheet->getCell( 'F' . $row )->getValue(),
                'VHouseNo' =>$sheet->getCell( 'G' . $row )->getValue(),
                'EName' =>$sheet->getCell( 'H' . $row )->getValue(),
                'VEName' =>$sheet->getCell( 'I' . $row )->getValue(),
                'sex' =>$sheet->getCell( 'J' . $row )->getValue(),
                'RName' =>$sheet->getCell( 'K' . $row )->getValue(),
                'VRName' =>$sheet->getCell( 'L' . $row )->getValue(),
                'RType' =>$sheet->getCell( 'M' . $row )->getValue(),
                'Age' =>$sheet->getCell( 'N' . $row )->getValue(),
                'IDCardNo' =>$sheet->getCell( 'O' . $row )->getValue(),
                'statustype' =>$sheet->getCell( 'P' . $row )->getValue(),
                'Addition' =>$sheet->getCell( 'Q' . $row )->getValue(),
                'ContactNo' =>$sheet->getCell( 'R' . $row )->getValue(),
                'Address' =>$sheet->getCell( 'S' . $row )->getValue(),
                'VAddress' =>$sheet->getCell( 'T' . $row )->getValue(),
                'PSName' =>$sheet->getCell( 'U' . $row )->getValue(),
                'VPSName' =>$sheet->getCell( 'V' . $row )->getValue(),
                'whatsappNumber' =>$sheet->getCell( 'W' . $row )->getValue(),
                'caste' =>$sheet->getCell( 'X' . $row )->getValue(),
                'casteCategory' =>$sheet->getCell( 'Y' . $row )->getValue(),
                'education' =>$sheet->getCell( 'Z' . $row )->getValue(),
                'currentCity' =>$sheet->getCell( 'AA' . $row )->getValue(),
                'currentDistrict' =>$sheet->getCell( 'AB' . $row )->getValue(),
                'currentState' =>$sheet->getCell( 'AC' . $row )->getValue(),
                'livelihood' =>$sheet->getCell( 'AD' . $row )->getValue(),
                'voted' =>$sheet->getCell( 'AE' . $row )->getValue(),
                'support' =>$sheet->getCell( 'AF' . $row )->getValue(),
                'created_at' =>$sheet->getCell( 'AG' . $row )->getValue(),
                'updated_at' =>$sheet->getCell( 'AH' . $row )->getValue(),

            ];
            $startcount++;
            Voter::create($data);
        }                   
    } 
    catch (Exception $e) 
    {
        // $error_code = $e->errorInfo[0];
        // 
        // echo $error_code;
        echo $e;
        exit();
    }
}

Here

$extension is the extension of file which should be .xlsx

and i need to optimize the flow.... so i can upload nearly 1 million of data using this method

1 Answers
Related