How to Upload a CSV File in Background Process with PHP (Laravel 5.8)

Viewed 4082

I want to build a functionality in Laravel 5.8 that can help me to upload a CSV file and import the data into database, but it all should be in background(server side) process and once it is completed send an email to the logged in user. I want to get a better understanding before I start the process should I do this with a help of Scheduler or there would be a better way or library that can help me to achieve this functionality.

Looking forward to hear your thoughts :)

Thanks.

3 Answers

Starting from scratch:

Create a invoices table and the corresponding model:

php artisan make:model Invoice -m

Your model should look like this:

<?php

namespace App;

use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Model;

class Invoice extends Model
{
    protected $table = 'invoices';
    protected $fillable = [
        'user_id',
        'processed',
        'path'
    ];

    public function scopeNotProcessed(Builder $query)
    {
        return $this->where('processed', '=', false);
    }
}

And here is the invoices table:

public function up()
{
    Schema::create('invoices', function (Blueprint $table) {
        $table->increments('id');
        $table->string('path')->nullable(false);
        $table->boolean('processed')->default(false)->nullable(false);
        $table->timestamps();
    });
}

After get those things done, proceed as follows:

Create a repository, what will upload your csv file. This file should be placed at app/Repositories/CSVRepository:

<?php

namespace App\Repositories;

use Illuminate\Support\Facades\Storage;
use App\Invoice;

class CSVRepository {

    /**
     * CSVRepository constructor.
     */
    public function __construct()
    {
        //
    }

    /**
     * @param $file
     * @param $extension
     * @return mixed
     */
    public function uploadCSV($file, $extension){
        return $this->upload($file, $extension);
    }

    /**
     * @param $file
     * @param $extension
     * @return mixed 
     */
    private function upload($file, $extension){
        $path = Storage::putFileAs("myFileName", $file, uniqid().".".$extension);
        $uploadedFile = Invoice::create([
            'path' => $path,
            'processed' => false,
        ]);

        return $uploadedFile;
    }
}

Now, create your controller, that will upload the file to the server using the CSVRepository: The upload function should looks like this one:

public function upload(CSVRepository $CSVRepository)
{
    try{
        $file = Input::file('file');
        $extension = strtolower($file->getClientOriginalExtension());
        if ($extension !== 'csv'){
            $errors['file'] = 'This is not a .csv file!';
            return redirect()->back()->withInput()->withErrors($errors);
        }
        $CSVRepository->uploadCSV($file, $extension); 
        $message = array(
            'type' => 'success',
            'text' => 'Your file has been uploaded! You will receive an email when processing is complete!',
            'title' => 'Success',
        );
        session()->flash('message', $message);
        return redirect('route-to-redirect');
    }catch (\Exception $exception){
        return abort(Response::HTTP_INTERNAL_SERVER_ERROR, 'Internal Server Error');
    }
}

Now, you need a job, which process the file to you:

Start creating a command with the artisan command:

php artisan make:command ProcessCSVCommand

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use App\Invoice;

use Illuminate\Support\Facades\Storage;

class ProcessCSVCommand extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'csv:process';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Process an uploaded CSV file';

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        try{
            //Retrieve only no processed files:
            $invoices = Invoice::notProcessed()->get();
            if (count($invoices) < 1){
                $this->info('No files found');
                return;
            }
            //Process the files:
            $invoices->map(function($invoice){ 
                $file = fopen("storage/app/".$invoice->path, "r");
                while (!feof($file)){
                    $line = fgets($file); 
                    //Here you have a loop to each line of the file, and can do whatever you need with this line:
                    if(strlen($line) > 0){ //If the line is not empty:
                        // Add your logic here:
                    }
                    // Don't forgot to change your `processed` flag to true:
                    $invoice->processed = true;
                    $invoice->save(); 
                } 
            });
        }catch (\Exception $exception){
            $this->error("Something went wrong");
            return $exception->getMessage();
        }
    }
}

Now, open your app/Console/Kernel.php file:

Register your new command in the $commands array:

$commands = [
    Commands\ProcessCSVCommand::class,
];

Schedule a job that runs on your server, checking for files to be processed and, if so, processing them:

In the same file, now at the schedule function:

protected function schedule(Schedule $schedule)
{
     $schedule->command('csv:process')
              ->everyFiveMinutes();
}

Hope it helps.

Queue's are the way to go for this kind of work. Uploading the file essentially needs to happen in that one request, but past that, you can create a queue/s that will chip away at converting the CSV into records in the database.

Related