Use faker in laravel when inserting thousand of records

Viewed 2257

I have one issue with laravel faker, I was looking for a tutorial to insert thousands of records using seeders

This was my PostSeeder.php:

<?php

namespace Database\Seeders;

use Illuminate\Database\Seeder;
use App\Models\Post;
use App\Models\User;

class PostSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {            
        Post::factory(10)->create();
    }
}

Here I was inserting 10 posts, but I need to test thousands or millions of records, so I saw a tutorial and modified the seeder

<?php

namespace Database\Seeders;

use Illuminate\Database\Seeder;
use App\Models\Post;
use App\Models\User;
use Illuminate\Support\Str;

class PostSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {        
        
        $users= collect(User::all()->modelKeys());
        $data = [];

        for ($i = 0; $i < 100000; $i++) {
            $data[] = [
                'body' => Str::random(50),
                'image' => 'https://via.placeholder.com/640x480.png/0077dd?text=inventore',
                'user_id' => $users->random(),
                'created_at' => now()->toDateTimeString(),
                'updated_at' => now()->toDateTimeString(),
            ];
        }

        $chunks = array_chunk($data, 10000);

        foreach ($chunks as $chunk) {
            Post::insert($chunk);
        }
        
    }
}

With this approach I can insert thousand of records faster, but the problem is that I am not inserting correctly the body and image field

I wanted to try something with faker, in my factory I have this:

PostFactory.php

<?php

namespace Database\Factories;

use App\Models\Post;
use App\Models\User;
use Illuminate\Database\Eloquent\Factories\Factory;

class PostFactory extends Factory
{
    /**
     * The name of the factory's corresponding model.
     *
     * @var string
     */
    protected $model = Post::class;

    /**
     * Define the model's default state.
     *
     * @return array
     */
    public function definition()
    {
        return [
            'body' => $this->faker->text,
            'image' => $this->faker->imageUrl(),
            'user_id' => function() {
                return User::factory()->create()->id;
            }
        ];
    }
}

I would like to use faker methods like these ones in the PostSeeder but I can't, what can I do? thank you.

EDIT:

I tried this:

public function run(Faker $faker)
    {                
        
        $users= collect(User::all()->modelKeys());
        $data = [];

        for ($i = 0; $i < 50000; $i++) {
            $data[] = [
                'content' => $faker->text,
                'image_path' => $faker->imageUrl(),
                'user_id' => $users->random(),
                'created_at' => now()->toDateTimeString(),
                'updated_at' => now()->toDateTimeString(),
            ];
        }

        $chunks = array_chunk($data, 5000);

        foreach ($chunks as $chunk) {
            Post::insert($chunk);
        }
        
    }

And I got this message: PDOException::("SQLSTATE[HY000]: General error: 2006 MySQL server has gone away") But when I try with fewer records it works, so, I changed the seeder like this:

$users= collect(User::all()->modelKeys());
$posts = Post::factory(10)->create();        
$posts = collect($posts->only(['content','image_path']));
...
...
'content' => $posts->random()->content,
'image_path' => $posts->random()->image_path
...

and this doesn't work, it got this error: You requested 1 items, but there are only 0 items available. It looks like $posts->only(['content','image_path']) is not working properly. So I tried this:

Post::factory(10)->create();
$tweets = Tweet::select(['content','image_path'])->get();
...
'content' => $posts->random()->content,
'image_path' => $posts->random()->image_path
...

And again it works with a few records, but when I try with thousands, I get this error again: PDOException::("SQLSTATE[HY000]: General error: 2006 MySQL server has gone away")

What can I do? thank you

1 Answers

Since model factories create in-memory objects, because of big memory usages it is not suitable for large seeds.

But you can use Faker for data generation:

use Faker\Generator as Faker;
class PostSeeder extends Seeder
{
    public function run(Faker $faker)
    {
        $users= collect(User::all()->modelKeys());
        $data = [];

        for ($i = 0; $i < 100000; $i++) {
            $data[] = [
                'body' => $faker->text,
                'image' => $faker->imageUrl(),
                'user_id' => $users->random(),
                'created_at' => now()->toDateTimeString(),
                'updated_at' => now()->toDateTimeString(),
            ];
        }

        $chunks = array_chunk($data, 10000);

        foreach ($chunks as $chunk) {
            Post::insert($chunk);
        }
    }
}
Related