Delete duplicates records from large Laravel collection

user_696860 0
Viewed 1666

I have a large database table (~1 million records) which I need to purge the duplicate records. The table structure is as follows:

|----|-------------|-----|-----|--------------------|
| id | relation_id | foo | bar | timestamp          |
|----|-------------|-----|-----|--------------------|
| 1  | 1           |14.20|0.22 |2019-10-21 14:00:01 |
| 2  | 1           |14.20|0.22 |2019-10-21 14:00:01 |
| 3  | 1           |14.20|0.22 |2019-10-21 14:00:01 |
| 4  | 2           |10.36|0.75 |2019-10-21 14:00:01 |
| 5  | 2           |10.36|0.75 |2019-10-21 14:00:01 |
| 6  | 2           |10.36|0.75 |2019-10-21 14:00:01 |
|----|-------------|-----|-----|--------------------|

As per the example above, there are a lot of records that have the exact same combination of values relation_id, foo, bar and timestamp. I need to create a script that will run to identify the unique values and then delete and duplicate references. So I would end up with something like:

|----|-------------|-----|-----|--------------------|
| id | relation_id | foo | bar | timestamp          |
|----|-------------|-----|-----|--------------------|
| 1  | 1           |14.20|0.22 |2019-10-21 14:00:01 |
| 4  | 2           |10.36|0.75 |2019-10-21 14:00:01 |
|----|-------------|-----|-----|--------------------|

I have tested looping through the relation_id (as there are only 20 unique values) and then running something like this to create a collection of the unique records:

$unique     = collect([]);
$collection = Model::where('relation_id', $relation_id)->chunk(100, function($items) use ($unique) {
    $unique->push($items->unique()->values()->all());
});

From that, I had planned to loop through all of the Model records and delete if the item was not within the $unique collection. Something like this:

Model::chunk(100, function($items) {
    foreach ($items as $item) {
        if(!$unique->contains('id', $item->id)){
            $item->delete;
        }
    }
});

My problem is as the database table is so large, I cannot test if this logic works. Running the first part of the above script (to populate $unique) for a single $relation_id ran in tinker for 30 minutes without yielding results.

I'm relatively confident this isn't the best approach to delete duplicate records as my approach requires multiple queries which I assume could be optimised (which is critical when dealing with such a large table).

So what is the most efficient way to query a database table to check for unique records (based on multiple columns) and delete the duplicate records?

laravel-5 eloquent laravel
2 Answers

You can allow the database to do the heaving lifting here. You can query the database using GROUP BY, then remove everything that doesn't match your query.

$ids = Model::groupBy(['relation_id', 'foo', 'bar', 'timestamp'])
            ->get(['id'])
            ->all();

This translates to the following SQL:

SELECT id FROM models GROUP BY relation_id, foo, bar, timestamp;

So now $ids is an array of IDs where the other columns are unique ([1, 4]). So you can execute the following to remove all other rows from the DB:

Model::whereNotIn('id', $ids)->delete();

However, since $ids is probably huge, you are likely to hit some upper limit constraints. In that case, you can try using array_chunk() to add multiple whereNotIn clauses to the query:

$query = Model::query();

foreach(array_chunk($ids, 500) as $chunk) {
  $query->whereNotIn('id', $chunk);
}

$query->delete();

I created an SQL Fiddle where you can test this out.

user_31094730

For anyone experiencing a similar issue here, I opted to use MySQL to handle the tidy up of the database as it was far more efficient than loading it into memory using Eloquent.

This is the script I used:

DELETE table_name FROM table_name LEFT OUTER JOIN (
SELECT MIN(ID) AS minID FROM table_name GROUP BY table_name.relation_id, table_name.timestamp
) AS keepRowTable ON table_name.ID = keepRowTable.minID
WHERE keepRowTable.minID IS NULL

I accepted @Vince's answer because his approach works using Laravel but we ran into issues when trying to process such a large dataset. Plus he's a hero for being so responsive in the comments!

user_6968600
Related