remove duplicates on large csv file, only look at a subset of columns

user_9150012 0
Viewed 441

I need to remove duplicates for a large .csv file (50+GB), and I would like to do this using python. Several other questions address the issue broadly (ex: here and here), but they deal with exact duplicates.

In my case, the duplicates are not exact duplicates. The setup of my file is such that I compiled this file by pulling rows from several sources, and one column indicates the source of origin. This means that I would like to remove duplicates for a subset of columns. The size of the file means that I cannot load it into memory, so pandas is out.

How can I approach this problem (possibly modifying the solutions I linked to)?

large-data duplicates python
2 Answers

For large dataframes, you can use dask. It's syntax is similar to pandas so if you are familiar with it, you will have no problem. Example:

import dask.dataframe as dsk
df = dsk.read_csv('my_csv.csv')

The syntax for drop duplicates Here:

df.drop_duplicates(subset=None, split_every=None, split_out=1, ignore_index=False, **kwargs)

allows for subsets.

Example:

from dask.distributed import Client
import dask.dataframe as dsk
client = Client(memory_limit='32GB') # set your limit here
df = dsk.read_csv("my_csv", sample=100)
results = df.drop_duplicates(subset=['my_col'], split_out=df.npartitions)
results.to_csv(outdir, index=False)
user_41459410
user_29772560
Related