Speeding up data processing with Dask

Viewed 20

I need to read 7.2M files from an S3 bucket and do a simple data processing (filtering). For the data processing, I'm using Dask. I'm wondering how I can speed up the process.

Note that I use a machine with 64 processors and 500GB RAM.

Here's my code:

s3_path = 's3://some_bucket_name/*.csv'

df = dd.read_csv(s3_path, storage_options = {'key': key, 'secret': secret})

df = df[df['some_column'].isin(some_list)]
    
df.to_parquet('s3://some_bucket_name/some_folder_name/', 
                  storage_options = {'key': key, 'secret': secret}, write_index=False)

df = dd.read_parquet(f's3://some_bucket_name/some_folder_name/',
               storage_options = {'key': key, 'secret': secret})

df = df.compute()
0 Answers
Related