I need to read 7.2M files from an S3 bucket and do a simple data processing (filtering). For the data processing, I'm using Dask. I'm wondering how I can speed up the process.
Note that I use a machine with 64 processors and 500GB RAM.
Here's my code:
s3_path = 's3://some_bucket_name/*.csv'
df = dd.read_csv(s3_path, storage_options = {'key': key, 'secret': secret})
df = df[df['some_column'].isin(some_list)]
df.to_parquet('s3://some_bucket_name/some_folder_name/',
storage_options = {'key': key, 'secret': secret}, write_index=False)
df = dd.read_parquet(f's3://some_bucket_name/some_folder_name/',
storage_options = {'key': key, 'secret': secret})
df = df.compute()