I am moderately familiar with Pandas, but this is my first time using Dask.
When reading from a large CSV file (1 million rows), I get results from df.shape and df.column.value_counts that suggest I have over 200 Million rows! I have read in smaller dataframes with Dask and do not have this issue. Likewise, when I read in only 1000 rows from the large Csv, I do not have this issue.
df = dd.read_csv('product_locations_2022-09-19.csv',usecols=['custom_rp_code'], dtype={'custom_rp_code': 'object','supplier': 'float64'}, blocksize="10MB")
df.columns
df.custom_rp_code.value_counts().compute()
Proportionally, the value counts are correct, but rows must be being duplicated somehow.