Does Dask take advantage by the use of parquet partition for groupby task?

Viewed 49

I want to know if dask can perform a clever groupby task?

As example we have the parquet dataset:

  • category=foo/data.pq
  • category=bar/data.pq

those parquet file have following columns: name,age,other

I would like to know for a group by ['category', 'name', 'age'] if dask would take advantage of the parquet partition category?

According to my understanding pandas would try to perform a comparison on all rows. While in such case, it could use the parquet partition in order to do only a group by ['name', 'age'] on each partition to save time and memory.

Thanks

Best regards

0 Answers
Related