I want to know if dask can perform a clever groupby task?
As example we have the parquet dataset:
category=foo/data.pqcategory=bar/data.pq
those parquet file have following columns: name,age,other
I would like to know for a group by ['category', 'name', 'age'] if dask would take advantage of the parquet partition category?
According to my understanding pandas would try to perform a comparison on all rows. While in such case, it could use the parquet partition in order to do only a group by ['name', 'age'] on each partition to save time and memory.
Thanks
Best regards