groupy and aggregation in Dask

Question

My dataframe look like the below:

    # initialize list of lists
    data = [['tom', 10], ['nick', 15], ['juli', 14],['tom', 10], ['juli', 15] ]
     
    # Create the pandas DataFrame
    df = pd.DataFrame(data, columns = ['Name', 'Age'])

            Name    Age
        0   tom     10
        1   nick    15
        2   juli    14
        3   tom     10
        4   juli    15

I want to group by the 'Name', count the 'Age' and unique count of 'Age'.

Using pandas I got the result:

           Age
           count    nunique
    Name        
    juli    2      2
    nick    1      1
    tom     2      1

Pandas code :

    types = ['count', 'nunique'] 
    df.groupby('Name').agg({'Age': types})

How can i achieve this in Dask?

In dask, I can do either count or nunique...

    ddf = daskdf.from_pandas(df, npartitions=4)     
    ddf.groupby('Name').Age.count().to_frame().compute()

               Age
        Name    
        nick    1
        tom     2
        juli    2

user_10693596 · Answer

The advantage of lazy computations is that you can specify them one at a time, but the actual computation will be done with some optimization to avoid redundant calculations.

Specifically, you can create lazy computation for nunique and count separately, then combine the computed results:

# calculation with dask
dask_series = ddf.groupby("Name")["Age"]

# these are lazy results that will need to be computed
lazy_results = [
    dask_series.nunique().to_frame(name="age_nunique"),
    dask_series.count().to_frame(name="age_count"),
]

# note that concatenation happens on computed results
print(pd.concat(*dd.compute(lazy_results), axis=1))

Here's the full snippet:

import dask.dataframe as dd
import pandas as pd

# initialize list of lists
data = [["tom", 10], ["nick", 15], ["juli", 14], ["tom", 10], ["juli", 15]]

# Create the pandas DataFrame
df = pd.DataFrame(data, columns=["Name", "Age"])

# calculation with pandas
types = ["count", "nunique"]
print(df.groupby("Name").agg({"Age": types}))
#        Age
#      count nunique
# Name
# juli     2       2
# nick     1       1
# tom      2       1

# calculation with dask
dask_series = ddf.groupby("Name")["Age"]

# these are lazy results that will need to be computed
lazy_results = [
    dask_series.nunique().to_frame(name="age_nunique"),
    dask_series.count().to_frame(name="age_count"),
]

# note that concatenation happens on computed results
print(pd.concat(*dd.compute(lazy_results), axis=1))
#       age_nunique  age_count
# Name
# nick            1          1
# tom             1          2
# juli            2          2

groupy and aggregation in Dask

1 Answers