Dask - How to call ".compute()" on multiple dataframes

Viewed 24

I have two dataframes that are interdependent in my calculation and I would like to get the results on both with one compute() call. The code can be summarized as follows:

import dask
import dask.dataframe
import dask.distributed
import pandas as pd

df = dask.dataframe.from_pandas(
    pd.DataFrame({
        "group": ['a', 'b', 'a', 'b', 'a', 'b', 'b'],
        "var_1": [0, 1, 2, 1, 2, 1, 0],
        "var_2": [1, 1, 2, 1, 2, 1, 0]}), npartitions=2)

with dask.distributed.Client() as client:
    for i in range(10):
        df_agg = foo(df)
        df = bar(df, df_agg)

print(df.compute())
print(df_agg.compute()) # -> I would like to have only one .compute() call and get the results of both dataframes (df and df_agg)

Many thanks for your help

1 Answers
Related