how to reverse the row order in a column of a dask dataframe?

user2woI 0
Viewed 23

here is the code

import dask.dataframe as dd
import pandas as pd


df = pd.DataFrame({
'col1' : ['A', 'A', 'E', np.nan, 'D', 'C','B','C'],
'col2' : [2, 1, 9, 8, 7, 4,10,5],
'col3': [0, 1, 9, 4, 2, 3,1,2],
'col4': [11,12,12,13,14,55,56,22], })

out_1=df.loc[::-1,"col4"]

dd_df=dd.from_pandas(df,npartitions=5)
out_2=dd_df.loc[::-1,"col4"]

#out_2 throws an error

I know that Dask doesn't work the same way as pandas. How can I get the same output like out_1 with DASK?

dask dataframe pandas python
1 Answers

You can reverse the order of the partitions, and also schedule a job to reverse the order of rows within each partition, like so:

In [30]: rev_df = dd.concat(
    ...:     [df.partitions[i] for i in range(df.npartitions - 1, -1, -1)]
    ...: ).map_partitions(lambda x: x[::-1], meta=df)
    ...:

In [31]: rev_df.compute()
Out[31]:
  col1  col2  col3  col4
7    C     5     2    22
6    B    10     1    56
5    C     4     3    55
4    D     7     2    14
3  NaN     8     4    13
2    E     9     9    12
1    A     1     1    12
0    A     2     0    11

This would work the same way for a column or series:

rev_col1 = dd.concat(
    [
        df["col1"].partitions[i]
        for i in range(df.npartitions - 1, -1, -1)
    ]
).map_partitions(lambda x: x[::-1], meta=df["col1"])
Michael Delgado0
Related