Pandas groupby with dropna set to True generating wrong output

user_14094460 0
Viewed 206

In the following snippet:

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "a": [1, 2, 3, 4, 5, 6, 7, 8, 9], 
        "b": [1, np.nan, 1, np.nan, 2, 1, 2, np.nan, 1]
    }
)
df_again = df.groupby("b", dropna=False).apply(lambda x: x)

I was expecting df and df_again to be identical. But they are not:

df
   a    b
0  1  1.0
1  2  NaN
2  3  1.0
3  4  NaN
4  5  2.0
5  6  1.0
6  7  2.0
7  8  NaN
8  9  1.0

df_again
   a    b
0  1  1.0
2  3  1.0
4  5  2.0
5  6  1.0
6  7  2.0
8  9  1.0

Now, if I tweak slightly the lambda expression to "see" what is going on by df.groupby("b", dropna=False).apply(lambda x: print(x)) I can actually visualize that also the portion of the df where b was NaN was processed.

What am I missing here? (Using pandas 1.3.1 and numpy 1.20.3)

pandas-groupby pandas python-3.x
2 Answers

It's because None and None are the same thing:

>>> None == None
True
>>>

You have to use np.nan:

>>> np.NaN == np.NaN
False
>>>

So try this:

df = pd.DataFrame(
    {
        "a": [1, 2, 3, 4, 5, 6, 7, 8, 9], 
        "b": [1, np.NaN, 1, np.NaN, 2, 1, 2, np.NaN, 1]
    }
)
df_again = df.groupby("b", dropna=False).apply(lambda x: x)

Now df and df_again would be the same:

>>> df
   a    b
0  1  1.0
1  2  NaN
2  3  1.0
3  4  NaN
4  5  2.0
5  6  1.0
6  7  2.0
7  8  NaN
8  9  1.0
>>> df_again
   a    b
0  1  1.0
1  2  NaN
2  3  1.0
3  4  NaN
4  5  2.0
5  6  1.0
6  7  2.0
7  8  NaN
8  9  1.0
>>> df.equals(df_again)
True
>>>
user_87083640

This was a bug introduced in pandas 1.2.0 as described here and was solved here.

user_140944600
Related