NaN in pandas Series.tolist() behaves differently from NaN in list

user_3198568 0
Viewed 820

Why does

>> import pandas as pd
>> import numpy as np

>> list(pd.Series([np.nan, np.nan, 2, np.nan, 2])) == [np.nan, np.nan, 2, np.nan, 2]

return False? I get the same result with pd.Series([np.nan, np.nan, 2, np.nan, 2]).tolist(). I was trying to count the most common element in a pandas groupby object (so basically, a pandas Series) through the following function

def get_most_common(srs):
    """
    Returns the most common value in a list. For ties, it returns whatever
    value collections.Counter.most_common(1) gives.
    """
    from collections import Counter

    x = list(srs)
    my_counter = Counter(x)
    most_common_value = my_counter.most_common(1)[0][0]

    return most_common_value

and just realized that I get wrong counts for multiple NaNs even if I have a step x = list(srs).

EDIT: Just to clarify why this is an issue for me:

>>> from collections import Counter
>>> Counter(pd.Series([np.nan, np.nan, np.nan, 2, 2, 1, 5]).tolist())
Counter({nan: 1, nan: 1, nan: 1, 2.0: 2, 1.0: 1, 5.0: 1}) # each nan is counted differently
>>> Counter([np.nan, np.nan, np.nan, 2, 2, 1, 5])
Counter({nan: 3, 2: 2, 1: 1, 5: 1}) # nan count of 3 is correct
pandas numpy python
2 Answers

The root issue, as @emilaz already stated, is that nan != nan in all cases. However, the object reference is what matters in your observation.

Observe the following object references between list and pd.Series:

>>> s = pd.Series([np.nan, np.nan, np.nan, 2, 2, 1, 5])
>>> s.apply(id)
0    149706480
1    202463472
2    202462336
3    149706912
4    149706288
5    149708784
6    149707200
dtype: int64

>>> l = [np.nan, np.nan, np.nan, 2, 2, 1, 5]
>>> list(map(id, l))
[68634768, 68634768, 68634768, 1389126848, 1389126848, 1389126832, 1389126896]

The np.nan object shares the same reference as the imported np.nan object in list, whereas a new reference is created for each Series (which makes sense for pandas usage).

The answer therefore is not to compare nan in such fashion. pandas have its own ways to deal with nan, so depending on your actual activity, there may be a much simpler answer (e.g. df.groupby('some col').count()) than you envisioned.

user_91833440

In python, equating to nan always returns False. So the following behavior is expected:

import numpy as np
np.nan == np.nan
>>>> False

Which is why your list comparisons return False.

A possible workaround would be this:

import pandas as pd
import numpy as np

foo= list(pd.Series([np.nan, np.nan, 2, np.nan, 2]))
bar= [np.nan, np.nan, 2, np.nan, 2]

np.allclose(foo,bar, equal_nan=True)
>>>> True

This might interest you: comparing numpy arrays containing NaN.

For finding the most common element, I'd suggest using pandas and the value_counts() method:

pd.Series([np.nan, np.nan, 2, np.nan, 2]).value_counts()
>>>> 2.0  2

If you care about nan counts, you can simply pass dropna=False to the method:

pd.Series([np.nan, np.nan, 2, np.nan, 2]).value_counts()
>>>> NaN  3
     2.0  2
user_89218670
Related