Why is np.nan stored as string and not able to use isnull()/isna()/notnull()/notna()?

user_13376788 0
Viewed 537

This is the piece of code where I assigned np.nan:

df['column1'] = pd.np.where(df5.column5.str.contains("ABC"), "GHI",
               pd.np.where(df5.column5.str.contains("DEF"), "KLM", np.nan))

And I want to filter out rows where 'column1' is NaN. I tried using isnull() and isna() but these didn't work. What worked here is the below:

df = df [df['column1'] != 'nan']

Which means np.nan is stored as string. Why is this happening? And how can I fix this?

pandas series numpy null python
1 Answers

Using np.where will create an numpy array, in this case with strings and np.nan. However, running something like

np.array(['a', np.nan])

will give array(['a', 'nan'], dtype='<U3'). Here np.nan has been converted to 'nan'.

The same happens in this case, running your code without assigning to a column with some example data like so (changed pd.np to np as commented by tidakdiinginkan):

np.where(df5.column5.str.contains("ABC"), "GHI", 
  np.where(df5.column5.str.contains("DEF"), "KLM", np.nan))

gives: 

array(['GHI', 'GHI', 'GHI', 'GHI', 'nan', 'nan', 'KLM', 'KLM'],
      dtype='<U32')

What you can do instead is to use None instead of np.nan:

df['column1'] = pd.np.where(df5.column5.str.contains("ABC"), "GHI",
               pd.np.where(df5.column5.str.contains("DEF"), "KLM", None))

which will allow you to use df['column1'].isnull() as expected.

user_75795470
Related