How to avoid convertion of data type when substituting certain values to NaN within a pandas DataFrame?

user_5399268 0
Viewed 38

I have a pandas DataFrame and I need to replace certain values to NaN based on a filter. I'm facing a change in data type when doing so. How can I avoid this data type conversion?

Toy example code

import pandas as pd
import numpy as np
df = pd.Series([False, True, False, True])
filter = pd.Series([True, True, False, False])
df[filter] = np.nan

I would expect df to have True and False values as well, appart from NaN. However True values were converted to 1 and False values were converted to 0 as seen in below output.

>>> df
0    NaN
1    NaN
2    0.0
3    1.0
dtype: float64

Partial solution

Only partial solution I can think of now is as follows:

df[df==1] = True
df[df==0] = False
print df

Which results in:

>>> df
0      NaN
1      NaN
2    False
3     True
dtype: object

Question

I know that if I check if a value is 1 and I compare to True it resolves to True, and same happens between 0 and False. However I would like to avoid my values True and False to be changed to 0 and 1 respectively when I convert any value to NaN. Is this possible so that I don´t need to use the partial solution I stated?

pandas dataframe nan python
1 Answers

Change to object before filter

df = pd.Series([False, True, False, True])
filter = pd.Series([True, True, False, False])
df=df.astype('object')
df[filter] = np.nan
df
Out[623]: 
0      NaN
1      NaN
2    False
3     True
dtype: object

More info

df.apply(type)
Out[625]: 
0    <class 'float'>
1    <class 'float'>
2     <class 'bool'>
3     <class 'bool'>
dtype: object
user_79645270
Related