Replace all inf, -inf values with NaN in a pandas dataframe

user_13067389 0
Viewed 4472

I have a large dataframe with inf, -inf values in different columns. I want to replace all inf, -inf values with NaN

I can do so column by column. So this works:

df['column name'] = df['column name'].replace(np.inf, np.nan)

But my code to do so in one go across the dataframe does not.

df.replace([np.inf, -np.inf], np.nan)

The output does not replace the inf values

inf pandas dataframe find replace
2 Answers

pandas.Series.replace doesn't happen in-place.

So the problem with your code to replace the whole dataframe does not work because you need to assign it back or, add inplace=True as a parameter. That's also why your column by column works, because you are assigning it back to the column df['column name'] = ...

Therefore, change df.replace([np.inf, -np.inf], np.nan) to either:

df.replace([np.inf, -np.inf], np.nan,inplace=True)

Or assign back to a new dataframe:

df = df.replace([np.inf, -np.inf], np.nan)

enter image description here

user_91673820

TL;DR

Replacing inf and -inf

df = df.replace([np.inf, -np.inf], np.nan)

Note that inplace is possible but not recommended and will soon be deprecated.

Slower df.applymap options:

  • df = df.applymap(lambda x: np.nan if x in [np.inf, -np.inf] else x)
  • df = df.applymap(lambda x: np.nan if np.isinf(x) else x)
  • df = df.applymap(lambda x: x if np.isfinite(x) else np.nan)

Setting mode.use_inf_as_na

Note that we don't actually have to modify df at all. Setting mode.use_inf_as_na will simply change the way inf and -inf are interpreted:

True means treat None, nan, -inf, inf as null
False means None and nan are null, but inf, -inf are not null (default)

  • Either enable globally

    pd.set_option('mode.use_inf_as_na', True)

  • Or locally via context manager

    with pd.option_context('mode.use_inf_as_na', True):
    ...
user_131383640
Related