Is it possible to ignore NA but not drop it in a dataframe?

user_15306662 0
Viewed 595

My dataframe has 80 columns by 1 million rows.

There is a lot of NaN in the dataset, if I were to drop all NA values I would lose way too much data for it to be useful.

I was wondering if there was a way to go through the columns and when it would do any calculations later on, like mean, medium, mode, or more advanced calculations, it would just pretend the values aren't there.

It's not always as just NaN, it sometimes includes values like "Infinity" which should be ignored.

Is there something like:

Find NaN and Infinity in Dataframe, flag them and DO NOT CONSIDER them IN CALCULATIONS

I tried finding a command to do it but any information on it, maybe I'm just being silly.

pandas dataframe python
2 Answers

Some pandas functions automatically ignore NaN values. For example:

import pandas as pd
s = pd.Series([np.nan, 1, 2, 3])
s.mean()

Gives the result 2.0, so it ignores NaN values. Same goes for median and mode.

For the inf values, I would ask myself if they are actually meaningful. If they are not you can replace them with NaN for the mean calculation. The median and mode functions are already able to handle it.

user_118896020

Just use nanmean() method in numpy for finding mean of particular column which has NaN values

import pandas as pd
import numpy as np
series = pd.Series([np.nan, 1, np.nan, 3,np.nan,2])
mean=np.nanmean(series)

Now if you print this:-

 print(mean)

It gives 2.0 as output which means It will not influence the mean calculation and after that use df.fillna()

series.fillna(mean,inplace=True)
user_142898920
Related