Finding and returning the rows of a Pandas dataframe where there is at least one np.inf value

user_8270077 0
Viewed 3114

I have a dataframe that has some np.inf values and I would like to isolate these rows where the np.inf appears and inspect them. However the dataframe has many columns and is not easy to examine one by one, although this could be done inside a loop.

I tried this but failed:

rows_with_inf = [df1[column][df1[column] == np.inf] for column in df1.columns if ((df1[column].isin([np.inf])).sum() !=0)]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-94-768652e951ec> in <module>
----> 1 rows_with_inf = [df1[column][df1[column] == np.inf] for column in df1.columns if ((df1[column].isin([np.inf])).sum() !=0)]

<ipython-input-94-768652e951ec> in <listcomp>(.0)
----> 1 rows_with_inf = [df1[column][df1[column] == np.inf] for column in df1.columns if ((df1[column].isin([np.inf])).sum() !=0)]

~\Anaconda3\envs\tf2\lib\site-packages\pandas\core\generic.py in __nonzero__(self)
   1553             "The truth value of a {0} is ambiguous. "
   1554             "Use a.empty, a.bool(), a.item(), a.any() or a.all().".format(
-> 1555                 self.__class__.__name__
   1556             )
   1557         )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

What would be a compact ('pythonic') way to achieve this objective?

pandas python
2 Answers

Use boolean indexing with DataFrame.any for test at least one np.inf per rows:

df = pd.DataFrame({
        'A':list('abcdef'),
         'B':[4,5,4,5,5,4],
         'C':[7,8,9,4,np.inf,3],
         'D':[1,np.inf,5,7,1,0],
         'E':[5,3,6,9,2,np.inf],
         'F':list('aaabbb')
})



df1 = df[(df == np.inf).any(axis=1)]
print (df1)
   A  B    C    D    E  F
1  b  5  8.0  inf  3.0  a
4  e  5  inf  1.0  2.0  b
5  f  4  3.0  0.0  inf  b

Or if need filter columns and also rows:

m = (df == np.inf)
df2 = df.loc[m.any(axis=1), m.any(axis=0)]
print (df2)

     C    D    E
1  8.0  inf  3.0
4  inf  1.0  2.0
5  3.0  0.0  inf

Details:

print (df == np.inf)
       A      B      C      D      E      F
0  False  False  False  False  False  False
1  False  False  False   True  False  False
2  False  False  False  False  False  False
3  False  False  False  False  False  False
4  False  False   True  False  False  False
5  False  False  False  False   True  False

print ((df == np.inf).any(axis=1))
0    False
1     True
2    False
3    False
4     True
5     True
dtype: bool
user_29010020

You can check if any value is inf for each row.

rows_with_inf = df1[df1.apply(lambda x: any(np.isinf(x)), axis=1)]
user_30188350
Related