How to apply math functions while filtering (using loc) in pandas

user_11502061 0
Viewed 363

I have retrived a value of x as x = 10.64589195722904 which I need to match in my existing dataframe using loc. As subtraction result can be a negative value which I must ignore and so I am using math.fabs to achieve it.

fdf = df.loc[(math.fabs(df['x'] - x) <= 0.01 & math.fabs(df['x'] - x) >= 0.001)]

But this is throwing error:

TypeError                                 Traceback (most recent call last)
<ipython-input-256-b8a71a5bd17c> in <module>
     10 #     fdf = df.loc[math.fabs((df['x'] - k) <= 0.001) & (math.fabs(df['x'] - k) >= 0.0001) ]
     11 
---> 12 df.loc[(math.fabs(df['x'] - x) <= 0.01 & math.fabs(df['x'] - x) >= 0.001)]
     13 fdf.head()

~\.conda\envs\pyenv\lib\site-packages\pandas\core\series.py in wrapper(self)
    110         if len(self) == 1:
    111             return converter(self.iloc[0])
--> 112         raise TypeError(f"cannot convert the series to {converter}")
    113 
    114     wrapper.__name__ = f"__{converter.__name__}__"

TypeError: cannot convert the series to <class 'float'>
pandas python
2 Answers

Use numpy.fabs for processing values vectorized way and also add () around masks because priority operators:

s = np.fabs(df['x'] - x)
fdf = df[(s <= 0.01) & (s >= 0.001)]

Alternative is use Series.between:

fdf = df[np.fabs(df['x'] - x).between(0.01, 0.001)]
user_29010020

  • math.fabs only takes a single value, so .apply can be used to create a new column, and then perform the Boolean selection.

  • As shown by jezrael, np.fabs can be used for a vectorized approach

    • The benefit is, numpy is faster
# apply math.fabs and create a column
df['fabs'] = df['x'].apply(lambda row: math.fabs(row) - x)

# filter on the new column
fdf = df[(df['fabs'] <= 0.01) & (df['fabs'] >= 0.001)]
user_77588040
Related