Vectorize eval Fast - Python Numpy Pandas

Zach Kramer 0
Viewed 28

I need a way to vectorize the eval statement that loops through the reference dataframe (df_ref) that has the strings that iloc to the source df (df)

Here's a reprex of the problem:

import pandas as pd
dataValues = ['a','b','c']
df = pd.DataFrame({'values': dataValues})
df_list = ['df.iloc[0,0]','df.iloc[2,0]']
df_ref = pd.DataFrame({'ref':df_list})
#looping 10000 times just to replicate a the amount of times this operation 
#may run in a typical scenario
def help():
    for i in range(10000):
        for index,row in df_ref.iterrows():
            eval(df_ref.ref[index])
%timeit help()

Outputs:

2.84 s ± 366 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Before you respond with an alternative.. My problem will always have some dynamic referencing that I have to replicate in python, so a more straightforward route may not solve my particular problem.

Thanks for the help!

eval vectorization pandas numpy python
1 Answers

As commented, extract the indexes first, then slice into the whole thing:

def help1():
    indexes = df_ref['ref'].str.extract('iloc\[(\d+),\s*(\d+)\]').astype(int)
    return df.to_numpy()[indexes[0], indexes[1]]

and

help1()
> array(['a', 'c'], dtype=object)

%timeit -n 1000 help1()
> 1.11 ms ± 89.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Quang Hoang0
Related