Python Pandas how to speed up the __init__ class function correctly with numba?

Serega 0
Viewed 25

I have a class that does different mathematical calculations on a cycle

I want to speed up its processing with Numba

And now I'm trying to apply Namba to init functions

The class itself and its init function looks like this:

class Generic(object):
    #@numba.vectorize
    def __init__(self, N, N1, S1, S2):
        A = [['Time'],['Time','Price'], ["Time", 'Qty'], ['Time', 'Open_interest'], ['Time','Operation','Quantity']] 
        self.table = pd.DataFrame(pd.read_csv('Datasets\\RobotMath\\table_OI.csv'))
        self.Reader()
        for a in A:
            if 'Time' in a:
                self.df = pd.DataFrame(pd.read_csv(Ex2_Csv, usecols=a, parse_dates=[0]))
                self.df['Time'] = self.df['Time'].dt.floor("S", 0)
                self.df['Time'] = pd.to_datetime(self.df['Time']).dt.time
                if a == ['Time']:
                    self.Tik()
                elif a == ['Time','Price']:
                    self.Poc()
                    self.Pmm()
                    self.SredPrice()
                    self.Delta_Ema(N, N1)
                    self.Comulative(S1, S2)
                    self.M()
                elif a == ["Time", 'Qty']:
                    self.Volume()
                elif a == ['Time', 'Open_interest']:
                    self.Open_intrest()
                elif a == ['Time','Operation','Quantity']:
                    self.D()
                    #self.Dataset()  
                else:
                    print('Something went wrong', f"Set Error: {0} ".format(a))

All functions of the class are ordinary column calculations using Pandas.

Here are two of them for example:

    def Tik(self):
        df2 = self.df.groupby('Time').value_counts(ascending=False)
        df2.to_csv('Datasets\\RobotMath\\Tik.csv', )

    def Poc(self):
        g = self.df.groupby('Time', sort=False)
        out = (g.last()-g.first()).reset_index()
        out.to_csv('Datasets\\RobotMath\\Poc.csv', index=False)

I tried to use Numba in different ways, but I got an error everywhere.

Is it possible to speed up exactly the init function? Or do I need to look for another way and I can 't do without rewriting the class ?

pandas numba python
1 Answers

So there's nothing special or magical about the init function, at run time, it's just another function like any other.

In terms of performance, your class is doing quite a lot here - it might be worth breaking down the timings of each component first to establish where your performance hang-ups lie.

For example, just reading in the files might be responsible for a fair amount of that time, and for Ex2_Csv you repeatedly do that within a loop, which is likely to be sub-optimal depending on the volume of data you're dealing with, but before targeting a resolution (like Numba for example) it'd be prudent to identify which aspect of the code is performing within expectations.

You can gather that information in a number of ways, but the simplest might be to add in some tagged print statements that emit the elapsed time since the last print statement.

e.g.

start = datetime.datetime.now()
print("starting", start)

## some block of code that does XXX

XXX_finish = datetime.datetime.now()
print("XXX_finished at", XXX_finish, "taking", XXX_finish-start)

## and repeat to generate a runtime report showing timings for each block of code

Then you can break the runtime of your program down into feature-aligned chunks, then when tweaking you'll be able to directly see the effect. Profiling the runtime of your code like this can really help when making performance tweaks, and it helps sharpen focus on what tweaks are benefiting/harming specific areas of your code.

For example, in the section that's performing the group-bys, with some timestamp outputs before and after you can compare running it with the numba turned on, and then again with it turned off.

From there, with a little careful debugging, sensible logging of times and a bit of old-fashioned sleuthing (I tend to jot these kinds of things down with paper and pencil) you (and if you share your findings, we) ought to be in a better position to answer your question more precisely.

Thomas Kimber0
Related