How can I make my python program run faster?

Viewed 195

I am reading in a .csv file and creating a pandas dataframe. The file is a file of stocks. I am only interested in the date, the company, and the closing cost. I want my program to find the max profit with the starting date, the ending date and the company. It needs to use the divide and conquer algorithm. I only know how to use for loops but it takes forever to run. The .csv file is 200,000 rows. How can I get this to run fast?

import pandas as pd
import numpy as np
import math

def cleanData(file):
    df = pd.read_csv(file)
    del df['open']
    del df['low']
    del df['high']
    del df['volume']
    return np.array(df)
    
df = cleanData('prices-split-adjusted.csv')

bestStock = [None, None, None, float(-math.inf)]

def DAC(data):
    global bestStock

    if len(data) > 1:
        mid = len(data)//2
        left = data[:mid]
        right = data[mid:]
    
        DAC(left)
        DAC(right)
    
        for i in range(len(data)):
            for j in range(i+1,len(data)):
                if data[i,1] == data[j,1]:
                    profit = data[j,2] - data[i,2]
                    if profit > bestStock[3]:
                        bestStock[0] = data[i,0]
                        bestStock[1] = data[j,0]
                        bestStock[2] = data[i,1]
                        bestStock[3] = profit
                    
                    print(bestStock)
    print('\n')
    return bestStock
    
print(DAC(df))
3 Answers

I've got two things for your consideration (my answer tries not to change your algorithm approach i.e. nested loops and recursive funcs and tackles the low lying fruits first):

  1. Unless you are debugging, try to avoid print() inside a loop. (in your case .. print(bestStock) ..) The I/O overhead can add up esp. if you are looping across large datasets and printing to screen often. Once you are OK with your code, comment it out to run on your full dataset and uncomment it only during debugging sessions. You can expect to see some improvement in speed without having to print to screen in the loop.

  2. If you are after even more ways to 'speed it up', I found in my case (similar to yours which I often encounter especially in search/sort problems) that simply by switching the expensive part (the python 'For' loops) to Cython (and statically defining variable types .. this is KEY! to SPEEEEDDDDDD) gives me several orders of magnitude speed ups even before optimizing implementation. Check Cython out https://cython.readthedocs.io/en/latest/index.html. If thats not enough, then parrelism is your next best friend which would require rethinking your code implementation.

The main problems causing slow system performance are:

  1. You manually iterate over 2 columns in nested loops without using pandas operations which make use of fast ndarray functions;
  2. you use recursive calls which looks nice and simple but slow.

Setting the sample data as follows:

          Date  Company         Close
0   2019-12-31     AAPL     73.412498
1   2019-12-31       FB    205.250000
2   2019-12-31     NFLX    323.570007
3   2020-01-02     AAPL     75.087502
4   2020-01-02       FB    209.779999
...        ...      ...           ...
184 2020-03-30       FB    165.949997
185 2020-03-30     NFLX    370.959991
186 2020-03-31     AAPL     63.572498
187 2020-03-31       FB    166.800003
188 2020-03-31     NFLX    375.500000

189 rows × 3 columns

Then use the following codes (modify the column labels to your labels if different):

df_result = df.groupby('Company').agg(Start_Date=pd.NamedAgg(column='Date', aggfunc="first"), End_Date=pd.NamedAgg(column='Date', aggfunc="last"), bestGain=pd.NamedAgg(column='Close', aggfunc=lambda x: x.max() - x.iloc[0]))

Resulting output:

        Start_Date    End_Date   bestGain
Company         
AAPL    2019-12-31  2020-03-31   8.387505
FB      2019-12-31  2020-03-31  17.979996
NFLX    2019-12-31  2020-03-31  64.209991

To get the entry with greatest gain:

df_result.loc[df_result['bestGain'].idxmax()]

Resulting output:

Start_Date    2019-12-31 00:00:00
End_Date      2020-03-31 00:00:00
bestGain                64.209991
Name: NFLX, dtype: object

Execution time comparison

With my scaled down data of 3 stocks over 3 months, the codes making use of pandas function (takes 8.9ms) which is about about half the execution time with the original codes manually iterate over the numpy array with nested loops and recursive calls (takes 16.9ms) even after the majority of print() function calls removed.

Your codes with print() inside DAC() function removed:

%%timeit
"""
def cleanData(df):
    # df = pd.read_csv(file)
    del df['Open']
    del df['Low']
    del df['High']
    del df['Volume']
    return np.array(df)
"""    
# df = cleanData('prices-split-adjusted.csv')
# df = cleanData(df0)
df = np.array(df0)

bestStock = [None, None, None, float(-math.inf)]

def DAC(data):
    global bestStock

    if len(data) > 1:
        mid = len(data)//2
        left = data[:mid]
        right = data[mid:]
    
        DAC(left)
        DAC(right)
    
        for i in range(len(data)):
            for j in range(i+1,len(data)):
                if data[i,1] == data[j,1]:
                    profit = data[j,2] - data[i,2]
                    if profit > bestStock[3]:
                        bestStock[0] = data[i,0]
                        bestStock[1] = data[j,0]
                        bestStock[2] = data[i,1]
                        bestStock[3] = profit
                    
                    # print(bestStock)
    # print('\n')
    return bestStock
    
print(DAC(df))

[Timestamp('2020-03-16 00:00:00'), Timestamp('2020-03-31 00:00:00'), 'NFLX', 76.66000366210938]
16.9 ms ± 303 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

New simplified codes in pandas' way of coding:

%%timeit
df_result = df.groupby('Company').agg(Start_Date=pd.NamedAgg(column='Date', aggfunc="first"), End_Date=pd.NamedAgg(column='Date', aggfunc="last"), bestGain=pd.NamedAgg(column='Close', aggfunc=lambda x: x.max() - x.iloc[0]))
df_result.loc[df_result['bestGain'].idxmax()]

8.9 ms ± 195 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Solution using recursive function:

The main problem of your recursive function lies in that you did not make use of the results of recursive calls of reduced size data.

To properly use recursive function as a divide-and-conquer approach, you should take 3 major steps:

  1. Divide the whole set of data into smaller pieces and handle the smaller pieces by recursive calls each taking one of the smaller pieces
  2. Handle the end-point case (the easiest case most of the time) in each recursive call
  3. Consolidate the results of all recursive calls of smaller pieces

The beauty of recursive calls is that you can solve a complicated problem by replacing the processing with 2 much more easier steps: 1st step is to handle the end-point case where you can handle for most of the time only ONE data item (which is most often easy). 2nd step is to just take another easy step to consolidate the results of the reduced-size calls.

You managed to take the first step but not the other 2 steps. In particular, you did not take advantage of simplifying the processing by making use of the results of smaller pieces. Instead, you handle the whole set of data in each call by looping all over all rows in the 2-dimensional numpy array. The nested loop logics is just like a "Bubble Sort" [with complexity order(n squared) instead of order(n)] . Hence, your recursive calls are just wasting time without value!

Suggest to modify your recursive functions as follows:

def DAC(data):
    # global bestStock                 # define bestStock as a local variable instead
    bestStock = [None, None, None, float(-math.inf)]    # init bestStock

    if len(data) = 1:                  # End-point case: data = 1 row
        bestStock[0] = data[0,0]
        bestStock[1] = data[0,0]
        bestStock[2] = data[0,1]
        bestStock[3] = 0.0
    elif len(data) = 2:                # End-point case: data = 2 rows
        bestStock[0] = data[0,0]
        bestStock[1] = data[1,0]
        bestStock[2] = data[0,1]       # Enhance here to allow stock break 
        bestStock[3] = data[1,2] - data[0,2]
    elif len(data) >= 3:               # Recursive calls and consolidate results
        mid = len(data)//2
        left = data[:mid]
        right = data[mid:]
    
        bestStock_left = DAC(left)
        bestStock_right = DAC(right)

        # Now make use of the results of divide-and-conquer and consolidate the results
        bestStock[0] = bestStock_left[0]
        bestStock[1] = bestStock_right[1]
        bestStock[2] = bestStock_left[2]    # Enhance here to allow stock break 
        bestStock[3] = bestStock_left[3] if bestStock_left[3] >= bestStock_right[3] else bestStock_right[3] 
    
    # print(bestStock)
    # print('\n')
    return bestStock

Here we need to handle 2 kinds of end-point cases: 1 row and 2 rows. The reason is that for case with only 1 row, we cannot calculate the gain and can only set the gain to zero. Gain can start to calculate with 2 rows. If not split into these 2 end-point cases, we could end up only propagating zero gain all the way up.

Here is a demo of how you should code the recursive calls to take advantage of it. There is limitation of the codes that you still need to fine-tune. You have to enhance it further to handle stock break case. The codes for 2 rows and >= 3 rows now assume no stock break at the moment.

Related