Fastest way to compute only last row in pandas dataframe

Viewed 45

I am trying to find the fastest way to compute the results only for the last row in a dataframe. For some reason, when I do so it is slower than computing the entire dataframe. What am I doing wrong here? What would be the correct way to access only the last two rows and compute their values?

Currently these are my results:

Processing time of add_complete(): 1.333 seconds
Processing time of add_last_row_only(): 1.502 seconds

import numpy as np
import pandas as pd


def add_complete(df):
    df['change_a'] = df['a'].diff()
    df['change_b'] = df['b'].diff()
    df['factor'] = df['change_a'] * df['change_b']

def add_last_row_only(df):
    df.at[df.index[-1], 'change_a_last_row'] = df['a'].iloc[-1] - df['a'].iloc[-2]
    df.at[df.index[-1], 'change_b_last_row'] = df['b'].iloc[-1] - df['b'].iloc[-2]
    df.at[df.index[-1], 'factor_last_row'] = df['change_a_last_row'].iloc[-1] * df['change_b_last_row'].iloc[-1]

def main():
    a = np.arange(200_000_000).reshape(100_000_000, 2)
    df = pd.DataFrame(a, columns=['a', 'b'])
    add_complete(df)
    add_last_row_only(df)
    print(df.tail())


1 Answers

Unless I am missing something, for this kind of operation I would use numpy on the two last lines:

%%timeit
changes = np.diff(df.values[-2:,:],axis=0)
factor = np.product(changes)

21µs just this operation, yes, microseconds.

If I add insertion it increases to 511ms, even filling all with same value. I suspect the problem comes from handling around a 1.5Gb dataframe, which actually doubles the size when inserting the two extra columns.

%%timeit
changes = np.diff(df.values[-2:,:],axis=0)
factor = np.product(changes)
df['factor']=factor
df['changes_a']=changes[0][0]
df['changes_b']=changes[0][1]
Related