Finding previous trading day in pandas is very slow

user_15542251 0
Viewed 330

I have this df

data1_txt = """
date
7/8/2021
7/6/2021
6/29/2021
"""

I need to get the previous trading day. I manage to do it with pandas_market_calendars python package. Here is my whole code

import io
import pandas_market_calendars as mcal
from pandas.tseries.offsets import CustomBusinessDay
import datetime
import pandas as pd

data1_txt = """
date
7/8/2021
7/6/2021
6/29/2021
"""
df = pd.read_fwf(io.StringIO(data1_txt))
df['date'] = pd.to_datetime(df['date'])

nyse = mcal.get_calendar('NYSE')
holidays = nyse.holidays()
holidays = list(holidays.holidays)
US_BUSINESS_DAY = CustomBusinessDay(holidays=holidays)
df['date_prev'] = df['date'] - 1 * US_BUSINESS_DAY

The code does the job. But the process is very slow for a large dataset. Is it possible to somehow increase the code speed?

P.S. When I run the code python gives me this warning:

PerformanceWarning: Non-vectorized DateOffset being applied to Series or DatetimeIndex
  warnings.warn(
pandas trading datetime python
1 Answers

I actually found a way, it makes use of np.busday_offset

def other(df):
    nyse = mcal.get_calendar('NYSE')
    holidays = nyse.holidays().holidays  

    # check out the docs for how to adjust roll to your preference
    result = np.busday_offset(df["date"].values.astype('datetime64[D]'),     
             [-1], roll= "forward",  weekmask= "1111100", holidays= holidays)
    return result

After creating a df with 10,000 rows of random dates and passing it to your and to other function. The speed went from

Stat(s) for 10 execution(s) of yours:
mean: 893.01282 ms
median: 883.1989 ms
stdv: 83.3837 ms
max: 1134.7835 ms
min: 760.0869 ms

to:

Stat(s) for 10 execution(s) of other:
mean: 278.60783 ms
median: 274.44165 ms
stdv: 27.9785 ms
max: 330.3079 ms
min: 235.4329 ms

And with your code:

data1_txt = """
date
7/8/2021
7/6/2021
6/29/2021
"""
df = pd.read_fwf(io.StringIO(data1_txt))
df['date'] = pd.to_datetime(df['date'])

def yours(df):
    nyse = mcal.get_calendar('NYSE')
    holidays = nyse.holidays()
    holidays = list(holidays.holidays)
    US_BUSINESS_DAY = CustomBusinessDay(holidays=holidays)
    result =  df['date'] - 1 * US_BUSINESS_DAY
    return result


display(yours(df), other(df))
>>> 
0   2021-07-07
1   2021-07-02
2   2021-06-28
Name: date, dtype: datetime64[ns]

array(['2021-07-07', '2021-07-02', '2021-06-28'], dtype='datetime64[D]')
user_141706720
Related