Best way to pull data from previous 5 rows in Pandas

Viewed 116

Looking for a method to do the following, but more efficient.

I am taking stock data and looking at the Highs, and lows for the last 5 days and putting it in a new data frame:

Date(index)     High    Low
datetime obj1   1        1 
datetime obj2   2        2 
datetime obj3   3        3
datetime obj4   4        4 
datetime obj5   5        5 
datetime obj6   6        6

Will become

Date(index)     High              Low
datetime obj1   []                [] 
datetime obj2   []                []
datetime obj3   []                []
datetime obj4   []                []
datetime obj5   [1,2,3,4,5]       [1,2,3,4,5]
datetime obj6   [2,3,4,5,6]       [2,3,4,5,6]

Here is the code I have, it works, but is a brute forced nested for loop. Is there a way to vectorize this or at least pull the data faster?

  df = getdata("SWBI",today,days_back) #just makes the df for the stock data
  date_list = df.index.to_list() #makes a list of dates to iterate over
  counter= 0
  df_predictions= pd.DataFrame({
      "date":[],
      "hi_his":[],
      "lo_his":[]
  })

  for i in date_list:
    dates = date_list[counter-5:counter] #Makes a list of the previous 5 dates
    counter += 1 
    hi = [] 
    lo = []
    for date in dates: #makes a list of the values for those 5 days 
      lo.append(df.loc[date]["Low"])
      hi.append(df.loc[date]["High"])
    #Make a temporary df to append
    df_temp= pd.DataFrame({
      "date":i,
      "hi_his":[hi],
      "lo_his":[lo]
      })
    df_predictions = df_predictions.append(df_temp) #df ready to de linear regression predictions

I know

2 Answers

You can build a matrix of sliding windows using shift():

windows = pd.concat([df.shift(n) for n in range(5)], axis=1)

#             High  Low  High  Low  High  Low  High  Low  High  Low
# Date                                                             
# 2021-01-01     1    1   NaN  NaN   NaN  NaN   NaN  NaN   NaN  NaN
# 2021-01-02     2    2   1.0  1.0   NaN  NaN   NaN  NaN   NaN  NaN
# 2021-01-03     3    3   2.0  2.0   1.0  1.0   NaN  NaN   NaN  NaN
# 2021-01-04     4    4   3.0  3.0   2.0  2.0   1.0  1.0   NaN  NaN
# 2021-01-05     5    5   4.0  4.0   3.0  3.0   2.0  2.0   1.0  1.0
# 2021-01-06     6    6   5.0  5.0   4.0  4.0   3.0  3.0   2.0  2.0

Then collapse the High and Low columns into respective lists:

df.High = pd.Series(windows.filter(like='High').values.tolist(), index=df.index)
df.Low = pd.Series(windows.filter(like='Low').values.tolist(), index=df.index)

#                                  High                        Low
# Date                                                            
# 2021-01-01  [1.0, nan, nan, nan, nan]  [1.0, nan, nan, nan, nan]
# 2021-01-02  [2.0, 1.0, nan, nan, nan]  [2.0, 1.0, nan, nan, nan]
# 2021-01-03  [3.0, 2.0, 1.0, nan, nan]  [3.0, 2.0, 1.0, nan, nan]
# 2021-01-04  [4.0, 3.0, 2.0, 1.0, nan]  [4.0, 3.0, 2.0, 1.0, nan]
# 2021-01-05  [5.0, 4.0, 3.0, 2.0, 1.0]  [5.0, 4.0, 3.0, 2.0, 1.0]
# 2021-01-06  [6.0, 5.0, 4.0, 3.0, 2.0]  [6.0, 5.0, 4.0, 3.0, 2.0]

And if you want to empty out the nan lists, check for isnan() with applymap():

df = df.applymap(lambda x: [] if np.isnan(x).any() else x)

#                                  High                        Low
# Date                                                            
# 2021-01-01                         []                         []
# 2021-01-02                         []                         []
# 2021-01-03                         []                         []
# 2021-01-04                         []                         []
# 2021-01-05  [5.0, 4.0, 3.0, 2.0, 1.0]  [5.0, 4.0, 3.0, 2.0, 1.0]
# 2021-01-06  [6.0, 5.0, 4.0, 3.0, 2.0]  [6.0, 5.0, 4.0, 3.0, 2.0]

You can use DataFrame.tail, in this case df.tail(5), to pull data from last 5 rows!

Related