Mapping between 2 different Dataframes with in between dates

user_12842243 0
Viewed 100

I am trying to map values from 1 Dataframe to another using the code below, it does the job but since the dataframes huge (2-3 million rows) it takes a lot of time to map these values. Is there a better and a more efficient way to write this code that can speed up the process?

for i in df.index:
    fc2 = STP[(STP['SKU'] == df.iloc[i]['SKU']) & (STP['EFF_DATE'] <= df.iloc[i]['Date']) & ((STP['END_DTTIME'] >= df.iloc[i]['Date']))]['PRICE_TYPE']
    if len(fc2.index) >0:
        df['Price_Type'].iloc[i] = fc2.values.tolist()[0]

Any help would be highly appreciated.

df = {'Date': ['2020-10-24', '2020-10-24', '2020-10-20', '2020-10-24', '2020-10-24'], 'SKU': [125,3245,165158,1651651,16561]}
STP = {'SKU': [125,3245,3245,165158,165158,1651651,16561], 'EDD_DATE': ['2020-10-14','2020-10-14','2020-10-24','2020-09-28','2020-06-30','2020-10-14','2020-10-14'], 'END_DATE': ['2020-10-25','2020-10-23','2020-10-31','2020-10-31','2020-09-27','2020-10-25','2020-10-25'], 'PRICE_TYPE': ['abc', 'abc', 'bca', 'abc', 'bbc', 'abc', 'bca']}

final = {'Date': ['2020-10-24', '2020-10-24', '2020-10-20', '2020-10-24', '2020-10-24'], 'SKU': [125,3245,165158,1651651,16561], 'Price_Type': ['abc', 'bca', 'abc', 'abc', 'bca']}
pandas dataframe mapping datetime
1 Answers

this is pretty simple in SQL, but I'm not aware (althought I'm sure there are) of a decent panda's approach to do this, I would wait for other answers but this is one way you could approach this without using python loops.

We will need to

  1. create a product of both dataframes (cross join) based on SKU.
  2. apply a boolean based on the dates.
  3. re-merge based on SKU and Date to get the Price Type.

Setup

df = pd.DataFrame(df)
stp = pd.DataFrame(STP)
df['Date'] = pd.to_datetime(df['Date'])
stp['EDD_DATE'] = pd.to_datetime(stp['EDD_DATE'])
stp['END_DATE'] = pd.to_datetime(stp['END_DATE'])

Edit handling duplicate left keys.

df1 = pd.merge(stp,df,on='SKU',how='outer')
m = (df1['Date'] >= df1['EDD_DATE']) & (df1['Date'] <= df1['END_DATE'])

df_new = pd.merge(df,df1[m].loc[:,['Date','SKU','PRICE_TYPE']]
         ,on=['SKU','Date'],how='left')


print(df)

    Date      SKU PRICE_TYPE
0 2020-10-24      125        abc
1 2020-10-24     3245        bca
2 2020-10-20   165158        abc
3 2020-10-24  1651651        abc
4 2020-10-24    16561        bca

Naturally, this is more expensive as your joining two fact tables but this will be much better and cleaner mind you, than a loop.

user_93751020
Related