Merge dataframes and fill in blank values based on start/end dates

Viewed 368

I have pandas dataframe (df) with start and end dates for certain value (in this case 'currency'). I need to merge it with another dataframe (tbl) and fill in blank currency rows based on start/end dates from the first DF. NULL means no end date - so everything going forward. In this case everything after 01/11/2020 is USD. Data coming from SQL therefore NULL.

df = pd.DataFrame(data={
        'port': 'PortA'
        'currency': ['USD', 'CAD', 'EUR', 'USD'],
        'start_date': ['01/01/2020', '01/04/2020', '01/06/2020', '01/11/2020'],
        'end_date': ['01/04/2020', '01/06/2020', '01/11/2020', '01/15/2020']
        })
df[['start_date', 'end_date']] = df[['start_date', 'end_date']].apply(pd.to_datetime, errors='ignore')
tbl = pd.DataFrame(data={
        'port': 'PortA',
        'as_of_date': [x for x in pd.date_range(start='01/01/2020', end='01/15/2020')]
        })

This is what I need

df_merge is final look I need. Secondary question - what if I don't have second dataframe (tbl) to merge with. Is there an easy way to 'unstack' the existing df so it looks the same way as df_merge?

Thank you.

2 Answers
  1. First create an as_of_date column with pd.date_range in your dataframe that is a list of the dates between the start and end date per row with lambda x: (drop duplicates and keep last as well).
  2. Explode the dataframe on the as_of_date in prepraration for a merge on date and port.
  3. Simply, merge the dataframes (Per your secondary question, you can simply exclude this step).

Step 1: Create Date Range Column

df['as_of_date'] = df.apply(lambda x: list(pd.date_range(x['start_date'], x['end_date'], freq='d')), axis=1)
df
Out[1]: 
    port currency start_date   end_date  \
0  PortA      USD 2020-01-01 2020-01-04   
1  PortA      CAD 2020-01-04 2020-01-06   
2  PortA      EUR 2020-01-06 2020-01-11   
3  PortA      USD 2020-01-11 2020-01-15   

                                          as_of_date  
0  [2020-01-01 00:00:00, 2020-01-02 00:00:00, 202...  
1  [2020-01-04 00:00:00, 2020-01-05 00:00:00, 202...  
2  [2020-01-06 00:00:00, 2020-01-07 00:00:00, 202...  
3  [2020-01-11 00:00:00, 2020-01-12 00:00:00, 202...  

Step 2: Explode the dataframe and drop duplicates

df = df.explode('as_of_date').drop_duplicates('as_of_date', keep='last')
df
Out[2]: 
    port currency start_date   end_date as_of_date
0  PortA      USD 2020-01-01 2020-01-04 2020-01-01
0  PortA      USD 2020-01-01 2020-01-04 2020-01-02
0  PortA      USD 2020-01-01 2020-01-04 2020-01-03
1  PortA      CAD 2020-01-04 2020-01-06 2020-01-04
1  PortA      CAD 2020-01-04 2020-01-06 2020-01-05
2  PortA      EUR 2020-01-06 2020-01-11 2020-01-06
2  PortA      EUR 2020-01-06 2020-01-11 2020-01-07
2  PortA      EUR 2020-01-06 2020-01-11 2020-01-08
2  PortA      EUR 2020-01-06 2020-01-11 2020-01-09
2  PortA      EUR 2020-01-06 2020-01-11 2020-01-10
3  PortA      USD 2020-01-11 2020-01-15 2020-01-11
3  PortA      USD 2020-01-11 2020-01-15 2020-01-12
3  PortA      USD 2020-01-11 2020-01-15 2020-01-13
3  PortA      USD 2020-01-11 2020-01-15 2020-01-14
3  PortA      USD 2020-01-11 2020-01-15 2020-01-15

Step 3: Merge the two dataframes (Per your secondary question - you can ignore this step if you don't have a tbl dataframe. Instead just run df = df[['port', 'as_of_date', 'currency']] to keep and reorder the columns that you need:

df_merge = pd.merge(df[['port', 'currency', 'as_of_date']], tbl, how='left', on=['as_of_date', 'port'])
df_merge
Out[3]: 
     port currency as_of_date
0   PortA      USD 2020-01-01
1   PortA      USD 2020-01-02
2   PortA      USD 2020-01-03
3   PortA      CAD 2020-01-04
4   PortA      CAD 2020-01-05
5   PortA      EUR 2020-01-06
6   PortA      EUR 2020-01-07
7   PortA      EUR 2020-01-08
8   PortA      EUR 2020-01-09
9   PortA      EUR 2020-01-10
10  PortA      USD 2020-01-11
11  PortA      USD 2020-01-12
12  PortA      USD 2020-01-13
13  PortA      USD 2020-01-14
14  PortA      USD 2020-01-15

Full Code:

df = pd.DataFrame(data={
        'port': ['PortA','PortA','PortA','PortA'],
        'currency': ['USD', 'CAD', 'EUR', 'USD'],
        'start_date': ['01/01/2020', '01/04/2020', '01/06/2020', '01/11/2020'],
        'end_date': ['01/04/2020', '01/06/2020', '01/11/2020', '01/15/2020']
        })
df[['start_date', 'end_date']] = df[['start_date', 'end_date']].apply(pd.to_datetime, errors='ignore')
tbl = pd.DataFrame(data={
        'port': 'PortA',
        'as_of_date': [x for x in pd.date_range(start='01/01/2020', end='01/15/2020')]
        })
df['as_of_date'] = df.apply(lambda x: list(pd.date_range(x['start_date'], x['end_date'], freq='d')), axis=1)
df = df.explode('as_of_date').drop_duplicates('as_of_date', keep='last')
df_merge = pd.merge(df[['port', 'currency', 'as_of_date']], tbl, how='left', on=['as_of_date', 'port'])
df_merge

You can do it this way using pd.date_range, explode and merge:

df_dates = df.assign(dates=[pd.date_range(i, j + pd.Timedelta(days=-1), freq='D') 
                             for i, j in zip(df['start_date'], df['end_date'])])\
             .explode('dates')

bl.merge(df_dates[['port','dates','currency']], 
         left_on=['port', 'as_of_date'], 
         right_on=['port', 'dates'])

Output:

     port as_of_date      dates currency
0   PortA 2020-01-01 2020-01-01      USD
1   PortA 2020-01-02 2020-01-02      USD
2   PortA 2020-01-03 2020-01-03      USD
3   PortA 2020-01-04 2020-01-04      CAD
4   PortA 2020-01-05 2020-01-05      CAD
5   PortA 2020-01-06 2020-01-06      EUR
6   PortA 2020-01-07 2020-01-07      EUR
7   PortA 2020-01-08 2020-01-08      EUR
8   PortA 2020-01-09 2020-01-09      EUR
9   PortA 2020-01-10 2020-01-10      EUR
10  PortA 2020-01-11 2020-01-11      USD
11  PortA 2020-01-12 2020-01-12      USD
12  PortA 2020-01-13 2020-01-13      USD
13  PortA 2020-01-14 2020-01-14      USD

Note: pd.Timedelta(days=-1) to handle duplicate dates on multiple rows. End_date overlaps with start_date next row.

Alter code to fix last date:

d = pd.Timedelta(days=-1)
l = pd.date_range #To shorten typing
df_dates = df.assign(dates=[l(i, j + d) if j != df.iloc[-1, df.columns.get_loc('end_date')] 
                               else l(i, j) for i, j in zip(df['start_date'], df['end_date'])])\
             .explode('dates')

print(tbl.merge(df_dates[['port','dates','currency']], left_on=['port', 'as_of_date'], right_on=['port', 'dates']))
Related