I am trying to match two dataframes using SQL in Python. I would like to match the most recent observation prior to a corresponding date on the other dataframe. So far, I have the following code, where I match all observations before the corresponding date (datadate).
In subsequent steps I create a variable that is the difference in days between filing_date and datadate, and then keep the observation with the least amount of days. However, this feels clunky and can present issues. Is there a way for me to do this within the SQL code? I would like to maintain all observations from datatable1. I included my current code and created tables below showing what I would like to do. Thanks for the help!
import sqlite3
sqlconn = sqlite3.connect(':memory:')
datatable1.to_sql('datatable1',sqlconn, index=False)
datatable2.to_sql('datatable2', sqlconn, index=False)
qry = '''
SELECT DISTINCT
a.*, b.filing_date, b.value
FROM datatable1 AS A
LEFT JOIN datatable2 AS B
ON a.id = b.id AND b.filing_date < a.datadate
'''
compq_01 = pd.read_sql_query(qry, sqlconn)
datatable1:
| id | datadate |
|---|---|
| 01 | 3/31/2015 |
| 01 | 6/30/2015 |
| 02 | 9/30/2015 |
| 03 | 12/31/2015 |
| 04 | 12/31/2015 |
| 05 | 12/31/2015 |
datatable2:
| id | filing_date | value |
|---|---|---|
| 01 | 2/15/2015 | 18 |
| 01 | 3/25/2015 | 12 |
| 01 | 6/28/2015 | 55 |
| 02 | 7/18/2015 | 67 |
| 02 | 8/25/2015 | 14 |
| 03 | 11/07/2015 | 18 |
What I want the resulting table to look like:
| id | datadate | filing_date | amount |
|---|---|---|---|
| 01 | 3/31/2015 | 3/25/2015 | 12 |
| 01 | 6/30/2015 | 6/28/2015 | 55 |
| 02 | 9/30/2015 | 8/25/2015 | 14 |
| 03 | 12/31/2015 | 11/07/2015 | 18 |
| 04 | 12/31/2015 | NaN | NaN |
| 05 | 12/31/2015 | NaN | NaN |