I have an IoT setup that includes 47 sensors. I am storing the data locally into sqlite3 because the cloud is not an option. I can write the data as it comes but reading data between date ranges takes long.
query = f'''SELECT id,{column},MONTH(Date) As Month
FROM Batch1
WHERE (Date between '21-Jun-2022 00:00:00.000000' and '25-Jun-2022 23:50:41.203898')
;
'''
df = pd.read_sql(query,
engine,
columns = [column],
index_col="Date",
parse_dates={"Date" : "%d-%b-%Y %H:%M:%S.%f"})
The database has 61 columns of type FLOAT and I am trying to get ~10 million rows.