How to optimize reading millions of rows and plot it in python using sqlite and matplotlib?

Viewed 29

I have an IoT setup that includes 47 sensors. I am storing the data locally into sqlite3 because the cloud is not an option. I can write the data as it comes but reading data between date ranges takes long.

query = f'''SELECT id,{column},MONTH(Date) As Month
                FROM Batch1 
                WHERE (Date between '21-Jun-2022 00:00:00.000000' and '25-Jun-2022 23:50:41.203898')
                ;
      '''

df = pd.read_sql(query,
                engine,
                columns = [column],
                index_col="Date",
                parse_dates={"Date" : "%d-%b-%Y %H:%M:%S.%f"})

The database has 61 columns of type FLOAT and I am trying to get ~10 million rows.

0 Answers
Related