Hi I have a simple web from streamlit where it's more like a form and the data is saved to csv before being push to bigquery and is deployed in heroku. Currently it's storing the data in csv and I created a button that the admin clicks to push the data to bigquery. However this seems impractical.
I tried doing this
from threading import Thread
def main funct():
xxxx
def bq_push_data():
xxx
if __name__ == '__main__':
Thread(target = main_funct()).start()
Thread(target = bq_push_data()).start()
#----------------------------------------------
From the sample above doing any action in the main funct triggers the bq_push_data. Which makes the app runs heavy. Ideally I would want to trigger the bq_push_data function to run every 10 mins regardless of any actions in main_funct.
I also tried using worker and web in heroku to:
worker: ping main website every 10 mins and keep open mainweb: run_bq_function if within certain time period
This also does not seem to work.
What is the best way of running push csv to bigquery in the background without impacting the main_funct? Or is there another approach without using a database, so that I can push current csv data to bigquery in the background without user having to wait for it.