Data streaming from raspberry pi CSV file to Bigquerry table

Sana 0
Viewed 30

I have some CSV files generated by raspberry pi that needs to be pushed into bigquery tables. Currently, we have a python script using bigquery.LoadJobConfig for batch upload and I run it manually. The goal is to have streaming data(or every 15 minutes) in a simple way.

I explored different solutions:

  1. Using airflow to run the python script (high complexity and maintenance)
  2. Dataflow (I am not familiar with it but if it does the job I will use it)
  3. Scheduling pipeline to run the script through GitLab CI (cron syntax: */15 * * * * )

Could you please help me and suggest to me the best way to push CSV files into bigquery tables in real-time or every 15 minutes?

google-cloud-dataflow airflow google-cloud-platform google-bigquery
2 Answers

Good news, you have many options! Perhaps the easiest would be to automate the python script that you have currently, since it does what you need. Assuming you are running it manually on a local machine, you could upload it to a lightweight VM on Google Cloud, the use CRON on the VM to automate the running of it, I used used this approach in the past and it worked well.

Another option would be to deploy your Python code to a Google Cloud Function, a way to let GCP run the code without you having to worry about maintaining the backend resource.

Find out more about Cloud Functions here: https://cloud.google.com/functions

A third option, depending on where your .csv files are being generated, perhaps you could use the BigQuery Data Transfer service to handle the imports into BigQuery.

More on that here: https://cloud.google.com/bigquery/docs/dts-introduction

Good luck!

Ben P0

Adding to @Ben's answer, you can also implement Cloud Composer to orchestrate this workflow. It is built on Apache Airflow and you can use Airflow-native tools, such as the powerful Airflow web interface and command-line tools, Airflow scheduler etc without worrying about your infrastructure and maintenance.

You can implement DAGs to

More on Cloud Composer

Sakshi Gatyan0
Related