I know this is kind of a generic question, but I need some starting point. I am being asked to create an ETL process to move some data from GCP over to SQL Server and I supposed to use python. Is there some python pseudo code that can accomplish this, documentations or best practices? Or what python libraries I need to install. I have no idea what I'm doing so any help will be appreciated.
Pseudo process:
- Python connects to GCP Big Query to read data.
- Data is converted to CSV or JSON
- JSON is stored in location (cloud)
- SQL Server picks up file and loads the data in staging table (may need to be in python to drop data into staging tables, not sure)
- Data gets cleaned and loaded into proper table for user consumption.