I am trying to create a batch ETL using a local python script and GCP.
I have built a python script that generates a random json file containing some fake data fields as a "fake data generator"...I am working on a project that will eventually use a more complex algorithm to generate a json, but for designing the framework of my ETL... I am just using a placeholder in the meantime.
The goal is to set up a system in GCP that would allow the scheduled running of this script (on my local repo in visual studio), and then the retrieving of the json file it generates. Then I need to append this file to a data table in BQ.
What would be the best way to do this operation? It is important to note that the request to run the python script on a schedule needs to come from GCP, or else I would just use a cronjobs system on my local machine.
I have tried learning how to create a scheduled query in Google Big Query, although I'm not very familiar with how this could work, and am still learning how to use Big Query and GCP.
All advice is welcome, thanks!!!