I'm fairly new to the google cloud platform where I'm trying to set up a python script to do a daily data pull from an external API and store the data in BigQuery. I've got the script setup and, after a lot of research & asking questions on StackOverflow, I managed to deploy it on google cloud run as a flask app.
Essentially, that means that I created two files - main.py & Dockerfile. The main.py file looks like this:
# Global libraries
import os
from flask import Flask
# Local libraries
import data_push
app = Flask(__name__)
@app.route('/')
def data():
return data_push.daily_data_push()
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
and the Dockerfile looks like this:
# Use the official lightweight Python image.
# https://hub.docker.com/_/python
FROM python:3.6-slim
# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
# Install production dependencies.
RUN pip install Flask gunicorn
# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 app:app
where data_push.daily_data_push() is the function I need to schedule in order to pull data from the API and store it in BigQuery. Now, based on what I'm reading, I need to first set up the app so that it can talk to pub/sub, and then I can use cloud scheduler to set up a daily schedule to pull the data.
I'm a bit lost on how to set up the script to talk to pub/sub. I'm going through google's documentation on it but I still haven't figured it out. Problem is that I'm a data analyst and both flask & pub/sub are way out my league in terms of expertise.
I'd appreciate your help with either showing me how the script should be set up to talk to pub/sub OR by pointing me towards a source that has the answer.