Where to host pub sub publisher on GCP?

user_6683176 0
Viewed 241

I'm looking to create a publisher that streams and sends tweets containing a certain hashtag to a pub/sub topic.

The tweets will then be ingested with cloud dataflow and then loaded into a Big Query database.

In the following article they do something similar where the publisher is hosted on a docker image on a Google Compute Engine instance.

Can anyone recommend alternative Google Cloud resources that could host the publisher code more simply, that avoids the need to create a docker file etc? The publisher would need to run constantly. Would cloud run for e.g. be a suitable alternative?

google-cloud-pubsub google-cloud-platform python
1 Answers

There are some workarounds I can think of:

  1. A quick way to avoid containers architecture is having the on_data method inside a loop, for example, by using something like while(true) or start a Stream like explained in Create your Python script and run the code in a Compute Engine in the background with nohup python -u myscript.py. Or follow the steps described in Script on GCE to capture tweets that uses tweepy.Stream to start the streaming.

  2. You might want to reconsider the Dockerfile option since its configuration could be not so difficult, see Tweets & pipelines where there is a script that read the data and publish to PubSub, you will see that 9 lines are used for the Docker file and it is deployed in App Engine using Cloud Build. Another implementation with a Docker file that requires more steps is twitter-for-bigquery, in case it helps, you will see that there are more specific steps and more configurations.

  3. Cloud Functions is also another option, in this guide Serverless Twitter with Google Cloud you can check the Design section to know if it fits your use case.

  4. Airflow with Twitter Scraper could work for you since Cloud Composer is a managed service for Airflow and you can create an Airflow environment quickly. It uses the Twint library, check the Technical section in the link for more details.

  5. Stream Twitter Data into BigQuery with Cloud Dataprep is a workaround that put aside complex configurations. In this case the job won't run constantly but can be scheduled to run within minutes.

user_94578430
Related