Streaming a large file from a GCP Bucket exceeds 1-hour max Cloud Run limit

Aali Rehman 0
Viewed 52

I have a large file that contains a few million rows in a GCP Bucket which I stream from a Cloud Run instance and push to a pub/sub queue.

Cloud Run has a max timeout of 1-hour which is not enough time to stream the whole file, is there an alternative option or better way of architecting/handling streaming large amounts of data from a GCP bucket?

Side thought: Does the npm package @google-cloud/storage have the ability to seek to a specific part of the file? Or could I increase the number of CPUs to do something such as process the file using multiple threads using a language like Go?

google-cloud-storage google-cloud-run google-cloud-platform node.js
1 Answers

You can create a Dataflow batch job :

  • Dataflow is a runner on Apache Beam. Beam is a model and open source library : https://beam.apache.org/documentation/
  • Dataflow is fully managed on GCP and serverless
  • It’s adapted for huge files and long running jobs
  • Dataflow uses autoscaling and creates 1 or more compute engine VM on the fly to execute a job (depending on your data and performance of the job)
  • Beam is proposed with Java, GO and Python sdk
  • Many IO are proposed natively to interact with GCP resources (Cloud Storage, Cloud Pub Sub…)

In this case, your job is a batch job because it reads data from a bounded source (file from GCS).

Mazlum Tosun0
Related