Google Cloud Run or Cloud Function for one-time-task-job

user_1643003 0
Viewed 848

We have startup script that will create monitoring alerts using Google APIs.

For this we created a postman collection where we have bundled all these calls and then we wish to run a shell script where we will use service account json file and call the postman collection

export GOOGLE_APPLICATION_CREDENTIALS=/home/abc/sac-5f63.json
export TOKEN=$(gcloud auth application-default print-access-token)
echo $TOKEN
echo "Running postman collection start"
newman run /home/abc/API.postman_collection.json

We created a docker file for this task with required dependencies and gcloud sdk as base image.

Now, We also want to give provision to execute this one time job in Google Cloud to users. For this we are considering options such as Cloud Run, Cloud Function, App Engine. But, all these services expose an endpoint and then it has to be called and the the service will perform the action.

Our requirement is a one time activity so we don't want to keep a service running all the time. Also, this is not a node js, java, go application that these services support.

It is more like a one time job task, that should get trigger do it's job and then stop.

Does google cloud support such a use case. Basic hello world docker is an example of this kind of requirement that we have - https://hub.docker.com/_/hello-world

We tried cloud run but it expects a portNo to be present with out which it fails

Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

Please share some thoughts.

thanks, Aakash

google-cloud-run google-cloud-functions docker google-cloud-platform
4 Answers

I recommend you to use GKE Autopilot. You build your container and you run it, as pod job in your cluster. No need to expose a port or a webserver.

The first cluster is free on your billing account. You pay only for the CPU time and the Memory time of your running pod.

A dirty workaround is to use Cloud Build to run your container also, it's serverless, nothing to provision. It's not done for, it's a hack, but it works!

user_113725930

What is the "event" that you have that you want to trigger your script?

The services that you list are often triggered by some user-triggered HTTP method call but that's not necessarily the case and you can use e.g. Eventarc to trigger a Cloud Run services and various events|triggers for Cloud Functions.

It may be that Cloud Scheduler addresses your needs. This provides a cron mechanism for jobs.

NOTE IIUC, you're postman script makes multiple REST API calls against Cloud Platform services. This is one way to address the problem but, as you've found, it is aided by incorporating the Cloud SDK (gcloud) to obtain credentials and hopefully (!?) includes some error handling if any of the calls fail. You may wish to consider leveraging one of Google's SDKs instead of scripting the REST APIs directly. The SDKs incorporate auth, logging, provide abstractions and encourage better error handling.

user_6092900

The hello-world container is not listening at any port port. That explains the error you got. Your container, must be listening for incoming HTTP requests as stated in the Container runtime contract. By default, Cloud Run requests are sent to 8080, but you can configure Cloud Run to send requests to the port of your choice.

So to be able to use Cloud Run as a solution, add a web server to your container.

Finally, keep in mind that the max timeout with cloud run is 1h.

user_85699280

Because your service may not support by cloud functions (check functions-framework) or app engine standard natively, and also want the instances to scale down to zero, I think Cloud Run is for you.

The question becomes "how to invoke the Cloud Run service"

You can invoke either by events (firebase, gcs, pubsub...) or HTTP requests. If you deploy the service as a HTTP service, you should listen to PORT environment, no matter what your program language (even in shell).

user_68248310
Related