Is there a way to update a Dataflow job using the gcloud command?

user_3285672 0
Viewed 1442

I am trying to write a script to automate the deployment of a Java Dataflow job. The script creates a template and then uses the command 

gcloud dataflow jobs run my-job --gcs-location=gs://my_bucket/template

The issue is, I want to update the job if the job already exists and it's running. I can do the update if I run the job via maven, but I need to do this via gcloud so I can have a service account for deployment and another one for running the job. I tried different things (adding --parameters update to the command line), but I always get an error. Is there a way to update a Dataflow job exclusively via gcloud dataflow jobs run?

google-cloud-dataflow gcloud
1 Answers

Referring to the official documentation, which describes gcloud beta dataflow jobs - a group of subcommands for working with Dataflow jobs, there is no possibility to use gcloud for update the job.

As for now, the Apache Beam SDKs provide a way to update an ongoing streaming job on the Dataflow managed service with new pipeline code, you can find more information here. Another way of updating an existing Dataflow job is by using REST API, where you can find Java example.

Additionally, please follow Feature Request regarding recreating job with gcloud.

user_70900160
Related