According to the Dataflow documentation, there are 2 ways to run a Dataflow pipeline/job in the cloud:
- Synchronous (
BlockingDataflowPipelineRunner) - Asynchronous (
DataflowPipelineRunner)
When using synchronous, it will of course block and return the PipelineResult when the job finishes. That's fine.
But what if I want to check the status of the job while it's running asynchronously? I'd like to:
- Run the job asynchronously i.e. use
DataflowPipelineRunner - Poll the job for its status, and check how long it's been running
- If it's been running too long then raise an alaram
Polling the status of the job is possible with the gcloud command line tool. However, how can this be done programatically via the Java SDK (REST api)? I cannot see any methods that would facilitate this.
The motivation for this is that we had a job hang, and we did not realise until 11 hours after it was kicked off. It usually takes 2 hours.