How to verify Dataflow Runner v2 is being used?

user_1168364 0
Viewed 214

Dataflow docs show how to enable Dataflow Runner v2

run your job with the following flag: --experiments=use_runner_v2

My system is complex so it's hard to know whether the flag is being set properly. Is there a way to see in the gcp UI whether v2 is being used?

google-cloud-dataflow google-cloud-platform
1 Answers

For now, the best way is for you to check your experiments. If the use_runner_v2 experiment is activated, then Dataflow is using V2.

Unfortunately, experiments are hidden by default (see fix to un-hide). AS A WORKAROUND you can create your own PipelineOption interface with a List<String> getExperiments() method - and this will make sure that they show up in the UI.

enter image description here

A few more details, and using logs to figure this out:

In Runner V2, we use a 'runner harness' that is written in C++, and not in Java as usual. In the logs you can look for logs like:

  • In streaming: Executing: /opt/google/dataflow/streaming_harness_main
  • In Batch Executing: /opt/google/dataflow/harness_main

As opposed to a java -jar command.

In general, you can look for harness or harness-startup logs in the logging of your job. If there aren't harness logs, then your job is NOT Runner V2.

For what it's worth, it should be easier to do this. I'll request to add a feature to log it more clearly / show it in the UI.

user_12553560
Related