I have a dataflow job that was running OK with this user agent (we only configure the Apache Beam version so the JDK environment is automatically configured)
Apache_Beam_SDK_for_Java/2.19.0(JDK_11_environment)
After upgrading the workflow, we stating using Beam 2.27 and the job was launched with the following user agent:
Apache_Beam_SDK_for_Java/2.27.0(JDK_11_environment)
and it started to fail with error:
Workflow failed. Causes: Subscription 'xxx' not found. Please supply an existing subscription.
The pub sub subscription exists, and it's the same we were using before and even the link in the job configuration properties works well and navigates to the subscription
We have previously successfully deployed the same job version that was failing in a different GCP instance, and here we saw the user agent was: Apache_Beam_SDK_for_Java/2.27.0(JRE_8_environment). Our project is configured to use Java 8, so we added the following parameter when launching the job on the failing environment
--userAgent="Apache_Beam_SDK_for_Java/2.27.0(JRE_8_environment)"
and it started to work. However we still see this on the job properties although it's working ok:
Apache_Beam_SDK_for_Java/2.27.0(JDK_11_environment)
Do you know any explanation for this and which is the better way to ensure Java 8 is used to run the workflow?
Thanks