Composer Dataflow job error using DataflowTempleOperator

user_14128359 0
Viewed 254

I am trying to run standard GCS_Text to Bigquery standard template job using Composer and get http error shown below. I could successfully run and create the job manually in Dataflow Am I passing the parameter correctly?

ERROR - <HttpError 400 when requesting https://dataflow.googleapis.com/v1b3/projects/XXXXXX/locations/xyz/templates:launch?gcsPath=gs%3A%2F%2Fdataflow-templates%2Flatest%2FGCS_Text_to_BigQuery&alt=json returned "The template parameters are invalid.". Details: "[{'@type': 'type.googleapis.com/google.dataflow.v1beta3.InvalidTemplateParameters', 'parameterViolations': [{'parameter': 'javascriptTextTransformFunctionName', 'description': 'Missing required parameter'}, {'parameter': 'TransformFunctionName', 'description': 'Unrecognized parameter'}, {'parameter': 'serviceAccount', 'description': 'Unrecognized parameter'}, {'parameter': 'project', 'description': 'Unrecognized parameter'}, {'parameter': 'subnetwork', 'description': 'Unrecognized parameter'}]}]"
orchestration google-cloud-composer airflow dataflow google-cloud-platform
1 Answers

The error that you are receiving is pretty clear. Some parameters (project, subnetwork, service_account, TransformFunctionName) are not recognised and javascriptTextTransformFunctionName is missing.

Check again the documentation examples on how to launch jobs with the DataflowTemplateOperator as well as the docs for the operator itself.

In a few words, TransformFunctionName should be renamed to javascriptTextTransformFunctionName and project, subnetwork and service_account should be removed from the parameters dict and specified in dataflow_default_options dict instead.

user_79031590
Related