Google Cloud Data flow JDBC to BigQuery job just hanging and producing no helpful logs/error message

stella227 0
Viewed 28

I am trying to use the Google provided Dataflow template JDBC to BigQuery. I have an on-prem SQL Server that I want to connect to. I have configured all the necessary networking and firewall rules to allow the Dataflow workers to communicate with the on-prem server. I have also confirmed that the provided connection string and credentials work.

This issue is that when I run the job, it just hangs and doesn't produce any helpful log/errors. I have check the logs explorer and can't seem to find anything to help troubleshoot this.

If I let it run, eventually it hits a timeout and produces this: useless log

google-cloud-dataflow jdbc google-cloud-platform
2 Answers

I would suggest opening a support ticket, as it may be necessary to have the Job ID to see exactly what went wrong.

Such symptom can be observed when created workers can't be reached by Dataflow -- either by having the subnetwork mismatching the job region, or because of strict firewall rules.

Bruno Volpato0

Usually when I have dataflow job hanging, I open the full log query and I keep only the job id in the log query.

The log query by default is:

resource.type="dataflow_step"
resource.labels.job_id="job_id"
logName=("projects/project/logs/dataflow.googleapis.com%2Fjob-message" OR "projects/project/logs/dataflow.googleapis.com%2Flauncher")

I just change it to:

resource.labels.job_id="job_id"

This usually gives me more details about the pipeline and what's wrong. It happened with me a lot when dealing with customer containers and flex templates etc.

Idhem0
Related