Dataflow jobs failing and showing no logs

Luiz 0
Viewed 17

I created pipelines in Dataflow using the standard template JDBC to BigQuery and there are a few jobs that are unexpectedly failing and not showing any logs.

The thing is, when a job fails because of the resources, the job needed more vCPUs than was avaliable in the region or the memory was not enough for example, these kind of errors are displayed in the logs, as you can see below.

enter image description here

But some jobs just fail with no logs and the resources are sufficient.

enter image description here

Does anyone know how to find the logs in this case?

google-cloud-logging google-cloud-dataflow
1 Answers

Change the severity of the logs. If you choose Default, you should see more logs. For how the job page looks like for that failed job, I would say you are probably going to need to have a look at the worker logs as well.

Depending on the error, the Diagnostics tab may have some summarized info of what kind error has made the job fail.

Israel Herraiz0
Related