View Dataproc Job driver log in Cloud Logging

user_16955896 0
Viewed 187

I cannot query Dataproc job-level logs for my job, though cluster-level logs are available.

Steps to reproduce:

  1. create a cluster using custom image based on 1.4, set up cluster properties as described here. Validation:
gcloud dataproc clusters describe --project <my-project> --region us-west1 <my-cluster> |grep file-backed-output

   dataproc:jobs.file-backed-output.enable: 'true'

gcloud dataproc clusters describe --project <my-project> --region us-west1 <my-cluster> |grep stackdrive

   dataproc:dataproc.logging.stackdriver.enable: 'true'

   dataproc:dataproc.logging.stackdriver.job.driver.enable: 'true'

  1. start Dataproc job on that cluster, run commands:
gcloud logging read resource.type=cloud_dataproc_job --project <my-project> //returns 0 rows

gcloud logging read resource.type=cloud_dataproc_job --project <my-project> resource.labels.region=us-west1 resource.labels.job_id=<my-job>

ERROR: (gcloud.logging.read) unrecognized arguments:
 resource.labels.region=us-west1
 resource.labels.job_id=<my-job>
 To search the help text of gcloud commands, run:
 gcloud help -- SEARCH_TERMS

When I look into the cluster master node, I can see the file /etc/google-fluentd/config.d/dataproc-job-driver.conf with a row path /var/log/google-dataproc-job/**/driver.log, and expected log file is available under that location.

google-cloud-dataproc
1 Answers

Did your job complete before you ran the logging commands?

I was unable to reproduce the issue with the project wide read command, but was able to reproduce the problem with the job specific command. According to https://cloud.google.com/sdk/gcloud/reference/logging/read, the arguments all need to be enclosed in quotation marks - so your command should instead look like:

gcloud logging read "resource.type=cloud_dataproc_job resource.labels.region=us-west1 resource.labels.job_id=<my-job>" --project=<my-project>

I will file a bug to ensure the documentation is correct to reflect the correct quotation marks.

user_169617980
Related