Dataproc job status

Question

I am running pyspark jobs on Dataproc using gcloud dataproc jobs submit pyspark command. I see that although my pyspark job errors out and doesn't complete, I receive complete status on the console. Even the Dataproc -> Jobs UI shows the job as complete.

Any pointers on how get the YARN job status out in the console?

pysparkJob:
  args:
  - ./config.yml
  fileUris:
  - gs://bucket/config/config.yml
  mainPythonFileUri: gs://bucket/scripts/job.py
reference:
  jobId:
  projectId: 
status:
  state: DONE
  stateStartTime: '2021-10-31T23:21:27.633773Z'
statusHistory:
- state: PENDING
  stateStartTime: '2021-10-31T23:20:32.268452Z'
- state: SETUP_DONE
  stateStartTime: '2021-10-31T23:20:32.294793Z'
- details: Agent reported job success
  state: RUNNING
  stateStartTime: '2021-10-31T23:20:32.546015Z'
yarnApplications:
- name: spark-load
  progress: 1.0
  state: ***FINISHED***

I get the above after running gcloud dataproc jobs submit pyspark gs://bucket/scripts/job.py --cluster=$cluster --files=$config. Apparently my job failed, how do i get the correct status in this case?

Due to this my downstream jobs are not running correctly.

Dataproc job status

0 Answers