How to retrieve the jobId of job submitted via Dataproc from within the Spark job

user_11624077 0
Viewed 47

I want to get the jobId of the Spark job which is running from within the Spark Context. Does Dataproc stores this info in Spark Context?

spark-streaming dataproc google-cloud-dataproc apache-spark
2 Answers

Dataproc job ID is not available directly within SparkContext. You might need to call Dataproc Jobs API to list all jobs associated with the cluster and find it out.

user_4042640

With a bit of patience, I found this solution (I'm using Dataproc image 2.0-debian10, Spark 3.1.2, Scala 2.12.14).

In short, I noticed that you can get the Dataproc job ID by following these steps

  1. acces the underlying SparkConf from the SparkContext
  2. get the value of the spark.yarn.tags configuration
  3. extract the id from such value with a little bit of string processing

I wrote this implicit class that extracts the ID as an optional value

implicit class RichSparkConf(protected val conf: SparkConf) {

  def dataprocJobId: Option[String] = {

    for {
      yarnTags <- conf.getOption(RichSparkConf.SparkYarnTags)
      dataProcJobIdTag <- yarnTags.split(",").find(_.startsWith(RichSparkConf.DataprocJobIdPrefix))
    } yield dataProcJobIdTag.substring(RichSparkConf.DataprocJobIdPrefix.length)
  }
}

object RichSparkConf {

  val DataprocJobIdPrefix = "dataproc_job_"
  val SparkYarnTags = "spark.yarn.tags"
}

As a rule of thumb, I usually try to take a look at all SparkConf key-value pairs and see if I can get something out of it that fits my requirement. Hope it helps ;)

user_143364770
Related