How to get spark logs from history server by submission id

Viewed 79

I try to get logs from a spark job from sparks history server, which is submitted by the Spark API. The successful response of an http call to the spark submit API returns an "submissionId" which can be used to get the status of the job (source):

curl http://192.168.133.128:6066/v1/submissions/status/driver-20180429125849-0001

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "FINISHED",
  "serverSparkVersion" : "2.0.1",
  "submissionId" : "driver-20180429125849-0001",
  "success" : true,
  "workerHostPort" : "192.168.133.128:38451",
  "workerId" : "worker-20180429124356-192.168.133.128-38451"
}

Now I want to get the complete log from spark history server given the submission ID. Sparks REST API documentation says that they need applications or job IDs to get a particular job.

Do you know how to get these logs, given the submissionId or WorkerId only?

0 Answers
Related