Why does Dataproc on GKE not support Docker images for jobs submission but uses raw code for python/R and jar for others. Shouldn't providing container image be the ideal way to submit job to GKE be it ordinary job or Dataproc job? Is this in plan or a work in progress?
Ref: https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke
As mentioned in the question comments, the main intent behind current iteration of Dataproc on GKE is to provide the same user experience as Dataproc on GCE. That's why it supports the same API which doesn't use Docker images to execute Spark jobs and there are no near-term plans to add support for this. That said, in the future Dataproc may decide to support some kind of containers as a first class way of submitting jobs (in addition to jars, etc).
