When I submit a Spark job to Kubernetes, everything in the containers is run as root. Is it possible to run the job as another user?
When I submit the job in Client mode, the driver runs as the user who submitted it but the executors run as root, which might lead to file access problems when accessing files created by the executors.
Unless the full customization of the K8s Pod is supported by Spark on K8s (in particular runAsUser feature) the only ways to control it (as I see for the moment) are:
- Build docker image specifying USER in Dockerfile
- Use some advanced K8s tools/controllers, eg Argo Events
- Customize spark-submit or submit Spark Pods directly as Kubernetes Pods through K8s APIs
Hope to see some improvements coming with Spark v3.0.0 soonish though.
