Permission denied executing jar file downloaded to .Ivy folder in Kubernetes

user_2476688 0
Viewed 454

I created a dockerfile and changed User to a non-root user nobody. Locally this works perfectly. When deployed on kubernetes howerever, I get error

java.nio.file.AccessDeniedException: ./xxxxxx_2.12-2.6.3.jar

When I dug in more I realised this jar file is download after the spark dependencies used in the dockerfiles are downloaded. Therefore any permissions given to the spark folder are not present for this newly dowloaded jar file which is downloaded at runtime into /opt/spark/.ivy2/xxx which has root permissions. This causes the pod in kubernetes to fail.

I am wondering if there is way to give permissions to execute this jar file. Since it seems this is not possible in the Dockerfile. Any suggestion as to how to solve this issue ??

As proposed by @mario

ARG SPARK_OPERATOR_BASE_IMAGE_VERSION=v2.4.5
FROM xxx/alpine:3.12 as preparator

ARG SCALA_VERSION=2.12
ARG SPARK_VERSION=2.4.7
ARG HADOOP_VERSION=3.2.1
ARG AWS_SDK_VERSION=1.11.375
ARG MAVEN_VERSION=3.6.2

RUN apk add --no-cache \
        bash \
        curl && \
        mkdir /target

COPY hashes /tmp/
COPY prepare /tmp/

WORKDIR /tmp

# Download Hadoop
RUN curl -L -O https://downloads.apache.org/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz && \
    sha256sum -c hadoop-${HADOOP_VERSION}.sha256 && \
    tar -xzf hadoop-${HADOOP_VERSION}.tar.gz && \
    mv hadoop-${HADOOP_VERSION} /target/hadoop

# Download Spark
RUN curl -L -O https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
    sha512sum -c spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.sha512 && \
    tar -xzf spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
    mv spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION} /target/spark && \
# Download Spark 3.0.0 entrypoint script from GitHub, bugfixing for 2.4.7
    curl -L -O https://raw.githubusercontent.com/apache/spark/v3.0.0/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh && \
    mv entrypoint.sh /target/entrypoint.sh && \
    chmod +x /target/entrypoint.sh

# Download AWS Jars
RUN curl -L -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar && \
    sha1sum -c hadoop-aws-${HADOOP_VERSION}.jar.sha1 && \
    mv hadoop-aws-${HADOOP_VERSION}.jar /target/spark/jars/ && \
    curl -L -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar && \
    sha1sum -c aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar.sha1 && \
    mv aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar /target/spark/jars/

# Directory needed for saving built jars
RUN mkdir /target/spark/custom-jars/

#### Download Prometheus + Metric dependencies ####
# install java, maven and prometheus fat jar using maven (pom.xml)
RUN apk add --update openjdk8 && \
    curl -L -O https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
    tar -xzf apache-maven-${MAVEN_VERSION}-bin.tar.gz && export PATH=./apache-maven-${MAVEN_VERSION}/bin:$PATH && \
    mv prometheus-pom.xml pom.xml && mvn clean package && mv target/prometheusMetricLibs-jar-with-dependencies.jar /target/spark/custom-jars/

RUN \
  chown -R nobody:99 /target/spark \
  && chown -R nobody:99 /target/hadoop \
  && chmod -R ugo+rw /target/spark \
  && chmod -R ugo+rw /target/hadoop


ARG SPARK_OPERATOR_BASE_IMAGE_VERSION
FROM gcr.io/spark-operator/spark:${SPARK_OPERATOR_BASE_IMAGE_VERSION}

RUN rm -rf /opt/spark/

COPY --from=preparator /target/ /opt/

ENV SPARK_HOME=/opt/spark \
    HADOOP_HOME=/opt/hadoop

ENV HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native" \
    LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native \
    PATH=${HADOOP_HOME}/bin:${SPARK_HOME}/bin:${PATH}

COPY conf /opt/spark/conf/

RUN echo "export JAVA_HOME=${JAVA_HOME}" >> ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh && \
    echo "export JAVA_HOME=${JAVA_HOME}" > ${SPARK_HOME}/conf/spark-env.sh && \
    echo "export SPARK_DIST_CLASSPATH=\$(hadoop classpath)" >> ${SPARK_HOME}/conf/spark-env.sh


# 99 used instead of nobody because in the alpine image the Group Id of nobody is different than in the spark operator image.
RUN \
  addgroup --gid 99 nobody \
  && echo "nobody:x:99:99:nobody:/nonexistent:/usr/sbin/nologin" >> /etc/passwd \
  && usermod -a -G users nobody \
  && chmod -R ugo+rw /var/lib/

USER nobody

# if we want local storage
#  "spark.eventLog.dir": "tmp/spark-events"
RUN mkdir -p /tmp/spark-events

in my pod the jar file is implemented like this

sparkConf:
    "spark.ui.port": "4045"
    "spark.eventLog.enabled": {{ .Values.spark.eventLogEnabled | quote }}
    "spark.eventLog.dir": "xx//"
    "spark.jars.ivySettings": "/vault/secrets/xx-ivysettings.xml"
    "spark.jars.ivy": "/opt/spark/.ivy2"
    "spark.jars.packages": "xxxx_2.12:{{ .Values.appVersion }}"
    "spark.blacklist.enabled": "false"
    "spark.driver.supervise": "true"
    "spark.app.name": {{ .Values.name | quote }}
    "spark.submit.deployMode": {{ .Values.spark.deployMode | quote }}
    "spark.driver.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"
    "spark.executor.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"
dockerfile kubernetes-pod kubernetes
1 Answers

It would be easier to answer your question if you share your Dockerfile and your kubernetes Pod template yaml manifest, but in short: you can manipulate your Pod's permissions using the securityContext like in the example from the docs below:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

For debugging purposes you can start from setting allowPrivilegeEscalation to true or runAsUser: 0 (root) but keep in mind this is not the solution that can be used in production. Running containers as root is generally a bad idea and in most cases it can be avoided.

Therefore any permissions given to the spark folder are not present for this newly dowloaded jar file which is downloaded at runtime into /opt/spark/.ivy2/xxx which has root permissions.

Is it really necessary for it to have root permissions ? Most likely it can be fixed in your Dockerfile.

user_117141140
Related