resolve dependencies, download packages with Spark without running it

Viewed 125

Apache Spark have the ability to resolve dependencies at runtime, when submit an application, via the --packages options.

As we use Docker to package our application, I would like to embed all my app dependencies in the Docker image:

FROM spark:3.1.2

COPY ./app /opt/app

RUN cd /opt/app && spark-submit --packages XXX:XXX:XXX

PORT 4040

CMD ["/opt/app/bin/run"]

Something like that, but without running Spark ?

0 Answers
Related