Question:
According to the description of the configuration spark.archives(https://spark.apache.org/docs/3.1.2/configuration.html), it extracts the archives into the working directory of each executor. Does this mean that no matter which deployment mode (client or cluster) is used, the archive files will not be extracted to the working directory of driver ?
Why I need the archives to be extracted to driver:
When using pyspark, I use conda to create a virtual environment for python. And I use conda pack to pack the virtual environment to a *.tar.gz file. So I can use the spark.archives to extract the archive file to the executors (under client mode in K8s). But the archive file will not be extracted to the working directory of driver. Hence, the python environment of driver and the executor is different.
Is there any way to extract the archive file to both the driver and executors under client mode?