I am confused about some similar spark configurations...
I have surveyed the major reference links are https://spark.apache.org/docs/latest/configuration.html and https://spark.apache.org/docs/latest/running-on-yarn.html.
But I am still confused about these configurations...
Could anyone help me to figure out the main differences?
Thanks very much!!
1. spark.yarn.jars vs. spark.jars
What is the difference between spark.yarn.jars vs. spark.jars?
Which configuration is the same to --jars ?
spark.yarn.jars: List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed.
spark.jars: Comma-separated list of jars to include on the driver and executor classpaths. Globs are allowed.
2. spark.yarn.dist.archives vs. spark.yarn.archive
What is the difference between spark.yarn.dist.archives vs. spark.yarn.archive?
Which configuration is the same to --archives ?
spark.yarn.dist.archives: Comma separated list of archives to be extracted into the working directory of each executor.
spark.yarn.archive: An archive containing needed Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution.
3. spark.yarn.dist.files vs. spark.files
What is the difference between spark.yarn.dist.files vs. spark.files?
Which configuration is the same to --files ?
spark.yarn.dist.files: Comma-separated list of files to be placed in the working directory of each executor.
spark.files: Comma-separated list of files to be placed in the working directory of each executor. Globs are allowed.