Spark, Alternative to Fat Jar

Spark launcher can used if spark job has to be launched through some application with the help of Spark launcher you can configure your jar patah and no need to create fat.jar for runing application.

With a fat-jar you have to have Java installed and launching the Spark application requires executing java -jar [your-fat-jar-here]. It's hard to automate it if you want to, say, launch the application from a web application.

With SparkLauncher you're given the option of launching a Spark application from another application, e.g. the web application above. It is just much easier.

import org.apache.spark.launcher.SparkLauncher

SparkLauncher extends App {

val spark = new SparkLauncher()
.setSparkHome("/home/knoldus/spark-1.4.0-bin-hadoop2.6")
.setAppResource("/home/knoldus/spark_launcher-assembly-1.0.jar")
.setMainClass("SparkApp")
.setMaster("local[*]")
.launch();
spark.waitFor();

}

Code: https://github.com/phalodi/Spark-launcher

Here

• setSparkHome(“/home/knoldus/spark-1.4.0-bin-hadoop2.6”) is use to set spark home which is use internally to call spark submit.

• .setAppResource(“/home/knoldus/spark_launcher-assembly-1.0.jar”) is use to specify jar of our spark application.

• .setMainClass(“SparkApp”) the entry point of the spark program i.e driver program.

• .setMaster(“local[*]”) set the address of master where its start here now we run it on loacal machine.

• .launch() is simply start our spark application

What are the benefits of SparkLauncher vs java -jar fat-jar?

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SparkLauncher.html

https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/launcher/SparkLauncher.html

http://henningpetersen.com/post/22/running-apache-spark-jobs-from-applications

For example on Cloudera's clusters, there is some set of libraries already available on all nodes which will be available on classpath for drivers, executors. Those are e.g. spark-core, spark-hive, hadoop etc

Versions are grouped by Cloudera, so e.g. you have spark-core-cdh5.9.0 where cdh5.9.0 suffix means that all libraries with that suffix were actually verified by Cloudera to be working together properly. Only thing you should do is to use libraries with the same group suffix and you shouldn't have any classpath conflicts.

What that allows is: set dependencies configured in an app as Maven's scope provided, so they will not be part of fat jar, but resolved from classpath on nodes.

You dind't write what kind of cluster you have, but maybe you you can use similliar approach.

maven shade plugin may be used to create fat jar which additionally allows to set libraries you want to include ina jar, and those not in a list are not included.

I think something similiar is described in this answer Spark, Alternative to Fat Jar but using S3 as dependency storage.

HubSpot has a (partial) solution: SlimFast. You can find an explanation here http://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-fat-jars and you can find the code here https://github.com/HubSpot/SlimFast

Effectively it stores all the jars it'll ever need on s3, so when it builds it does it without packaging the jars, but when it needs to run it gets them from s3. So you're builds are quick, and downloads don't take long.

I think if this also had the ability to shade the jar's paths on upload, in order to avoid conflicts, then it would be a perfect solution.

The fat jar indeed take a lot of time to create. I was able to optimize a little bit by removing the dependencies which were not required at runtime. But it is really a pain.