Submitting spring boot application jar to spark-submit

user_5733798 0
Viewed 7108

I am pretty new to spark, and i am trying to try out spark submit. I created an application in spring boot , used mvn package to create a jar . But when i am trying to submit the jar to spark-submit , it is not able to find the Main class . But the main class is present in the jar.

 spark-submit --class com.dip.sparkapp.SparkappApplication --master local target/sparkapp-0.0.1-SNAPSHOT.jar
spring-boot apache-spark spring
4 Answers

I have found that simply skipping the class name from spark-submit works, i.e. --class com.dip.sparkapp.SparkappApplication

user_48028400

If your are using gradle this will work for shadow plugin:

import com.github.jengelman.gradle.plugins.shadow.transformers.*

...

plugins {
    id 'com.github.johnrengelman.shadow' version '2.0.4'
}

...

shadowJar {
    zip64 true
    mergeServiceFiles()
    append 'META-INF/spring.handlers'
    append 'META-INF/spring.schemas'
    append 'META-INF/spring.tooling'
    transform(PropertiesFileTransformer) {
        paths = ['META-INF/spring.factories' ]
        mergeStrategy = "append"
    }
}

Taken from here: https://github.com/spring-projects/spring-boot/issues/1828#issuecomment-607352468

user_81650960

This works for me

<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
    <manifestEntries>
        <Main-Class>packagename.classname</Main-Class>
    </manifestEntries>
</transformer>
user_20398380
Related