How to attachHandler to sparkUI

user_1996296 0
Viewed 83

I am trying to a attach handler to spark UI using this code:

 private def getSparkUI(sparkContext: SparkContext): SparkUI = {
       sparkContext.ui.getOrElse {
           throw new SparkException("Parent SparkUI to attach this tab to not found!")
       }

}

import org.apache.spark.ui.JettyUtils.createServletHandler
val ui = getSparkUI(sparkContext)
val graphHttpServlet = new GraphHttpServlet("/graph/", sessionUtils)
val handler = createServletHandler("/graph/", graphHttpServlet, "")
ui.attachHandler(handler)

this code fails to compile with error:

Error:(20, 19) Symbol 'type org.eclipse.jetty.servlet.ServletContextHandler' is missing from the classpath.
This symbol is required by 'method org.apache.spark.ui.JettyUtils.createServletHandler'.
Make sure that type ServletContextHandler is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'JettyUtils.class' was compiled against an incompatible version of org.eclipse.jetty.servlet.
val handler = createServletHandler("/graph/", graphHttpServlet, "")

I know that spark shades jetty dependencies into a new package: org.spark_project.jetty.servlet.ServletContextHandler

I also decompile the jar and check that the jar has the shaded dependencies.

But why the compiler failed to see that? How I can attach the handler?

apache-spark
1 Answers

Also, faced similar issue and figured out after few experiments. Adding shade rule for org.eclipse.jetty libraries solves this issue.

This solves below exceptions:

  • java.lang.NoSuchMethodError: 'void org.apache.spark.ui.WebUI.attachHandler(org.eclipse.jetty.servlet.ServletContextHandler)'

  • java.lang.ClassCastException: class org.sparkproject.jetty.servlet.ServletContextHandler cannot be cast to class org.eclipse.jetty.servlet.ServletContextHandler (org.sparkproject.jetty.servlet.ServletContextHandler and org.eclipse.jetty.servlet.ServletContextHandler are in unnamed module of loader 'app')

sbt rule

assemblyShadeRules in assembly := Seq(
    ShadeRule
      .rename("org.eclipse.jetty.**" -> "org.sparkproject.jetty.@1")
      .inAll
  )

Maven rule

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <shadedArtifactAttached>false</shadedArtifactAttached>
        <artifactSet>
        <includes>
            <include>org.eclipse.jetty:jetty-io</include>
            <include>org.eclipse.jetty:jetty-http</include>
            <include>org.eclipse.jetty:jetty-proxy</include>
            <include>org.eclipse.jetty:jetty-client</include>
            <include>org.eclipse.jetty:jetty-continuation</include>
            <include>org.eclipse.jetty:jetty-servlet</include>
            <include>org.eclipse.jetty:jetty-servlets</include>
            <include>org.eclipse.jetty:jetty-plus</include>
            <include>org.eclipse.jetty:jetty-security</include>
            <include>org.eclipse.jetty:jetty-util</include>
            <include>org.eclipse.jetty:jetty-server</include>
        </includes>
        </artifactSet>
        <relocations>
        <relocation>
            <pattern>org.eclipse.jetty</pattern>
            <shadedPattern>org.sparkproject.jetty</shadedPattern>
            <includes>
            <include>org.eclipse.jetty.**</include>
            </includes>
        </relocation>
        </relocations>
    </configuration>
    <executions>
        <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
        </execution>
    </executions>
    </plugin>
</plugins>
user_20940990
Related