Apache Livy 0.7.0 Failed to create Interactive session

user_7025246 0
Viewed 590

While creating a new session using apache Livy 0.7.0 I am getting below error. I am also using zeppelin notebook(livy interpreter) to create the session.

Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 11.0.11
Spark 3.0.2 zeppelin 0.9.0

Any idea why I am getting the error?

I have already checked that we have livy-repl_2.11-0.7.1-incubating.jar in the classpath and the JAR already have the class it is not able to find.

$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class

Error Log

    21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)


21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
        at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411)
livy spark3 apache-zeppelin
1 Answers

Spark 3.0.x came with version of scala 2.12. You will need to be build with livy with Spark 3.0.x using scal 2.12 to solve this issue. Then you need to adjust your livy.conf Here is the article on how to rebuild your livy using maven (How to rebuild apache Livy with scala 2.12)

Good luck. I ran into the same issue and was able to solve with above steps.

user_172970840
Related