How to rebuild apache Livy with scala 2.12

user_14013690 0
Viewed 2355

I'm using Spark 3.1.1 which uses Scala 2.12, and the pre-built Livy downloaded from here uses Scala 2.11 (one could find the folder named repl_2.11-jars/ after unzip).

Referred to the comment made by Aliaksandr Sasnouskikh, Livy needs to be rebuilt or it'll throw error {'msg': 'requirement failed: Cannot find Livy REPL jars.'} even in POST Session.

In the README.md, it mentioned:

By default Livy is built against Apache Spark 2.4.5

If I'd like to rebuild Livy, how could I change the spark version that it is built with?

Thanks in advance.

livy apache-spark
4 Answers

You can rebuild Livy passing spark-3.0 profile in maven to create a custom build for spark 3, for example:

git clone https://github.com/apache/incubator-livy.git && \
cd incubator-livy && \
mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true

This profile is defined in pom.xml, the default one installs Spark 3.0.0. You can change it to use different spark version.

<profile>
      <id>spark-3.0</id>
      <activation>
        <property>
          <name>spark-3.0</name>
        </property>
      </activation>
      <properties>
        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
        <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
        <spark.version>${spark.scala-2.11.version}</spark.version>
        <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
        <netty.spark-2.11.version>4.1.47.Final</netty.spark-2.11.version>
        <netty.version>${netty.spark-2.11.version}</netty.version>
        <java.version>1.8</java.version>
        <py4j.version>0.10.9</py4j.version>
        <json4s.spark-2.11.version>3.5.3</json4s.spark-2.11.version>
        <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
        <json4s.version>${json4s.spark-2.11.version}</json4s.version>
        <spark.bin.download.url>
          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
        </spark.bin.download.url>
        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
      </properties>
    </profile>

As long as I know, Livy supports spark 3.0.x. But worth testing with 3.1.1, and let us know :)

user_111221240

I tried to build Livy for Spark 3.1.1 based on rmakoto's answer and it worked! I tinkered a lot and I couldn't exactly remember what I edited in the pom.xml so I am just going to attach my gist link here.

I also had to edit the python-api/pom.xml file to use Python3 to build since there's some syntax error issues when building with the default pom.xml file. Here's the pom.xml gist for python-api.

After that just build with

mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true
user_143265260

Based on @gamberooni 's changes (but using 3.1.2 instead of 3.1.1 for the Spark version and Hadoop 3.2.0 instead of 3.2.1), this is the diff:

diff --git a/pom.xml b/pom.xml
index d2e535a..5c28ee6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -79,12 +79,12 @@
 
   <properties>
     <asynchttpclient.version>2.10.1</asynchttpclient.version>
-    <hadoop.version>2.7.3</hadoop.version>
+    <hadoop.version>3.2.0</hadoop.version>
     <hadoop.scope>compile</hadoop.scope>
     <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
-    <spark.scala-2.12.version>2.4.5</spark.scala-2.12.version>
-    <spark.version>${spark.scala-2.11.version}</spark.version>
-    <hive.version>3.0.0</hive.version>
+    <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
+    <spark.version>${spark.scala-2.12.version}</spark.version>
+    <hive.version>3.1.2</hive.version>
     <commons-codec.version>1.9</commons-codec.version>
     <httpclient.version>4.5.3</httpclient.version>
     <httpcore.version>4.4.4</httpcore.version>
@@ -1060,7 +1060,7 @@
         </property>
       </activation>
       <properties>
-        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
+        <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
         <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
         <spark.version>${spark.scala-2.11.version}</spark.version>
         <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
@@ -1072,9 +1072,9 @@
         <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
         <json4s.version>${json4s.spark-2.11.version}</json4s.version>
         <spark.bin.download.url>
-          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
+          https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
         </spark.bin.download.url>
-        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
+        <spark.bin.name>spark-3.1.2-bin-hadoop3.2</spark.bin.name>
       </properties>
     </profile>
 
diff --git a/python-api/pom.xml b/python-api/pom.xml
index 8e5cdab..a8fb042 100644
--- a/python-api/pom.xml
+++ b/python-api/pom.xml
@@ -46,7 +46,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-            <executable>python</executable>
+            <executable>python3</executable>
               <arguments>
                 <argument>setup.py</argument>
                 <argument>sdist</argument>
@@ -60,7 +60,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-              <executable>python</executable>
+              <executable>python3</executable>
               <skip>${skipTests}</skip>
               <arguments>
                 <argument>setup.py</argument>
Herrmann_105750

I will just put additional option out there: Lighter. This is Spark REST API hugely inspired by Livy and it is compatible and built with latest Spark versions. More info.

Disclaimer: I took part in developing it.

user_9588990
Related