Apache spark 2.3 over Apache HBase 2.0

user_5141914 0
Viewed 1073

Need to add spark connector over HBase where

Spark version: 2.3.1
HBase Version: 2.0.0

Getting Bellow Exception:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.deploy.SparkHadoopUtil.getCurrentUserCredentials()Lorg/apache/hadoop/security/Credentials;
    at org.apache.hadoop.hbase.spark.HBaseContext.<init>(HBaseContext.scala:68)
    at org.apache.hadoop.hbase.spark.JavaHBaseContext.<init>(JavaHBaseContext.scala:46)
    at com.cloud.databaseroot.hbase.spark.JavaHBaseBulkPutExample.main(JavaHBaseBulkPutExample.java:60)

Snap from pom.xml:

    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-spark</artifactId>
        <version>2.0.0-alpha4</version>
        <exclusions>
            <exclusion>
                <artifactId>jackson-module-scala_2.10</artifactId>
                <groupId>com.fasterxml.jackson.module</groupId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency> 
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <version>4.1.17.Final</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_2.11</artifactId>
        <version>2.8.8</version>
    </dependency>

Let me know where am I getting wrong.

apache-spark hbase
1 Answers

It seems that hbase-spark version 2.0.0-alpha4 is not compatible with Spark 2.3.1. SparkHadoopUtil.getCurrentUserCredentials method is available in Spark version <= 2.2. Either downgrade Spark or build hbase-spark with Spark 2.3.1 which may require some code changes in it.

user_86174630
Related