When using spark-hbase, I got a ClassNotFoundException: org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter

user_16875977 0
Viewed 477

Thanks for your help!

When I try to read from Hbase, I got Exception!

I'm try to set --jars and set spark.sparkContext.addJar("./hbase-spark-1.0.0.jar"),but it doesn't work;

And I also I try to keep hbase and sbt as the same version, it's doesn't work too;

my sbt code:

      "org.apache.hbase.connectors.spark" % "hbase-spark" % "1.0.0",
      "org.apache.hbase" % "hbase" % "2.1.0" pomOnly(),
      "org.apache.hbase" % "hbase-client" % "2.1.0"

my scala code:

val df = sql.read.format("org.apache.hadoop.hbase.spark")
    .option("hbase.columns.mapping",
      "name STRING :key, " +
        "email STRING c:email, " +
        "birthDate STRING p:birthDate, " +
        "height FLOAT p:height"
    )
    .option("hbase.table", "person")
    .load()
  df.createOrReplaceTempView("personView")

  val results = sql.sql("SELECT * FROM personView WHERE name = 'alice'")
  results.show()

my exception:

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.spark.SparkSQLPushDownFilter

my spark-submit:

spark-submit \
--master yarn \
--driver-memory 3G \
--class cn.run.HbaseTest \
--num-executors 2 \
--executor-memory 3G \
--executor-cores 2 \
--conf spark.ui.port=4399 \
--conf spark.driver.maxResultSize=0 \
--jars ./hbase-spark-1.0.0.jar \
SparkFM_2.11-2.4.3.jar \
apache-spark hbase scala
1 Answers

The HBase Spark connector uses server-side filters and thus requires that you add several JAR files to the class path of each HBase region server:

  • hbase-spark-<connector.version>.jar
  • hbase-spark-protocol-shaded-<connector.version>.jar
  • scala-library-<scala.version>.jar

Here, <connector.version> is the version of the HBase Spark connector, which in your case is 1.0.0. <scala.version> is the version of the Scala run-time library, which in your case is 2.11.something. You can pull the library from the local Maven cache. Look under

$HOME/.m2/repository/org/scala-lang/scala-library/<scala.version>/

In my HBase deployments, I usually create a site-lib directory in $HBASE_HOME and put the three files there, then set HBASE_CLASSPATH in $HBASE_HOME/conf/hbase-env.sh. For example, using the snapshot version of the connector compiled for Spark 3.x:

export HBASE_CLASSPATH=$HBASE_HOME/site-lib/hbase-spark-1.0.1-SNAPSHOT.jar:$HBASE_HOME/site-lib/hbase-spark-protocol-shaded-1.0.1-SNAPSHOT.jar:$HBASE_HOME/site-lib/scala-library-2.12.12.jar

You still need to add the two JAR files (hbase-spark and hbase-spark-protocol-shaded) to your spark-submit command (or build a fat JAR) for the client-side components to work.

Another solution is to switch off filter pushdown by setting the hbase.spark.pushdown.columnfilter option to false:

val df = sql.read.format("org.apache.hadoop.hbase.spark")
    .option("hbase.columns.mapping", "...")
    .option("hbase.spark.pushdown.columnfilter", false)
    .option("hbase.table", "person")
    .load()

Note that this will prevent column filter operations such as .filter("height > 1.0") from running on the region servers and may result in unnecessary data transfers, therefore is not recommended unless you have no control over the HBase deployment.

user_13744370
Related