How to add hbase-site.xml config file using spark-shell

user_5949117 0
Viewed 1571

I have the following simple code:

import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.HBaseConfiguration
val hbaseconfLog = HBaseConfiguration.create()
val connectionLog = ConnectionFactory.createConnection(hbaseconfLog)

Which I'm running on spark-shell, and I'm getting the following error:

 14:23:42 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected 
error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:30)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Many these errors actually, and a few of these every now and then:

14:23:46 WARN client.ZooKeeperRegistry: Can't retrieve clusterId from 
Zookeeper org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

Through Cloudera's VM I'm able to solve this by simply restarting the hbase-master, regionserver and thrift, but here in my company I'm not allowed to do it, I also solved it once by copying the file hbase-site.xml to spark conf directory but I can't to it either, is there a way to set the path for this specific file in the spark-shell parameters?

apache-spark hbase
1 Answers

1) make sure that your zookeeper is running

2) need to copy hbase-site.xml to /etc/spark/conf folder just like we copy hive-site.xml to /etc/spark/conf to access the Hive tables.

3) export SPARK_CLASSPATH=/a/b/c/hbase-site.xml;/d/e/f/hive-site.xml

just as described in hortonworks forum.. like this

or open spark-shell with out adding hbase-site.xml

3 commands to execute in spark-shell

val conf = HBaseConfiguration.create()
  conf.addResource(new Path("/home/spark/development/hbase/conf/hbase-site.xml"))
   conf.set(TableInputFormat.INPUT_TABLE, table_name)
user_6470530
Related