I read post on quora which tell that Spark Thrift server is related to Apache Thrift which is d binary communication protocol. Spark Thrift server is the interface to Hive, but how does Spark Thrift server use Apache Thrift for communication with Hive via binary protocol/rpc?
2 Answers
You can bring up the Spark thrift Server on AWS EMR using the following command - sudo /usr/lib/spark/sbin/start-thriftserver.sh --master yarn-client
On EMR, the default port for Spark thrift Server is 10001
While using the beeline for spark use the following command on EMR
/usr/lib/spark/bin/beeline -u 'jdbc:hive2://:10001/default' -e "show databases;"
By Default Hive thrift Server is always up and running on EMR but not the Spark thrift Server
You can also connect any application to the Spark thrift Server using ODBC/JDBC and can also monitor the query on EMR Cluster by Clicking the Application Master link for "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2" job on Yarn Resource Manager:8088 on EMR
Related