We are trying to connect to the Glue catalog using spark on EMR 5.30.2 and applications: Spark 2.4.5.
I am able to connect to it with a small program.
import org.apache.spark.sql.SparkSession;
public class Glue {
public static void main(String[] args){
SparkSession spark = SparkSession.builder().enableHiveSupport().getOrCreate();
spark.sql.show("show databases");
}
}
This is working and showing the databases as
--------------
Databases
--------------
glue_database1
But I have to same code in a fat jar project where I have a lot of other dependencies. Somehow it is not working in that.
I am getting warnings as
022-01-12 14:32:03 Driver [Info] HiveUtils - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
2022-01-12 14:32:03 Driver [Warn] HiveConf - HiveConf of name hive.metastore.client.factory.class does not exist.
2022-01-12 14:32:03 Driver [INFO] ObjectStore - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes=“Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order”
2022-01-12 14:32:03 Driver [INFO] DataStore - The class “org.apache.hadoop.hive.metastore.model.MFieldSchema” is tagged as “embedded-only” so does not have its own datastore table.
2022-01-12 14:32:03 Driver [INFO] DataStore - The class “org.apache.hadoop.hive.metastore.model.MOrder” is tagged as “embedded-only” so does not have its own datastore table.
2022-01-12 14:32:03 Driver [INFO] DataStore - MetaStoreDirectSql - Using direct SQL, underlying DB is Derby
2022-01-12 14:32:03 Driver [INFO] ObjectStore - Initiating the ObjectStore
2022-01-12 14:32:03 Driver [INFO] ObjectStore - Version information not found in megastore. Hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
2022-01-12 14:32:03 Driver [INFO] ObjectStore - Failed to get database default, returning NoSuchObjectException
So on the same cluster when we submit a small program it's working but the fat jar containing code is not working.
We have enabled the use of glue as metastore option by setting the configuration.
[{"classification":"spark-hive-site","properties":{"hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}}]
Please note, Starting here I am adding details which are after trying to edit pom.xml to resolve the above issue. So Adding the pom file part where I added provided scope for spark hive dependencies.
compile time Spark version 2.4.3 (EMR has 2.4.5) Scala version 2.11
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<scope>provided<scope>
</dependency>
Error logs after pom change. Now it's able to reach glue.
org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.amazonaws.protocol.json.JsonErrorShapeMetadata.withExceptionUnmarshaller(Lcom/amazonaws/transform/JsonErrorUnmarshaller;)Lcom/amazonaws/protocol/json/JsonErrorShapeMetadata;;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.org$apache$spark$sql$hive$HiveSessionStateBuilder$$externalCatalog(HiveSessionStateBuilder.scala:39)
at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:90)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:90)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listDatabases(SessionCatalog.scala:247)
at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.execution.command.ShowDatabasesCommand.run(databases.scala:44)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:195)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:80)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
Question:-
- What other properties can I set to make sure that configurations are not overridden or set.
- do I need to shade dependencies.
Please Note:- I can modify and add content if needed.