Spark Catalog w/ AWS Glue: database not found

user_3066956 0
Viewed 13477

Ive created an EMR cluster with the Glue Data catalog. When I invoke the spark-shell, I am able to successfully list tables stored within a Glue database via

spark.catalog.setCurrentDatabase("test")
spark.catalog.listTables

However when I submit a job via spark-submit I get a fatal error 

ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Database 'test' does not exist.;

I am creating my SparkSession within the job being submitted via spark-submit via

SparkSession.builder.enableHiveSupport.getOrCreate
amazon-emr aws-glue apache-spark
5 Answers

I had the same issue: spark-submit will not discover the AWS Glue libraries, but spark-shell working on the master node will.

It turns out that my spark-submit job uses a fat .jar which was compiled with the standard org.apache.spark and org.apache.hive libraries. The jar libraries were being used in stead of the custom classes installed on EMR. If this is the case with you, make sure to exclude all:

'org.apache.spark:' 'org.apache.hive:' 'org.apache.hadoop:' modules from you .jar

Here is the reference I used for .Gradle: http://unethicalblogger.com/2015/07/15/gradle-goodness-excluding-depends-from-shadow.html.

Adding compileOnly keyword in front of all spark libraries fixed it.

user_85724340

Our issue was IAM permissions on the EMR cluster; make sure that the cluster IAM instance profile has full access to glue.

user_30669560

My problem ended up being that another classification configuration had been interfering with the spark-hive-site one. I deleted all others, and it finally was able to connect.

user_147973970
user_1761000
Related