I am able to connect to Redshift and Glue from spark job on EMR separately but when I want to query redshift tables and glue catalog at the same time in the spark job. When I use jdbc driver given by EMR(/usr/share/aws/redshift/jdbc/RedshiftJDBC41.jar) in spark submit. I am running into this error --- pyspark.sql.utils.AnalysisException: u'java.lang.ExceptionInInitializerError: null;'
After the HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. step.( it is supposed to talk to Glue after this step)
My spark submit looks like below
spark-submit --packages com.databricks:spark-redshift_2.10:0.5.2 --jars /usr/share/aws/redshift/jdbc/RedshiftJDBC41.jar,spark-redshift_2.10-2.0.0.jar,minimal-json-0.9.4.jar t.py
sample code
from pyspark.sql import SparkSession
spark = SparkSession.builder\
.config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory")\
.enableHiveSupport()\
.getOrCreate();
query_k = """
select * from table limit 10 """
s3TempDir = "s3://path"
jdbcURL = "jdbc:redshift:XXX!"
df = spark.read.format("com.databricks.spark.redshift")\
.option("url", jdbcURL)\
.option("tempdir", s3TempDir)\
.option("query", query_k).load()
df.show()