Using Redshift JDBC Jar in spark submit , doesnt let the spark connect to Glue catalog

Viewed 317

I am able to connect to Redshift and Glue from spark job on EMR separately but when I want to query redshift tables and glue catalog at the same time in the spark job. When I use jdbc driver given by EMR(/usr/share/aws/redshift/jdbc/RedshiftJDBC41.jar) in spark submit. I am running into this error --- pyspark.sql.utils.AnalysisException: u'java.lang.ExceptionInInitializerError: null;'

After the HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. step.( it is supposed to talk to Glue after this step)

My spark submit looks like below

spark-submit --packages com.databricks:spark-redshift_2.10:0.5.2 --jars /usr/share/aws/redshift/jdbc/RedshiftJDBC41.jar,spark-redshift_2.10-2.0.0.jar,minimal-json-0.9.4.jar t.py

sample code

from pyspark.sql import SparkSession

spark = SparkSession.builder\
            .config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory")\
            .enableHiveSupport()\
            .getOrCreate();

query_k = """
          select * from table limit 10  """

s3TempDir = "s3://path"
jdbcURL = "jdbc:redshift:XXX!"

df = spark.read.format("com.databricks.spark.redshift")\
        .option("url", jdbcURL)\
        .option("tempdir", s3TempDir)\
        .option("query", query_k).load()
df.show()
0 Answers
Related