Pyspark - Creating a delta table while enableHiveSupport()

Viewed 440

I'm creating a delta table on a EMR (6.2) using the following code:

try:

        self.spark.sql(f'''
            CREATE TABLE default.features_scd
                (`{entity_key}` {entity_value_type}, `{CURRENT}` BOOLEAN, 
                `{EFFECTIVE_TIMESTAMP}` TIMESTAMP, `{END_TIMESTAMP}` TIMESTAMP, `date` DATE)
                USING DELTA
                PARTITIONED BY (DATE)
                LOCATION 's3://mybucket/some/path'
            ''')
except IllegalArgumentException as e:

        self.logger.error('got an illegal argument exception')
        pass

I have enableHiveSupport() on the spark session.

I'm getting the warning:

WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider delta. Persisting data source table default.features_scd into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.

And the exception:

IllegalArgumentException: Can not create a Path from an empty string

Basically the table is being created well and I'm able to get my goal, but I wish to not have to try except and pass on that error.

If I run the same code without the enableHiveSupport() it runs smoothly. But I need the hive support in the same session for creating/updating a hive table.

Is there a way to prevent this exception?

0 Answers
Related