Running on PySpark on jupyter lab while trying to using delta lake, and this error appears after running the code:
spark.sql("CREATE TABLE events USING DELTA LOCATION '/mnt/delta/events/'")
I have made sure my scala versions are aligned with respect to spark-shell and pyspark
this is pyspark version: Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 14.0.2
this is spark-shell version: Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 14.0.2
My spark session is configured as follows:`
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.jars.packages", "io.delta:delta-core_2.12:0.1.0") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.getOrCreate()
I've been on this for 24 hrs, can't find a solution.