Error writing delta file with pyspark - Caused by: java.lang.ClassNotFoundException: delta.DefaultSource

Viewed 761

I have a parquet file that I am trying to write to a delta table. My code is straight forward I think.

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .master("local[*]") \
    .appName('test') \
    .config("spark.jars.packages", "io.delta:delta-core_2.12:0.7.0") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .config('spark.ui.port', '4050') \
    .getOrCreate()
df = spark.read.format('parquet').load('fhvhv_tripdata_2021-01.parquet')
df.write.format('delta').save('deltafiles')

When I try to write it though I receive this error

...
Caused by: java.lang.ClassNotFoundException: delta.DefaultSource
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:65
...

I'm on: Python version 3.8.10 When I run pyspark --version I get:

      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
                        
Using Scala version 2.12.15, OpenJDK 64-Bit Server VM, 11.0.15

I am running in a Jupyter Notebook. What might be the cause of this/what else should I check?

0 Answers
Related