`AnalysisException("Database 'delta' not found" ...` when creating delta table over DeltaTableBuilder API locally

Viewed 1182

I am creating delta tables in Databricks Runtime 9.1 LTS using the DeltaTableBuilder API in PySpark. This works fine. When running the same code locally (for a unit test) I get a strange error.

Im a setting up my local environment and the local PySpark Session as described in the Quickstart guide.

Steps to reproduce:

Setup environment as in DB Runtime 9.1: Python 3.8

pip install delta-spark==1.0.0
pip install pyspark==3.1.2

Create SparkSession:

from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip

session_builder = SparkSession.builder
session_builder = (
    session_builder.config("spark.jars.packages", "io.delta:delta-core_2.12:1.0.0")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
)

spark = configure_spark_with_delta_pip(session_builder).getOrCreate()

Try to create delta table:

from delta import DeltaTable

DeltaTable.createIfNotExists(spark) \
        .location("test_tables/test.delta") \
        .addColumn("id", "LONG") \
        .execute()

Error:

pyspark.sql.utils.AnalysisException: Database 'delta' not found

In Databricks this error does not appear. It does not require any database "delta" it just creates the delta table directory with the delta_log in it - no database involved. But ok lets create the database locally:

from delta import DeltaTable

spark.sql("CREATE DATABASE delta")

DeltaTable.createOrReplace(spark) \
    .location("test_delta") \
    .addColumn("id", "LONG") \
    .execute()

Error:

pyspark.sql.utils.AnalysisException: `delta`.`test_delta` is not a Delta table.

How can I make the behaviour locally as in Databricks?

0 Answers
Related