pyspark.sql can query local Delta Lake, but fails in case of remote Delta Lake

user_1860314 0
Viewed 47

Setup

Apache Spark v3.2.2 was installed on Ubuntu 22.04.1 LTS (x64). Standalone mode. Plus, openjdk 11.0.16 (x64), Python v3.10.4 (x64) and PySpark (Python package) v3.2.2 were available. I started PySpark with arguments:

pyspark --packages io.delta:delta-core_2.12:2.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

Note the Delta Lake feature.

I created a Delta Lake table named loans_delta_method_2, whose data are stored at /home/ubuntu/spark-warehouse/loans_delta_method_2/. Verified:

ls /home/ubuntu/spark-warehouse/loans_delta_method_2/

# _delta_log                                                           part-00001-1fb10e5a-c354-438a-bed9-21ebb9942adb-c000.snappy.parquet
# part-00000-081b47cd-b692-4fd8-8ec1-bc359121e360-c000.snappy.parquet  part-00003-be1da90f-6b1b-457f-9a21-6101da27618f-c000.snappy.parquet
# part-00000-67fadb6e-b823-4503-9b0f-ac771ac5df7f-c000.snappy.parquet  part-00007-a90cde32-d3ec-48b6-97e7-c63ec2ebc1e9-c000.snappy.parquet

With the Python code snippet below, executed in Python Shell (not PySpark Shell), I could query the table:

from pyspark.sql import SparkSession

warehouse_location = "/home/ubuntu/spark-warehouse"
spark = SparkSession.builder.\
    master("local[*]").\
    appName("TestApp").\
    enableHiveSupport().\
    config("spark.sql.warehouse.dir", warehouse_location).\
    config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension").\
    config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog").\
    config("spark.jars.packages", "io.delta:delta-core_2.12:2.0.0").\
    getOrCreate()
spark.catalog.listTables("default")

# [Table(name='loans_delta_method_2', database='default', description=None, tableType='MANAGED', isTemporary=False)]

Problem

In another Windows 10 machine (x64), openjdk 11.0.16.1 (Temurin) (x64), Python 3.10.6 (x64) and PySpark (Python package) v3.2.2 were installed. Same Python code snippet like above, except the master URL being different:

from pyspark.sql import SparkSession

warehouse_location = "/home/ubuntu/spark-warehouse"
spark = SparkSession.builder.\
    master("spark://10.5.129.21:7077").\
    appName("TestApp").\
    enableHiveSupport().\
    config("spark.sql.warehouse.dir", warehouse_location).\
    config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension").\
    config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog").\
    config("spark.jars.packages", "io.delta:delta-core_2.12:2.0.0").\
    getOrCreate()
spark.catalog.listTables("default")

# []

But no tables were returned! In addition, if changing the warehouse_location to some incorrect path, still no tables returned! (no crash! no error reported!)

The firewall of both machines was disabled. Both machines were in the same subnet. The binding address of Spark was already changed to "0.0.0.0".

Guess

There may be some magical way to configure the warehouse_location, e.g. file:///home/ubuntu/spark-warehouse, file://home/ubuntu/spark-warehouse, or //home//ubuntu//spark-warehouse (unfortunately, nothing worked).

Spark may have got confused between absolute path and relative path.

Any advice is appreciated.

delta-lake pyspark apache-spark-sql apache-spark
1 Answers

As discussed in another Stackoverflow post, if accessing Master node with local[*], one can load local files, with prefix file:///.... But as soon as Master/Executors is remote relatively to Driver, the files to be loaded must be accessible to both Driver and Executors. Typically, such files are stored in HDFS.

In my case, the files can be seen by Executors (Ubuntu machine), but not by Driver (Windows machine), which led to Spark unable to load the files. Apache Spark is the backbone of Delta Lake; so Delta Lake has the same restriction as Spark.

Caveat: I haven't tested the case where all Driver, Executors, Master are Linux, and the files are cloned into each machine, e.g. /etc/my_files. In this case, not sure if despite Master accessed with spark://some_host:some_port, Spark can access the files or not.

user_18603140
Related