Pyspark cannot load from pathlib object

Question

Python Version 3.7.5
Spark Version 3.0
Databricks Runtime 7.3

I'm currently working with paths in my datalake file system.

This is

p = dbutils.fs.ls('dbfs:/databricks-datasets/nyctaxi')
print(p)
 [FileInfo(path='dbfs:/databricks-datasets/nyctaxi/readme_nyctaxi.txt', name='readme_nyctaxi.txt', size=916),
 FileInfo(path='dbfs:/databricks-datasets/nyctaxi/reference/', name='reference/', size=0),
 FileInfo(path='dbfs:/databricks-datasets/nyctaxi/taxizone/', name='taxizone/', size=0),
 FileInfo(path='dbfs:/databricks-datasets/nyctaxi/tripdata/', name='tripdata/', size=0)]

now, to turn this into a valid Pathlib Posix Object I pass this through a function

def create_valid_path(paths):
    return Path('/dbfs').joinpath(*[part for part in Path(paths).parts[1:]])

the output for tripdata is

PosixPath('/dbfs/databricks-datasets/nyctaxi/tripdata')

now, if I want to read this into a sparkdata frame after collecting a subset of csvs into a list.

from pyspark.sql.functions import * 
df = spark.read.format('csv').load(paths)

this returns

AttributeError: 'PosixPath' object has no attribute '_get_object_id'

now, the only way I can get this to work is to manually prepend the path dbfs:/.. and return each item to a string however it's necessary to use Pathlib to do some basic I/O operations. am I missing something simple or can Pyspark simply not read a pathlib object?

e.g

trip_paths_str = [str(Path('dbfs:').joinpath(*part.parts[2:])) for part in trip_paths]

print(trip_paths_str)

['dbfs:/databricks-datasets/nyctaxi/tripdata/fhv/fhv_tripdata_2015-01.csv.gz',
 'dbfs:/databricks-datasets/nyctaxi/tripdata/fhv/fhv_tripdata_2015-02.csv.gz'...]

user_6933993 · Answer

What about doing this then instead?

from pyspark.sql.functions import * 
import os

def db_list_files(file_path):
  file_list = [file.path for file in dbutils.fs.ls(file_path) if os.path.basename(file.path)]
  return file_list

files = db_list_files('dbfs:/FileStore/tables/')
 
df = spark.read.format('text').load(files)
df.show()

Pyspark cannot load from pathlib object

1 Answers