`path can be only string, list or RDD` error with spark read csv from sftp with Paramiko

user_6273496 0
Viewed 488

I'm dealing with CSV files hosted in an SFTP. I can access them using pd.read_csv() but this is not working with spark.read.csv(). I'd like to understand why.

ssh.connect(host_name, username=username, password=password, pkey=pk)
ftp_client = ssh.open_sftp()

# reading csv
path = 'some_path'
files = ftp_client.listdir('{}'.format(path))
for file in files:
    with ftp_client.open('{}/{}'.format(path, file)) as f:
        # this is not working
        # df = spark.read.csv(f)

        # this is working
        df = pd.read_csv(f)

        
print(df)

When running this command with spark read CSV method I have following error path can be only string, list or RDD

pyspark azure-databricks databricks
1 Answers

The problem occurs because you're trying to pass an instance of some custom class to Spark, and it doesn't know about that class. Solution would be something like this - read as Pandas, and convert into Spark DataFrame using spark.createDataFrame (not tested, but should work):

all_df = None
for file in files:
    with ftp_client.open('{}/{}'.format(path, file)) as f:
        pandas_df = pd.read_csv(f)
        df = spark.createDataFrame(pandas_df)
        if all_df is not None:
          all_df = all_df.union(df)
        else:
          all_df = df
Alex_Ott_186270
Related