I have a MS SQL table which contains a list of files that are stored within an ADLS gen2 account. All files have the same schema and structure.
I have concatenated the results of the table into a string.
mystring = ""
for index, row in files.iterrows():
mystring += "'"+ row["path"] + "',"
mystring = mystring[:-1]
print(mystring)
OUTPUT
'abfss://[file]@[container].dfs.core.windows.net/ARCHIVE/2021/08/26/003156/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/08/30/002554/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/09/02/003115/file.parquet'
I am now attempting to pass the string using
sdf = spark.read.parquet(mystring)
however I am getting the error
IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: 'abfss://[file]@[container].dfs.core.windows.net/ARCHIVE/2021/08/26/003156/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/08/30/002554/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/09/02/003115/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/09/24/003516/file.parquet','abfss:/[file]@[container].dfs.core.windows.net/ARCHIVE/2021/10/07/002659/file.parquet'
When I manually copy and past mystring into read.parquet the code executes with no errors.
Maybe I'm going down a rabbit hole but some feedback would be much appreciated