Looping the files inside the folder and load data into delta table using databricks

user_17155606 0
Viewed 392

1.I have folder inside the blob storage

conatiner--->folder--->2021-10-17---->14-10-20-->filename1.txt,filename2.txt,filename3.parquet

conatiner--->folder--->2021-10-17---->15-10-20-->filename1.parquet,filename2.parquet,filename3.parquet

i need to process all the .txt and .parquet files inside the databricks folder and load the data into delta table using databricks.Can you please help me on this issue?

Some of the .txt file don't have schema .how to define the schema and need to proess the files?

delta-lake databricks apache-spark
1 Answers

You just need to pass a wildcard expression that will match your folder structure, something like this (add more * characters if necessary):

# read
df = spark.read.parquet("base_path_of_container/folder/*/*/*.parquet")
# write data into Delta
df.write.format("delta").save("path_to_delta")
Alex_Ott_186270
Related