I want to read multiple parquet files from Azure blob storage through databricks but problem will be the schema. If I use inferSchema as True then it will take out schema from first file it will read. Is there any way to infer schema after reading a number of files or after reading a definite volume of data. We don't want to use mergeSchema as True.
1 Answers
Is there any way to infer schema after reading a number of files or after reading a definite volume of data.
AFAIK, There might not be any such methods.
You can try the below approach by which I am able to get the desired result.
First get the parquet file path which has a greater number of columns from the files list. Now, get the schema of that particular file and enforce this schema to all files.
Please go through a sample demonstration below:
These are my parquet files in the Blob storage in which the xyz.parquet has one extra column than others.
After Mounting, Get the list of file paths and find the parquet which has a greater number of columns.
Now, get the schema of this parquet file
Enforce this custom schema to multiple parquet files, so that we can get the dataframe with desired schema.
You can see that we got the extra column and for the files which does not have that columns the value will be null.
My Source Code:
#File paths list
fileinfo=dbutils.fs.ls("dbfs:/mnt/blob1/myfiles/")
paths=[i[0] for i in fileinfo]
print(paths)
#Finding the required file path
col_num=[]
for file_path in paths:
df=spark.read.parquet(file_path)
col_num.append(len(df.columns))
i=col_num.index(max(col_num))
our_path=paths[i]
print(our_path)
#Schema
myschema=spark.read.parquet(our_path).schema
print(myschema)
#Enforcing the schema to all files
result_df = spark.read.format("parquet").schema(myschema).load("dbfs:/mnt/blob1/myfiles/")
display(result_df)
Related