Mixed Schema CSV Import Pyspark

user_11517047 0
Viewed 155

I have a folder of CSV files that I want to read into a dataframe. The problem is that While all contain the set of columns I need, some of them also contain other columns. So for every CSV in the folder I want to only read in the common set of columns that I need.

For example:

Sheet 1 contains the columns:

Column 1, Column 2, Column 3, X

Sheet 2 contains the columns:

Column 1, Column 2, Column 3

I only need Column 1, Column 2, and Column 3. Is it possible to take care of that on read or do I need to read them in separately and then select the appropriate columns and append them together.

pyspark apache-spark-sql python-3.x python
1 Answers

Try with for loop over all files in the directory and get only the required columns read from the file.

Example:

#files path list
file_lst=['<path1>','<path2>']

from pyspark.sql.functions import *
from pyspark.sql.types import *

#define schema for the required columns
schema = StructType([StructField("column1",StringType(),True),StructField("column2",StringType(),True)])

#create an empty dataframe
df=spark.createDataFrame([],schema)

#loop through files with reading header from the file then select only req cols
#union all dataframes

for i in file_lst:
    tmp_df=spark.read.option("header","true").csv(i).select("column1","column2")
    df=df.unionAll(tmp_df)

#display results
df.show()

In case if your files in the directory have column1,column2,column3..etc(required columns) in specific order over all the files then you can try as below:

spark.read.option("header","true").csv("<directory>").select("column1","column2","column3").show()
user_76326950
Related