Create one dataframe from multi csv files with different headers in Spark

Viewed 960

In Spark, with Pyspark, i want to create one dataframe (where the path is actually a folder in S3), which contains multi csv files with common columns and different columns. To say it more easily, i want only one dataframe from multiple csv files with different headers.

I can have a file with this header "raw_id, title, civility", and another file with this header "raw_id, first_name, civility".

This is my code in python 3 :

df = spark.read.load(
    s3_bucket + 'data/contacts/normalized' + '/*/*/*/*',
    format = 'csv',
    delimiter = '|',
    encoding = 'utf-8',
    header = 'true',
    quote = ''
)

This is an example of file_1.csv :

|raw_id|title|civility|
|1     |M    |male    |

And an example of file2.csv :

|raw_id|first_name|civility|
|2     |Tom       |male    |

The result i expect in my dataframe is :

|raw_id|first_name|title|civility|
|1     |          |M    |male    |
|2     |Tom       |     |male    |

But, what is happening is that i have all united columns but the data is not in the right place after the first file. Do you know how to do this ?

Thank you very much by advance.

1 Answers

You need to load each of them in a different dataframe and join them together on the raw_id column.

Related