Is is possible to read csv or parquet file using same code

user_227986 0
Viewed 551

Does any one know if it is possible to read either a csv or parquet file into a spark using the same code.

My use case here is that in production, I will be using large parquet files, but for unit tests, I would like to use CSVs. I'm using something like the following code:

spark.read().schema(schema).load(path);

This fails in the CSV case with the following exception:

file.csv is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [78, 9, 78, 10]

I suspect that spark defaults to parquet, and this won't work, but I wanted to check first.

pyspark apache-spark-sql apache-spark
1 Answers

spark.read.schema(schema).load(path); Without mentioning format() then spark defaults to reading parquet file.

If you are reading csv file then we need to mention .format("csv") to let spark know we are trying to read a csv file or else spark will read the file as parquet.

spark.read.format("csv").schema(schema).load(path)

user_76326950
Related