spark write: CSV data source does not support null data type

user_59704 0
Viewed 7655

I have an error in my code. The code is dumping some data into Redshift database.

After some investigation I found an easy way to reproduce it in spark console.

This is working fine:

scala> Seq("France", "Germany").toDF.agg(avg(lit(null))).write.csv("1.csv")
scala>

But if I replace avg with max I got an error "CSV data source does not support null data type."

scala> Seq("France", "Germany").toDF.agg(max(lit(null))).write.csv("2.csv")
java.lang.UnsupportedOperationException: CSV data source does not support null data type.

What's wrong with max ?

apache-spark-sql apache-spark scala
1 Answers

The error is correct as AVG returns the DOUBLE datatype

Seq("France", "Germany").toDF.agg(avg(lit(null)).alias("col1")).printSchema

enter image description here

where as MAX returns the type as null

Seq("France", "Germany").toDF.agg(max(lit(null)).alias("col1")).printSchema

enter image description here

so while you are writing the dataframe having MAX its throwing the error, if you want to save the dataframe with the max explicitely convert it into another type

Seq("France", "Germany").toDF.agg(max(lit(null)).alias("col1").cast(DoubleType)).write.csv("path")
user_95876550
Related