PySpark - How to read BooleanType using schema, when values are "t" and "f"

Viewed 1587

I defined a Schema using StructType, for reading a dataframe of data that I have in Redsfhit. The table has 350+ columns, many of which are configured as Boolean.

After Unloading the data from this table, I'm trying to read the data using the Schema I created. But every BooleanType column that I'm expecting to read, has "f" / "t" values. This causes a parsing exception. Exception is:

java.lang.IllegalArgumentException: For input string: "f"
at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)
at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:30)
at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:270)
at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:125)
at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:94)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:167)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:166)
....

Is there a way to overcome this issue? I prefer not to define those columns as StringType and then casting each and every one of them to BooleanType. Hoped there might be a way to modify the parseBoolean function.

*A less preferred solution can also come from the Redshift side, that the unload will give the boolean columns with a valid values for the Boolean parsing. But again, I don't want to get into "Case When" statements for every single boolean column.

Thanks in advance

1 Answers

When you do select <fields> FROM * ... you can use case when <field>='f' then True else False end construction, but you have to apply it to every column you want to be boolean. The other way is to load everything to dataframe, run map function or udf and apply new schema

Related