I defined a Schema using StructType, for reading a dataframe of data that I have in Redsfhit. The table has 350+ columns, many of which are configured as Boolean.
After Unloading the data from this table, I'm trying to read the data using the Schema I created. But every BooleanType column that I'm expecting to read, has "f" / "t" values. This causes a parsing exception. Exception is:
java.lang.IllegalArgumentException: For input string: "f"
at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)
at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:30)
at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:270)
at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:125)
at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:94)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:167)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:166)
....
Is there a way to overcome this issue? I prefer not to define those columns as StringType and then casting each and every one of them to BooleanType. Hoped there might be a way to modify the parseBoolean function.
*A less preferred solution can also come from the Redshift side, that the unload will give the boolean columns with a valid values for the Boolean parsing. But again, I don't want to get into "Case When" statements for every single boolean column.
Thanks in advance