How to escape minus sign in scala case class

user_5439546 0
Viewed 138

I have a parquet file, which has - minus sign in its column name (i.e. student-name). I'm trying to use case class as an Encoder while reading the parquet file like below:

case class Student (student-name : String, student_age : String)

object abcd {

   val student_details = spark.read.parquet('/path/to/parquet-file').as[Student]

 }

But the problem is that minus sign - in the case class throwing errors. I have tried enclosing with backtick (`) it didn't help. please find the error list below.

Error without backtick

 case class Student (student-name : String, student_age : String)

:expected
Wrong top statement declaration

Error with backtick

case class Student (`student-name` : String, student_age : String)

cannot resolve '`student$minusname`' given input columns:[student-name,student_age]

Any help would be greatly appreciated!

Thanks

apache-spark scala
1 Answers

You cannot read those kinds of fields with Spark, every name that is invalid in Java will give an error when trying to read through Spark. That is due to the fact that Spark generates code in Java, and they use the same names in the code generation step.

I have an open PR that improves the error message that Spark gives in the case when the field name starts with a number: https://github.com/apache/spark/pull/26319.

A work around is to do as @Alex Ott said, you need to first rename the column to a valid name, and then read it normally.

user_112909030
Related