overloaded method value select with alternatives

user_15084581 0
Viewed 603

I'm trying to select more columns and cast all of them but I receive this error

"overloaded method value select with alternatives: (col: String,cols: String*)org.apache.spark.sql.DataFrame (cols: org.apache.spark.sql.Column*)org.apache.spark.sql.DataFrame cannot be applied to (org.apache.spark.sql.Column, org.apache.spark.sql.Column, String)"

the code is this:

val result = df.select(
  col(s"${Constant.CS}_exp.${Constant.DATI_CONTRATTO}.${Constant.NUMERO_CONTRATTO}").cast(IntegerType),
  col(s"${Constant.CS}_exp.${Constant.DATI_CONTRATTO}.${Constant.CODICE_PORTAFOGLIO}").cast(IntegerType), 
  col(s"${Constant.CS}_exp.${Constant.RATEALE}.${Constant.STORIA_DEL_CONTRATTO}"))
apache-spark scala
1 Answers

The last part of the error message means that the compiler cannot find a method "select" with an api that fit your code: select(Column, Column, String)

However, the compiler found 2 possible methods, but they don't fit:

  • select(col: String, cols: String*)
  • select(cols: Column*) (the * means "any number of")

This, I am sure of.

However, I don't understand why you get that error with the code you've given that actually is select(Column, Column, Column) which fits the select(cols: Column*) api. For some reason, it consider the last argument to be a String. Maybe some parenthesis are wrongly placed

What I do in such cases, is to split the code to validate types:

val col1: Column = col(s"${Constant.CS}_exp.${Constant.DATI_CONTRATTO}.${Constant.NUMERO_CONTRATTO}").cast(IntegerType)
val col2: Column = col(s"${Constant.CS}_exp.${Constant.DATI_CONTRATTO}.${Constant.CODICE_PORTAFOGLIO}").cast(IntegerType)
val col3: Column = col(s"${Constant.CS}_exp.${Constant.RATEALE}.${Constant.STORIA_DEL_CONTRATTO}")

val result = df.select(col1, col2, col3)

and check it compiles alright

user_12069980
Related