Converting string "decimal" to DecimalType

user_2751573 0
Viewed 766

Due to an over-complicated process, I need to convert strings representing a data type to an actual org.apache.spark.sql.types data type. I've got a simple function almost working.

def getType (s: String): DataType = {
  s match {
    case "string" => StringType
    ...
    case "decimal" => DecimalType
    //case "decimal" => org.apache.spark.sql.types.DecimalType

  }
}

Both versions of the decimal line cause it to fail:

found  : org.apache.spark.sql.types.DecimalType.type
required: org.apache.spark.sql.types.DataType

How do I handle this?

apache-spark scala
1 Answers

You have several ways to create a DecimalType.

You could use new(Java version) to create a DecimalType.

The default precision and scale is (10, 0) new DecimalType(), but you can provide precision and scale. The precision can be up to 38, scale can also be up to 38 (less or equal to precision).

For example

    def getType (s: String): DataType = {
      s match {
        case "string" => StringType
        case "decimal" => new DecimalType(precision = 18, scale = 12)
        case "decimalV2" =>  DataTypes.createDecimalType() // This works too, default precision and scale
        case "decimalV3" => DataTypes.createDecimalType(precision = 12, scale = 8) // if you want to provide precision and scale
        //case "decimal" => org.apache.spark.sql.types.DecimalType
      }
    }
user_85714980
Related