DecimalType(20, 0) does not hold 7 digit integer in spark

user_9599466 0
Viewed 126

I've tried replacing int with Decimal in spark, but I'm getting the below error.

spark.createDataFrame([{'abc': 7010930}], StructType([StructField('abc', DecimalType(20, 0), True)]))

field abc: DecimalType(20,0) can not accept object 7010930 in type <class 'int'>

I understand that if the Precision is 20 it should be able to get 7 digits, but where am I going wrong?

pyspark apache-spark-sql apache-spark dataframe
1 Answers

You can pass a decimal point instead of int object using the python class decimal.Decimal:

from decimal import Decimal

df = spark.createDataFrame(
    [(Decimal(7010930),)],
    StructType([StructField('abc', DecimalType(20, 0), True)])
)

df.printSchema()

#root
# |-- abc: decimal(20,0) (nullable = true))
user_13865510
Related