PySpark create histogram dataframe for DecimalType column

user_5316326 0
Viewed 308

How to create the count of equally distributed bins for a single columned DataFrame with the following schema:

>>> df.schema
StructType(List(StructField(a,DecimalType(38,0),true)))

Create the dataframe as an MWE:

from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, DecimalType
from decimal import Decimal

schema = StructType([StructField('a', DecimalType(38,0), True)])

df = spark.createDataFrame(
    spark.sparkContext.parallelize([Row(a=Decimal(x)) for x in range(100)]),
    schema
)

Now this is an often accepted answer:

df.select('a').rdd.flatMap(lambda x: x).histogram(10)

but it gives a type error:

TypeError: unsupported operand type(s) for *: 'decimal.Decimal' and 'float'

So something goes wrong for decimal types. What is the approach to resolve the issue, or how to cast the values into an accepted data type?

rdd pyspark apache-spark python
1 Answers

You can cast to float using a map:

df.select('a').rdd.flatMap(lambda x: x).map(float).histogram(10)
user_141657300
Related