How do I increase decimal precision in Spark?

Viewed 22397

I have a large DataFrame made up of ~550 columns of doubles and two columns of longs (ids). The 550 columns are being read in from a csv, and I add two id columns. The only other things I do with the data is change some of the csv data from strings to doubles ("Inf" -> "0" then cast the column to double) and replace NaN's with 0:

df = df.withColumn(col.name + "temp", 
                             regexp_replace(
                                 regexp_replace(df(col.name),"Inf","0")
                                 ,"NaN","0").cast(DoubleType))
df = df.drop(col.name).withColumnRenamed(col.name + "temp",col.name)
df = df.withColumn("timeId", monotonically_increasing_id.cast(LongType))
df = df.withColumn("patId", lit(num).cast(LongType))
df = df.na.fill(0)

When I do a count, I get the following error:

IllegalArgumentException: requirement failed: Decimal precision 6 exceeds max precision 5

There are hundreds of thousands of rows, and I'm reading in the data from multiple csvs. How do I increase the decimal precision? Is there something else that could be going on? I am only getting this error when I read in some of the csvs. Could they have more decimals than the others?

1 Answers
Related