Disable scientific notation in Spark Scala

user_17907636 0
Viewed 227

I have a table to consume from Spark and do data transformations.

There is a column that have values with 8 numbers after . something like this: 0.00000001

This column have the absolute value like this number above.

I need to take the absolute value of that column, without let Spark transform this in scientific notation. When Spark reads the table, it transforms the column in scientific notation.

I have tried to convert in String, Double or Float, but nothing works. I need this field in the type Decimal. The type of the field is already Decimal natively in the column.

Is there some way to do this?

Code to simulate:

val df = spark.sparkContext.parallelize(Seq((0.00000001))).toDF("Value")

Spark shows me this: 0E-8

And i need this in Decimal Type: 0.00000001

Thanks!

apache-spark scala
1 Answers

You can use format_number function to get desired result.

spark.sparkContext.parallelize(Seq((0.00000001))).toDF("Value").
   selectExpr("format_number(Value,'#.########') as Value").show(false)

/*
+----------+
|Value     |
+----------+
|0.00000001|
+----------+*/
user_87733090
Related