Decimal precision works differently along same dataset

user_11588217 0
Viewed 37

I'm currently trying to round standard deviations to the sixth decimal place from an array of data.

Python round() didn't work as I wanted it to, given that some numbers were displayed oddly. For example, what I meant to be 0.013931 showed up as 0.013931099999999998. I fixed the gist of the issue by using Decimal and setting the context precision to 5, but now some standard deviations show up rounded to the 6th decimal while others are rounded to the 7th!

from decimal import *

getcontext().prec = 4
getcontext().rounding = ROUND_HALF_UP

print(Decimal(0.005855795678472189)/10)
print(Decimal(0.013931099999999998)/10)

I expect the output to be 0.00058558 and 0.0013931, yet the actual output is 0.0005856 and 0.001393, which have different lengths!

decimal python
1 Answers

The precision in the decimal package is applied to the fraction part of the floating point number. That is, in scientific notation you will always see 4 digits if you set getcontext().prec = 4, like so

>>> print(Decimal(0.005855795678472189)/10)
0.0005856
>>> print(Decimal(0.0005855795678472189)/10)
0.00005856
>>> print(Decimal(0.00005855795678472189)/10)
0.000005856
>>> print(Decimal(0.000005855795678472189)/10)
5.856E-7
>>> print(Decimal(0.0000005855795678472189)/10)
5.856E-8
>>> print(Decimal(0.00000005855795678472189)/10)
5.856E-9
>>> print(Decimal(0.000000005855795678472189)/10)
5.856E-10

Note that floating point numbers are stored in three parts

  • the first bit is the sign (plus/minus),
  • the next few bits are the exponent part. (This is 11 bits in floating point numbers that follow the IEEE 754 standard for 64 bit. This includes C++ double and python float. The exponent part is then the binary scientific notation exponent -1023 so we can have numbers between e-1024 of the exponent part is 0 (all zeros) and e+1023 if the exponent part is 2**11-1=2047 (all ones).)
  • the remaining bits are the fractional part.

The wikipedia article has details.

user_17352150
Related