precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Why does Rust only use 16 significant digits for f64 equality checks?
user_49087040
•
3
2
594
rust precision floating-point
Is it possible to have more than 15 decimal digits in a float without Decimal module?
user_135177010
•
2
1
520
floating-accuracy precision floating-point python
numeric precision in Oracle
user_5741870
•
4
3
352
floating-accuracy precision floating-point oracle sql
Why do different step sizes produce large numerical error while approximating derivatives?
user_149744570
•
2
1
119
derivative numeric ieee-754 precision python
Inconsistent atan2 results between 64-bit and 32-bit architecture
user_14258340
•
3
1
145
precision floating-point x86 gcc c++
Prettier floats using yaml
user_15750660
•
2
2
738
precision double yaml python
Why the numeric_limits Machine Epsilon does not satisfy the 1+e>1 condition?
user_22913100
•
3
1
115
epsilon floating-accuracy precision floating-point c++
9**66 != n**66 with n=9 in Node 11.6.0
user_20048410
•
6
0
171
node.js precision
Floating point error in ULPs (units in the last place)
user_67923560
•
2
1
579
precision floating-point
Double to string conversion different for .NET Framework and .NET Core
user_57551660
•
2
1
365
precision json.net double c#
Hot Questions