precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Python/Mpmath: Why don't I get any decimal points for large number division, but do for smaller numbers
user_114691710
•
2
1
221
precision decimal python
How to avoid math.sin(math.pi*2*VERY LARGE NUMBER) having a much larger error margin than math.sin(math.pi*2)?
user_114691710
•
3
3
188
trigonometry precision math numpy python
Python multiplication losing floating point precision
user_129869610
•
2
2
542
precision floating-point python
C++ default precision for scientific floating hex and decimal output
user_20792300
•
3
0
113
precision c++
arctan2 discrepancy in NumPy
user_120735370
•
2
2
244
atan2 numeric precision numpy python
Is Mathf.Approximately(0.0f, float.Epsilon) == true its correct behavior?
user_80416320
•
6
2
1506
unity3d floating-accuracy precision floating-point c#
How can I justify that f64::from_bits(0x3fe9000000000000 u64 ) == 0.781250 f64
user_117719360
•
2
2
136
fixed precision floating-point double hex
Floating point accuracy with different languages
user_52466550
•
6
2
681
transcendental-equation precision floating-point language-agnostic c++
Swift encodes double 0.1 to JSON as 0.10000000000000001
user_9039870
•
4
1
1690
swifty-json swift precision type-conversion json
Difference between double and float in floating point accuracy
user_78826210
•
3
1
213
precision floating-point casting c#
Hot Questions