precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
64bit floating point math in JavaScript, why does division work better than multiplication in this situation
user_136673920
•
3
0
35
floating-accuracy precision floating-point javascript
Is 17-digit decimal representation of float in Python 3 (using IEEE-754) always "exact" up to numerical accuracy?
user_10320990
•
2
0
96
python-3.10 ieee-754 precision floating-point python
What drives numerical instability in eigenvalue computations in python?
user_173722030
•
2
1
74
cluster-analysis precision matrix numpy python
Why does Clang add extra FMA instructions?
user_110722900
•
2
1
77
numerical-methods precision x86 assembly c++
What are the correct C types to use in Idris FFI?
user_59869070
•
2
0
37
idris ffi precision integer c
Why does C or C++ “-0” not produce a floating-point −0?
user_100481440
•
4
1
76
ieee-754 precision floating-point c++
How to limit decimals when adjusting a raster array with rasterio
user_173979790
•
1
0
25
rasterio precision decimal python
Precision error when finding eigenvalues of Hecke operator
user_192341850
•
1
0
28
eigenvalue sage precision
find cubic root of a number from 0 to 125
user_176550270
•
3
1
46
cubic precision c
Loss of precision in decimal display of double
user_139630860
•
3
1
69
precision double java
Hot Questions