precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
R: Computing maximum floating-point error for log(exp(...))
user_93254130
•
4
3
132
floating-accuracy precision floating-point r
How to set decimal places in plotly subplots hoverlabel?
user_161480560
•
2
1
1097
plotly precision hover decimal
Get item closest to a value in a std::vector of doubles
user_66265540
•
2
2
61
precision stdvector double c++11
How does Matlab store this large of a number with only 64 bits?
user_148579400
•
1
1
83
precision floating-point matlab
Approximating derivatves and second derivatives of numerical functions in Common Lisp - not working as expected
user_116426910
•
3
2
66
numerical-analysis precision common-lisp lisp lambda
How is printf printing 53 digits after decimal point?
user_113355340
•
3
2
97
precision c++
For IEEE-754 floating point arithmetic, is the mantissa in [0.5, 1) or in [1, 2)?
user_148579400
•
2
1
180
ieee-754 precision floating-point math
Long double precision error saturation in RK integrator
user_110516110
•
2
1
81
runge-kutta long-double numerical-methods precision c
Pytorch getting RuntimeError: Found dtype Double but expected Float
user_151668850
•
8
2
22691
pytorch deep-learning precision casting python
Why does MinGW-w64 floating-point precision depend on winpthreads version?
user_24928010
•
2
2
313
mingw-w64 precision g++ pthreads c++
Hot Questions