precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Using Julia's range function such that the range values have the same number of significant digits
user_80987810
•
1
2
38
julia precision floating-point
How common are 31b ints in modern Ocaml
user_203940
•
1
1
48
precision ocaml
Why this simple expression is resulting in the wrong value
user_199454390
•
4
0
53
precision r
How can I get Quadruple precision in Swift?
user_175999760
•
1
1
55
quadruple-precision swift precision
c++ std::nextbefore function?
user_1582850
•
3
1
104
precision c++
VB6 - Different behaviour when summing 2 variables and when summing them using raw values
user_152780760
•
2
0
50
precision vb6 double
floating-point error fligner.test r function?
user_65536000
•
6
1
43
floating-accuracy precision floating-point r
Does the average of floats satisfy conditions for machine-precision bisection?
user_111308540
•
3
1
59
floating-accuracy numeric precision floating-point
Why does Double.ToString("#") and BigInteger.ToString() produce different results for the same value?
user_10336860
•
1
1
48
biginteger precision c#
Can I naively check if a/b == c/d?
user_173609860
•
3
3
247
precision python c++
Hot Questions