precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
zsh loses precision when miltiplying two floats of significantly different magnitude
user_83629820
•
2
2
88
floating-accuracy precision zsh
What is a significance of 9.536743e-7?
user_153766400
•
2
1
151
bit precision floating-point python
Precision limits of numpy floats?
user_10191290
•
2
2
420
floating-accuracy precision numpy python types
One-dimensional numerical integration in R with values > 2^1024?
user_89264250
•
2
1
144
numerical-integration precision r
Prevent ActiveRecord from rounding small numerical attributes ( < 1e-20)?
user_4007900
•
2
1
91
precision rails-activerecord ruby-on-rails
What's the maximum precision (after the decimal point) of a float in Javascript
user_4007900
•
4
2
1267
ieee-754 precision floating-point javascript
High precision calculation in R using Rmpfr
user_149998160
•
3
1
96
precision r
How to generate seeded random floats of at least 20 decimal places in python?
user_112822030
•
2
2
298
mpmath precision decimal numpy python
Are there any binary values that don't have exact representation in decimal?
user_125970
•
2
1
131
computer-science precision floating-point
How do I specify the printing precision for Sympy.pretty_print?
user_120659870
•
2
0
57
sympy precision pretty-print
Hot Questions