precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
vectorised matching of values with numerical tolerance
user_93195560
•
2
2
175
match precision r
Pandas - replace, float and sum of columns
user_131740830
•
3
2
736
pandas dataframe precision python-3.x python
Understanding sys.float_info for maximum floats in python
user_97361110
•
2
2
1859
precision floating-point python
Square root calculation using continued fractions to n bits of precision
user_61995170
•
2
1
334
rational-number continued-fractions arbitrary-precision square-root precision
Different slices give different inequalities for same elements
user_101337970
•
6
1
87
precision numpy python
Trouble when computing modulos with floats in C
user_120423490
•
2
1
67
precision modulo c
what is meant by 'Most C system provide for logically infinite floating values'?
user_114178100
•
3
2
75
divide-by-zero format-specifiers precision c
Hamming numbers and double precision
user_76550810
•
3
2
215
hamming-numbers precision floating-point algorithm haskell
Floating Point Arithmetic: Deriving Wobble
user_67144260
•
3
1
171
numerical-methods precision floating-point
Which single-precision floating point numbers need 9 significant decimal digits for unambiguous decimal represenatation?
user_99525290
•
3
2
412
precision floating-point
Hot Questions