precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Get whole decimal digits for precision after Splitting List of float numbers in Pandas column
Admin1
•
2
1
59
pandas dataframe precision python-3.x python
What range of numbers can I represent using scaled_float in Elasticsearch?
user_11269710
•
3
1
312
elasticsearch floating-accuracy precision limit floating-point
How to get higher precision for sine using the Arb library?
user_39128360
•
3
1
103
arbitrary-precision trigonometry precision c
Is there a double floating point data type in Godot shaders?
user_131667350
•
2
1
301
godot precision shader double
Multiplying the probability matrix, incorrect values on a large number of iterations
user_130434940
•
2
2
551
matrix-multiplication precision matrix arrays java
Mathematical precision at 19th decimal place and beyond
user_44005900
•
6
1
165
precision math raku
Understanding Float>>asFraction and its variants
user_9805500
•
8
4
203
fractions smalltalk precision floating-point
Why is this result more accurate than another result by equivalent functions
user_148585130
•
2
2
239
precision floating-point python
Accuracy of the computation
user_151598640
•
3
1
87
floating-accuracy precision floating-point matlab
Defined constant for max sequential integer that can be stored in a float?
user_1208880
•
2
3
152
c++20 c++17 precision floating-point c++
Hot Questions