precision

318 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Is there a way to speed up python via float precision or limiting float range?
user_22247270
•
4
1
142
precision math optimization python performance
1e400 == 10**400 is False?
user_124971410
•
1
1
65
exponent precision overflow integer python
why 0.3 + 0 == 0.3 and 0.1 + 0 == 0.1 in javascript when 0.3 and 0.1 cannot be represented exactly in binary
user_175688170
•
2
1
282
ieee-754 numbers precision floating-point javascript
What determines NumPy default integer precision?
user_20915850
•
1
1
67
precision integer numpy python
Control printout of floating point decimals, in Pandas DataFrame with decimal.Decimal entries?
user_61974390
•
1
1
598
pandas precision decimal python
How does numpy manage to divide float32 by 2**63?
user_179545360
•
1
1
49
floating-accuracy precision floating-point numpy
What is the minimum decimal which can be converted to float and then back to decimal?
user_5074340
•
2
1
76
precision decimal floating-point converters c#
Running the same script on the same pandas data produces very slightly different dataframes floating-point values
user_77980120
•
2
1
118
pandas precision floating-point csv python
Why some double precision divisions are correct in java operation?
user_180451610
•
3
1
104
precision decimal double binary java
Logstash 8.0 @timestamp with nanoseconds precision
user_57316070
•
2
3
588
logstash precision timestamp
Hot Questions