lua_numbertointeger - why can we assume INT_MIN has an exact representation as a float?

user_673679 0
Viewed 112

Lua stores numbers internally as either integers or floats. When doing a comparison operation (e.g. less-than) between numbers of the two different types, it needs to convert from one type to the other to make a comparison.

The integer and float types vary depending on configuration / platform, but certain assumptions are made. (I'm only really interested in "normal" 32 or 64 bit two's complement ints and 32 or 64 bit IEEE754 floats).

When the integer type may not have an exact representation in the floating point format, Lua tries to convert the floating point value to an integer.

This is done in a macro lua_numbertointeger:

/*
@@ lua_numbertointeger converts a float number with an integral value
** to an integer, or returns 0 if float is not within the range of
** a lua_Integer.  (The range comparisons are tricky because of
** rounding. The tests here assume a two-complement representation,
** where MININTEGER always has an exact representation as a float;
** MAXINTEGER may not have one, and therefore its conversion to float
** may have an ill-defined value.)
*/
#define lua_numbertointeger(n,p) \
  ((n) >= (LUA_NUMBER)(LUA_MININTEGER) && \
   (n) < -(LUA_NUMBER)(LUA_MININTEGER) && \
      (*(p) = (LUA_INTEGER)(n), 1))

My question is thus:

Assuming that LUA_NUMBER is a 32 bit float, and LUA_INTEGER is a 32 bit int why can we assume that LUA_MIN_INTEGER (i.e. INT_MIN) has an exact representation in the floating point type.

(And the same for a 64 bit int and 64 bit floating point... and would it work with 64 bit int and 32 bit float?)

lua c
2 Answers

In 2's complement representation, the minimum value typically has only 1 bit set, as in C's INT_MIN which might be 0x80000000. So it can be exactly represented in a float because it has only one significant bit, and can be normalised without loss of precision.

But INT_MAX which is say 0x7FFFFFFF has 31 significant bits, more than the significand of float has, and so it cannot be exactly represented by a float.

user_41429240

INT_MIN, as 2's complement, is a power of 2 (negated) and so has an exact FP representation. (Potential range exception 128-bit int and 32-bit float)

Thus the math in (n) >= (LUA_NUMBER)(LUA_MININTEGER) is exact without conversion/rounding error.

Yet a functional problem exist for 64-bit floating point and 32-bit int in the if() tests.

lua_numbertointeger(n,p) is wrong in an edge case. As (LUA_INTEGER)(n) truncates the fractional portion of n, I'd expect the correct preceding if test to be (n) - (LUA_NUMBER)(LUA_MININTEGER) > -1 ... instead of (n) >= (LUA_NUMBER)(LUA_MININTEGER). This allows values in (LUA_MININTEGER - 1 ... LUA_MININTEGER) to convert.

The value range on conversion should be

(INT_MIN - 1 .... INT_MAX + 1)
// not 
[INT_MIN     .... INT_MAX + 1)
user_24103590
Related