Ensure that calculations are done 64 bits (or at least warn of overflow)

user_3183950 0
Viewed 36

I am using python and NumPy. I had the following basic quantity to compute:

(QL * (7**k))**2

Where 

QL = 200003
k = 4

What puzzled me is that it returned a wrong (negative) number, which doesn't make sense. Then I realised after looking on the internet that the problem was because k was a 32-bit numpy integer.

A minimal working example can be the following:

QL = 200000
k = np.arange(10)[4]
print((QL * 7**k)**2)

This returns 406556672 instead of the correct answer 230592040000000000. The number is not negative here, but the same problem still occurs.

My question is:

How can I make sure that all the numbers used in my code are of the biggest possible integer size?

I don't want to explicitly specify it for each number that I create.

How can I at least force python to warn me when such things happen?

integer-overflow numpy python
1 Answers

When you write QL = 200003; k = 4 in Python, the numbers are interpreted as ints. By default, if you were to convert these into numpy arrays or scalars, you would end up with whatever the default integer type is on your system.

Here is an example using one-element arrays:

QL = np.array([200003])
k = np.array([4])

On my system, I find that the dtype of both arrays is int32. You can change that by selecting your preferred dtype:

QL = np.array([200003], dtype=np.int64)
k = np.array([4], dtype=np.int64)

If you don't have access to the arrays at creation time, you can always convert them:

QL = QL.astype(np.int64)
k = k.astype(int64)

An option that is worth considering for integer math is skipping numpy altogether and using Python's infinite precision integers. If one of the numbers is a numpy scalar or one-element array, you can retrieve the corresponding Python object using the item method:

QL = QL.item()
k = k.item()

Numpy should raise at least a warning for overflow, but apparently this fails for some operations: https://github.com/numpy/numpy/issues/8987

TL;DR

In your case, k is a numpy scalar of type int32. You can do either one of the following:

For a numpy 64-bit result:

k = np.int64(k)

For an infinite-precision Python result:

k = k.item()

If you don't want to cast each k explicitly, you can create the range using the correct type:

k = np.arange(10, dtype=np.int64)[4]

There is no reliable way to set the default integer type for all new arrays without specifying it explicitly.

user_29887300
Related