Why does pd.to_numeric not work with large numbers?

user_3899735 0
Viewed 1353

Let's say I have a large number in a string, like '555555555555555555555'. One could choose to convert it to an int, float or even a numpy float:

int('555555555555555555555')
float('555555555555555555555')
np.float('555555555555555555555')

However, when I use the pandas function pd.to_numeric, things go wrong:

pd.to_numeric('555555555555555555555')

With error: 

Traceback (most recent call last):
  File "pandas/_libs/src/inference.pyx", line 1173, in pandas._libs.lib.maybe_convert_numeric
ValueError: Integer out of range.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\path_to_conda\lib\site-packages\IPython\core\interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-34-6a735441ab7b>", line 1, in <module>
    pd.to_numeric('555555555555555555555')
  File "C:\path_to_conda\lib\site-packages\pandas\core\tools\numeric.py", line 133, in to_numeric
    coerce_numeric=coerce_numeric)
  File "pandas/_libs/src/inference.pyx", line 1185, in pandas._libs.lib.maybe_convert_numeric
ValueError: Integer out of range. at position 0

What's going wrong? Why can't pandas to_numeric handle larger values? Are there any use cases why you would use pd.to_numeric instead of functions like np.float?

pandas numpy python
1 Answers

Because your number is larger that the maximum size of an integer that your system is capable of saving:

In [4]: import sys

In [5]: sys.maxsize
Out[5]: 9223372036854775807

In [6]: 555555555555555555555 > sys.maxsize
Out[6]: True

Here is part of the source code that raises the ValueError:

if not (seen.float_ or as_int in na_values):
    if as_int < oINT64_MIN or as_int > oUINT64_MAX:
        raise ValueError('Integer out of range.')

As you can see, because your number is not a float it treats it as an integer and checks if the number is in the proper range oINT64_MIN, oUINT64_MAX. If you've passed a float number instead it'd gave you the proper result:

In [9]: pd.to_numeric('555555555555555555555.0')
Out[9]: 5.5555555555555554e+20
user_28679280
Related