I receive a very weird error that occurs when running the code as a script from the terminal (both Ubuntu WSL on Windows and Ubuntu on virtual machine), which does not occur when I run the same code in a jupyter notebook.
I have a large DataFrame (dtypes int64, uint8, float64) and I want to slice it as follows:
import pandas as pd
import numpy as np
features_df = a lot of features
for i in range(5):
#This is just an example that raises the same exception
idx_train = np.random.randint(0, high=len(features_df), size=3125)
df_train = features_df.iloc[idx_train]
other stuff...
No index is out of range. The dataframe needs 1.8 GB of memory, which should not be a problem for my computer.
This line (both with loc and iloc) raises the following exception:
Traceback (most recent call last):
File "myscript.py", line 165, in <module>
df_train = features_df.iloc[idx_train]
File "mypath/ML/lib/python3.8/site-packages/pandas/core/generic.py", line 1522, in __invert__
new_data = self._mgr.apply(operator.invert)
File "mypath/ML/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 302, in apply
applied = b.apply(f, **kwargs)
File "mypath/ML/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 402, in apply
result = func(self.values, **kwargs)
TypeError: ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
The same exception occurs sometimes one line before when I create the array for testing and there is no call to pandas.
The same copy-pasted code runs without problems in jupyterlab and from the terminal with
import pdb; pdb.set_trace()
right before this line.
I run the script in a terminal with:
python myscript.py
The pandas version is 1.4.3, a downgrade to 1.4.2 did not solve the problem. Deleting all the pycache of the virtual environment did not solve the problem, either. Sometimes the process just gets killed in the first loop.
Is this a problem with the cache of python or the memory allocation? Or a bug in pandas? I would exclude a problem in the pandas call, since the same line of code runs elsewhere.
I would be very thankful for any hint about this issue.