Sagemaker: MemoryError: Unable to allocate ___for an array with shape ___ and data type float64

user_9322983 0
Viewed 2821

I am running a notebook in sagemaker and it seems like one of the arrays produced after vectorizing text is causing issues.

Reading other answers it seems like it is an issue with overcommit. And one of the solutions proposed is to set it to always overcommit with this:

$ echo 1 > /proc/sys/vm/overcommit_memory

Is there any documentation or do you have any suggestion on how to do the same thing in sagemaker?

Thank you very much.

amazon-sagemaker pandas memory python-3.x arrays
2 Answers

Open a root shell by sudo -i and then run the echo 1 > /proc/sys/vm/overcommit_memory

user_140825920

Changing the default kernel in the SageMaker Notebook to conda_python3 resolved this issue for me.

user_136905500
Related