Numpy and memory allocation on Mac OS X vs. Linux

Viewed 776

I use numpy to load a large matrix using 64bit Python.

It works fine on Macbook Pro with 8GB memory.

>>> from sklearn.preprocessing import MultiLabelBinarizer
>>> mb = MultiLabelBinarizer()
>>> matrix = mb.fit_transform(questions_topics)
>>> sys.getsizeof(matrix) 
47975472376
>>> matrix.shape
(2999967, 1999)

But it raises MemoryError on Ubuntu Google VM instance with 16GB memory and 10GB swap.

>>> y = mb.fit_transform(questions_topics)
/home/Liwink/anaconda3/lib/python3.5/site-packages/scipy/sparse/base.py in _process_toarray_args(self, order, out)
1037             return out
1038         else:
-> 1039             return np.zeros(self.shape, dtype=self.dtype, order=order)
1040
1041     def __numpy_ufunc__(self, func, method, pos, inputs, **kwargs):
MemoryError:

When the matrix is loaded on Mac OS, it takes 50G VIRT. enter image description here

I have two questions:

  1. Where is the matrix(about 50GB) kept, in memory or disk?
  2. How can I load this matrix on VM?
1 Answers
Related