I use numpy to load a large matrix using 64bit Python.
It works fine on Macbook Pro with 8GB memory.
>>> from sklearn.preprocessing import MultiLabelBinarizer
>>> mb = MultiLabelBinarizer()
>>> matrix = mb.fit_transform(questions_topics)
>>> sys.getsizeof(matrix)
47975472376
>>> matrix.shape
(2999967, 1999)
But it raises MemoryError on Ubuntu Google VM instance with 16GB memory and 10GB swap.
>>> y = mb.fit_transform(questions_topics)
/home/Liwink/anaconda3/lib/python3.5/site-packages/scipy/sparse/base.py in _process_toarray_args(self, order, out)
1037 return out
1038 else:
-> 1039 return np.zeros(self.shape, dtype=self.dtype, order=order)
1040
1041 def __numpy_ufunc__(self, func, method, pos, inputs, **kwargs):
MemoryError:
When the matrix is loaded on Mac OS, it takes 50G VIRT.

I have two questions:
- Where is the matrix(about 50GB) kept, in memory or disk?
- How can I load this matrix on VM?