Writing into a NumPy memmap still loads into RAM memory

Viewed 6894

I'm testing NumPy's memmap through IPython Notebook, with the following code

Ymap = np.memmap('Y.dat', dtype='float32', mode='w+', shape=(5e6, 4e4))

As you can see, Ymap's shape is pretty large. I'm trying to fill up Ymap like a sparse matrix. I'm not using scipy.sparse matrices because I will eventually need to dot-product it with another dense matrix, which will definitely not fit into memory.

Anyways, I'm performing a very long series of indexing operations:

Ymap = np.memmap('Y.dat', dtype='float32', mode='w+', shape=(5e6, 4e4))
with open("somefile.txt", 'rb') as somefile:
    for i in xrange(5e6):
        # Read a line
        line = somefile.readline()
        # For each token in the line, lookup its j value
        # Assign the value 1.0 to Ymap[i,j]
        for token in line.split():
            j = some_dictionary[token]
            Ymap[i,j] = 1.0

These operations somehow quickly eat up my RAM. I thought mem-mapping was basically an out-of-core numpy.ndarray. Am I mistaken? Why is my memory usage sky-rocketing like crazy?

2 Answers
Related