Numpy/Scipy: Memory consumption increases linearly per iteration

user_2131200 0
Viewed 142

When running the following code:

import numpy as np
import scipy.sparse
import time

def test():
    m = 10000
    n = 10000 
    for i in range(5):
        A = scipy.sparse.random(m, n, density=0.1, format='csr')
        x = np.random.randn(n)
        Ax = A.dot(x)
        time.sleep(2)

if __name__ == "__main__":
    test()

I observed that the memory consumption increased linearly to >4.8Gb!

I tested again with the following function:

def test2():
    m = 10000
    n = 10000
    for i in range(5):
        print(i)
        A = np.random.rand(m, n)
        x = np.random.randn(A.shape[1])
        Ax = A.dot(x)
        time.sleep(2)

The memory consumption increased linearly to >800Mb.

I have two questions:

  1. Why does the memory consumption increase linearly in each case? No new variables were declared at each iteration...

  2. Why in the first test the memory consumption is much much higher than in the second, given that the matrices are sparse (with only 0.1 density)?

Thank you in advance for your answers!

scipy numpy python
1 Answers

1. Because all your variable declarations are inside an un-nested for...loop, they are re-run in each iteration with each line contributing varying linear increments (O(n)) to the total memory usage. Also since there is no operation to free memory like a delete operation after an iteration, memory usage is incremented from previous iterations. Below are the results of a memory profile of def test()

|Line|  |Mem usage|    |Increment|   |Line Contents|
-----------------------------------------------------
     5   1844.9 MiB   1844.9 MiB   def test():
     6   1844.9 MiB      0.0 MiB       m = 10000
     7   1844.9 MiB      0.0 MiB       n = 10000 
     8   4518.7 MiB      0.0 MiB       for i in range(5):
     9   4518.7 MiB    763.7 MiB           A = scipy.sparse.random(m, n, density=0.1, format='csr')
    10   4518.7 MiB      0.0 MiB           x = np.random.randn(n)
    11   4518.7 MiB      0.0 MiB           Ax = A.dot(x)
    12   4518.7 MiB      0.0 MiB           time.sleep(2)
  1. I would attribute the difference in memory occupied by the two functions to the fact that numpy data structures are more optimized compared to scipy.i.e they take up less space.
user_104180860
Related