Pandas memory usage gives weird estimates

user_2258675 0
Viewed 61

Pandas gives very strange memory estimates. It can't be that a column with lists of 2 numbers, takes almost the same memory as a column with 100 numbers. Numpy's nbytes makes more sense (see example below), but doesn't the memory_usage with deep=True knows how to deal with it?

    import pandas as pd
    import numpy as np

    n_rows = 100000
    vecs_1x2 = np.random.random((n_rows, 2))
    mats_10x10 = np.random.random((n_rows, 10, 10))
    df = pd.DataFrame({'vecs_1x2': list(vecs_1x2), 'mats_10x10': list(mats_10x10)})

    print('numpy - vecs_1x2.nbytes: {} Mb'.format(vecs_1x2.nbytes / 1024.**2))
    print('numpy - mats_10x10.nbytes: {} Mb'.format(mats_10x10.nbytes / 1024.**2))
    print('pandas - df.memory_usage (in Mb):\n{}'.format(df.memory_usage(deep=True, index=False) / 1024.**2))

Outputs:

    numpy - vecs_1x2.nbytes: 1.52587890625 Mb
    numpy - mats_10x10.nbytes: 76.2939453125 Mb
    pandas - df.memory_usage (in Mb):
    mats_10x10    11.444092
    vecs_1x2       9.918213

How?!

pandas memory numpy python
1 Answers

What's going on is that you are creating columns with dtype of object. Basically, it's storing a NumPy array of Python objects.

So what you get is a NumPy array of pointers, basically. If you choose the deep option it tries to get individual items memory usage too, using sys.getsizeof() I suspect.

So deep=True is "size of the array, plus whatever guess is at object size":

>>> sys.getsizeof(df.vecs_1x2.iloc[0]) * n_rows + df.vecs_1x2.memory_usage(deep=False)
11200128
>>> df.vecs_1x2.memory_usage(deep=True)
11200128

Now, sys.getsizeof() will add the Python object overhead:

>>> sys.getsizeof(df.vecs_1x2.iloc[0])
104
>>> df.vecs_1x2.iloc[0].nbytes
16

But sys.getsizeof() doesn't know how much memory the object stores internally, necessarily, given arbitrary extension types:

>>> sys.getsizeof(df.mats_10x10.iloc[0])
120
>>> df.mats_10x10.iloc[0].nbytes
800
>>> sys.getsizeof(df.mats_10x10.iloc[0]) * n_rows + df.mats_10x10.memory_usage(deep=False)
12800128
>>> df.mats_10x10.memory_usage(deep=True)
12800128

TL;DR: For object dtypes, memory_usage() is not helpful.

So how do you measure memory usage?

One option is a memory profiler. For example, I work on the Fil memory profiler for Python which specifically is designed for data science and scientific computing. It will tell you peak memory usage and which lines of code were responsible.

Another profiler can show you how much memory was allocated on each line of code, which can be useful for profiling little tiny snippets of code.

user_62140340
Related