Pandas memory usage inconsistencies

user_10171000 0
Viewed 831

I have some memory inconsistencies when I am using a Pandas DataFrame.

Here is my code skeleton:

import pandas as pd
import numpy as np

columns_dtype = {'A': np.int16, 'B': np.int8, ...}
df = pd.read_csv('my_file.csv', dtype=columns_dtype)

That is basically just reading a csv file with pandas while controlling column data types. But then, when I am looking for how much memory is allocated to my program, the information does not seem coherent.

Info 1:

df.info(memory_usage='deep')

That gives : memory usage: 482.6 MB

Info 2:

import dill, sys
sys.getsizeof(dill.dumps(df))

That gives : 506049782 (so 506 Mb)

Info 3:

The RAM allocated to my program is : 1.1 GiB (or 1.2 Gb)

Additional info (but i do not think it is relevant):

the size of my_file.csv is 888 Mb (ls -lh)

The issue:

As I am just loading my csv file into a Pandas DataFrame object, why does my program needs more than 1 Gb of RAM whereas the object size is about 0.5 Gb ?

Many thanks

pandas memory python
1 Answers

I'm not going to pretend to know the deep underlying use of dynamic memory to hold data in Pandas. Pandas is hungry when it loads large flat files, as a rule of thumb, pandas will use 5-10x the amount of ram as the file size you're loading to do analysis on.

To avoid these you can chunk the data when loading:

  df = pd.load_csv(file_path, chunksize = 30000)

or if you're doing analysis down columns:

  df = pd.load_csv(file_path, usecols = list_of_columns_index)

or both!

  df = pd.load_csv(file_path, chunksize = 30000, usecols = list_of_columns_index)

Hope this helps speed up your analysis.

user_110191730
Related