How can a csv that is 3.6 gb take up all my memory of 64gb

user_12385251 0
Viewed 118

I am loading in a csv into a pandas dataframe. The csv is 3.6 gb and I have 64 gb of ram. How is it possible that the memory exceeds 64 gb when loading a 3.6 gb file?

Is there a better way to load the entire dataframe that does not take up so much memory or perhaps there is something wrong with my computer.

Here is the code I am using to load the csv

df = pd.read_csv('../input/ML_DATA.csv')

I could also provide the csv file if that is of interest.

Here is a sample of what the data looks like

df = pd.read_csv('../input/ML_DATA.csv', nrows=10)
df.shape -> (10, 4247)

Here is a screen shot

enter image description here

Here is a print of df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Columns: 4247 entries, Location+Type to Pct of housing units in 4+ unit buildings
dtypes: float64(1132), int64(3), object(3112)
memory usage: 99.7+ KB
None
pandas python
1 Answers

Nothing is wrong with your computer. You are experiencing the overhead associated with storing the data.

Each of these values is probably being stored as a 64-bit value inside your computer, but a column that is stored as a string is being stored as a python object, which is going to be more like 240 bytes per cell.

So drop the columns you don’t need, enable swapping, and if you want to get really fancy, learn how to use Dask.

Also, as an aside, if you put this whole table into a SQLlite3 database, it will be a lot easier.

user_511670
Related