I'm testing pandas memory usage with text corpora of different lengths. Memory usage is inconsistent and I'm trying to understand why.
I have two text corpora of radically different length (a corpus of yelp reviews: ~ 6 million words - and the British National Corpus ~ 100 million words). I'm using two pandas dataframes to store and manipulate these data. The shape and datatypes of the two corpora are identical. In each dataframe, each row is one word and each column is a piece of metadata about that word (e.g., part-of-speech tag, lemma, the word's text's id). The datatype of all columns in both dataframes is category as all values are strings and most are repeated. Using
dataframe.memory_usage(deep=True, index=True).sum()
pandas tells me the Yelp corpus takes up about 0.2 GBs and the BNC takes up about 1.8 GB. Even though the BNC is about 17 times larger than the yelp corpus, it takes up nine times the memory. OK, fair, the corpora are different. They're not directly comparable. If I concatenate the yelp corpus to the end of the BNC and relevel the categorical variables using
pandas.concat([bnc, yelp]).astype('category')
I expect the resulting dataframe to take up somewhat less than 2 GBs of memory (1.8 + 0.2 - overhead and duplicated category levels in the two corpora), but it actually takes up 3.2 GBs. Additionally, if I double the yelp corpus using pandas.concat([yelp, yelp]), the memory usage increases to about 0.4 GB. On the other hand, if I double the BNC using the same code, the memory usages increases to 5.2 GBs, nearly three times the memory usage of the original corpus.
I've read up on how pandas represents series/dataframes in memory, but nothing I can find explains this. I'm also sorry to say I can't provide links to the corpora as I don't have permission to distribute them. I'm hoping someone can provide insight rather than code.