I am working on a search engine implementation in Python, and would ideally like to store as much of an inverted index into memory.
The current data structure of the index is as follows:
{term: [doc_frequency, {doc_number: [doc_positions]}]}, where term is a string and doc_frequency, doc_number and doc_positions are integers. The size of this blows up really quickly for multiple documents.
As the document numbers and positions are frequently recurring integers for different terms, I was thinking if there is a way to leverage this characteristic by not storing a new integer every time but just referencing to the same integer multiple times?
I'm not an expert in data structures or Python at all, so forgive me if this is a stupid question. If there are any other suggestions than my specific question that could improve the memory usage, those are more than welcome as well.