We use a dict which contains about 4GB of data for data processing. It's convenient and fast.
The problem we are having is that this dict might grow over 32GB.
I'm looking for a way to use a dict (just like a variable with get()-method etc) which can be bigger than the available memory. It would be great if this dict somehow stored the data on disk and retrieved the data from disk when get(key) is called and value for the key is not in memory.
Preferably I wouldn't like to use an external service, like a SQL database.
I did find Shelve, but it seems to need the memory too.
Any ideas on how to approach this problem?