How to replace large python dictionary with a memory efficient data structure?

user_9844342 0
Viewed 152

I use a python dictionary to store key value pairs and the dictionary gets too large(>100GB) and hits a memory limit.

What is a better memory efficient data structure to store key value pairs in python? E.g. we can use generators to replace lists

shared-memory dynamic-memory-allocation out-of-memory memory-management python
2 Answers

You can use sqlitedict which provides key-value interface to SQLite database. About memory usage. SQLite doesn't need your dataset to fit in RAM. By default it caches up to cache_size pages, which is barely 2MB .

user_58387550
user_2747570
Related