Find the memory size of a set of strings vs. set of bytestrings

user_1422096 0
Viewed 152

Edit: the answer from Memory usage of a list of millions of strings in Python can be adapted to sets too.

By analyzing the RAM usage on my machine (with the process manager), I noticed that a set of millions of strings like 'abcd' takes much less memory than a set of millions of bytes b'abcd' (Edit: I was wrong, it was due to an error elsewhere). I would like to test this:

import random, string, sys
randomstring = lambda length: ''.join(random.choice(string.ascii_lowercase) for _ in range(length))
s1 = {randomstring(10) for i in range(100_000)}
s2 = {randomstring(50) for i in range(100_000)}
s3 = {randomstring(10).encode() for i in range(100_000)} 
s4 = {randomstring(50).encode() for i in range(100_000)} 
print(sys.getsizeof(s1), sys.getsizeof(s2), sys.getsizeof(s3), sys.getsizeof(s4))

but here it always gives the same size: 4194528 whereas the size should vary with a factor x5, and probably be different for the string vs bytes case.

How to measure the memory size taken by these sets and all its elements?

Note: I know that finding the whole memory taken by a structure is not easy in Python (see also In-memory size of a Python structure), because we need to take in account all the linked elements.

TL;DR: Is there a tool in Python to automatically measure the memory size of a set + the memory taken by the internal references (pointers?), the hashtable buckets, the elements (strings here) that are hosted in the set...? In short: every byte that is necessary for this set of strings. Does such a memory measurement tool exist?

ram memory set python
1 Answers

sys.getsizeof does not measure the size of the full target data structure. It only measure the memory taken by the set object which contains references to strings/bytes objects. The references are not included in the returned memory consumption (ie. it does not walk recursively in each object of the target data structure). A reference takes typically 8 bytes on a 64-bit platform and a CPython set is not as compact as a list: it is implemented like a hash-table with many buckets and some buckets are unused. In fact, this is mandatory for this data structure to be fast (in general, the occupancy should be 50%-90%). Moreover, each bucket contains a hash which usually takes 8 bytes.

The string themselves take much more space than a bucket (at least on my machine):

sys.getsizeof(randomstring(50))           # 99
sys.getsizeof(randomstring(50).encode())  # 83

On my machine, it turns out that CPython strings are 16 bytes bigger than bytes.

user_129395570
Related