How to print the memory structure of a Python data type?

Question

I'd like to be able to produce a hex dump of the data structure of Python data types. The closest I can get right now is something like this:

from ctypes import string_at
a = [n for n in range(20)]
print(f"{string_at(id(a), getsizeof(a)).hex()}")

Is there a Python library that might take a variable and produce a nice data dump showing pointers, reference count, data type, etc.?

I tried googling by Google doesn't seem to understand what I am asking for. Most all results are about memory allocation and related topics.

EDIT: What might the output look like?

The CPython structure for a list, as defined in cpython\listobject.h looks like this:

typedef struct {
    PyObject_VAR_HEAD
    /* Vector of pointers to list elements.  list[0] is ob_item[0], etc. */
    PyObject **ob_item;

    /* ob_item contains space for 'allocated' elements.  The number
     * currently in use is ob_size.
     * Invariants:
     *     0 <= ob_size <= allocated
     *     len(list) == ob_size
     *     ob_item == NULL implies ob_size == allocated == 0
     * list.sort() temporarily sets allocated to -1 to detect mutations.
     *
     * Items must normally not be NULL, except during construction when
     * the list is not yet visible outside the function that builds it.
     */
    Py_ssize_t allocated;
} PyListObject;

What I would like to do within Python is to pass a variable into a function and have all of the above printed out in some useful manner, likely in hex.

Here's a contrived example (not an actual structure or data, just made-up to show the idea):

>>> a = ["a","b","c","d"]

>>> hex_dump_object(a)

location:  0x01FFFFFFFF
type:      0x40
ref count: 0x32 
elements:
[0] location: 0x0222222222
00000000 4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00 B8 00 00 00 00 00 MZ....................
00000016 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ..@...................

[1] location: 0x0222222222
0000002C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 01 00 00 0E 1F ......................
00000042 BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68 69 73 20 70 72 6F 67 72 ......!..L.!This progr

[2] location: 0x0222222222
0000006E 53 20 6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00 A3 60 D5 4B S mode....$........`.K
00000084 E7 01 BB 18 E7 01 BB 18 E7 01 BB 18 EE 79 28 18 ED 01 BB 18 74 4F .............y(.....tO

[3] location: 0x0222222222
00000000 4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00 B8 00 00 00 00 00 MZ....................
0000006E 53 20 6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00 A3 60 D5 4B S mode....$........`.K
00000084 E7 01 BB 18 E7 01 BB 18 E7 01 BB 18 EE 79 28 18 ED 01 BB 18 74 4F .............y(.....tO

allocated: 0x1000

Yes, I would include pointer addresses. This isn't about reaching into memory to manipulate any of it. It's about unrolling a data structure and showing all the internals in order to play with it and gain a better understanding of what's going on under the hood.

For example, you could add, remove or insert an element into the list, print a hex_dump_object() and see how it changed.

I'd like to believe someone has done something like this before. If not, when I have the time, I'll create a module for this. I think it could be quite useful, particularly in teaching Python. I'd have to read through CPython source and create a parser to output the different types based on the underlying structure. For example, a bytes object is far simpler than a mutable list.

How to print the memory structure of a Python data type?

0 Answers