Modifying/debugging frozen builtins like importlib/_boostrap_external.py

Question

Short version: How do I debug frozen libs? Can I tell Python not to use them (and use sourcefiles) or re-freeze them some how?

I'm trying to learn more about the inner functionality of pythons import mechanisms. And by going down that rabbit hole, I want to debug /usr/lib/python37/importlib/_bootstrap_external.py.

Using python -m trace --trace myscript.py gave me indications of where in _bootstrap_external.py I am landing, but not values. So I turned to pdb and it gave me a bit of information but appears to be skipping _frozen objects. (?).

So I added a few print('moo') where I saw my script ending up in _boostrap_external.py but no prints are ever made, because Python internally uses

That lead me in a desperate attempt to try and renaming/removing /usr/lib/python37/importlib/__pycache__ in a hope that Python would re-compile my changes. But no such luck, Python still uses a frozen version.

So I modified /usr/lib/python37/__init__.py where it imports the _bootstrap module, so I changed this:

try:
    import _frozen_importlib_external as _bootstrap_external
except ImportError:
    from . import _bootstrap_external
    _bootstrap_external._setup(_bootstrap)
    _bootstrap._bootstrap_external = _bootstrap_external
else:
    _bootstrap_external.__name__ = 'importlib._bootstrap_external'
    _bootstrap_external.__package__ = 'importlib'
    try:
        _bootstrap_external.__file__ = __file__.replace('__init__.py', '_bootstrap_external.py')
    except NameError:
        # __file__ is not guaranteed to be defined, e.g. if this code gets
        # frozen by a tool like cx_Freeze.
        pass
    sys.modules['importlib._bootstrap_external'] = _bootstrap_external

to this:

from . import _bootstrap_external
_bootstrap_external._setup(_bootstrap)
_bootstrap._bootstrap_external = _bootstrap_external
_bootstrap_external.__name__ = 'importlib._bootstrap_external'
_bootstrap_external.__package__ = 'importlib'
try:
    _bootstrap_external.__file__ = __file__.replace('__init__.py', '_bootstrap_external.py')
except NameError:
    # __file__ is not guaranteed to be defined, e.g. if this code gets
    # frozen by a tool like cx_Freeze.
    pass
sys.modules['importlib._bootstrap_external'] = _bootstrap_external

And that worked, ish. Obviously the problem will become more and more complex, and there has to be a more generic solution. But I continued my journey and found that _bootstrap.py is indeed loaded with my print('moo') changes. But when /usr/lib/python37/importlib/_bootstrap.py later calls the function _find_spec, and it does:

    for finder in meta_path:
        with _ImportLockContext():
            find_spec = finder.find_spec
            spec = find_spec(name, path, target)

find_spec will refer to the frozen version again. (Code snippet shortened to the relevant code in _bootstrap.py)

So my question finally ends up in, how do I debug these frozen files? or how do I make it so Python ignores frozen libraries (I can take the performance impact when debugging). Every attempt to find information on this (searching, docs, irc, etc) ends up on "why?" or "don't do this" or it points me to py2exe and how to freeze my libraries. I just want to be able to debug and understand more of how the inner mechanics work by trying things and looking at the variables.

Modifying/debugging frozen builtins like importlib/_boostrap_external.py

0 Answers