Why are so many modules initially loaded in CPython?

llll0 0
Viewed 93

The python docs state:

A complete Python program is executed in a minimally initialized environment: all built-in and standard modules are available, but none have been initialized, except for sys (various system services), builtins (built-in functions, exceptions and None) and __main__.

This would suggest that only those three modules should be listed as loaded modules with the following code snippet:

import sys
print(sys.modules.keys())

However, running the code snippet using CPython v3.10 (with -S option) returns the following on my PC:

dict_keys(['sys', 'builtins', '_frozen_importlib', '_imp', '_thread', '_warnings', '_weakref', '_io', 'marshal', 'nt', 'winreg', '_frozen_importlib_external', 'time', 'zipimport', '_codecs', 'codecs', 'encodings.aliases', 'encodings', 'encodings.utf_8', 'encodings.cp1252', '_signal', '_abc', 'abc', 'io', '__main__'])

Why are there 22 extra modules loaded at runtime as compared to the "minimally initialized environment" mentioned in the docs?

I am updating my understanding of CPython's extra loaded modules with my own answer below.

cpython python-module python
1 Answers

What I have found so far:

The majority of the extra modules are active to provide the import keyword functionality, and for text encodings.

  • During interpreter initialisation, the import functionality is provided by importing _frozen_importlib and _imp (during pycore_interp_init, within init_importlib)
  • _setup() in importlib's _bootstrap.py then imports _thread, _warnings, and _weakref because they are builtin modules that are explicitly imported during bootstrap, and hence not really extra modules.
  • Interpreter initialisation then imports _frozen_importlib_external (during init_interp_main, within init_importlib_external)
  • importlib's _bootstrap_external.py imports four new module dependencies: _io, marshal, nt, and winreg packages. If not on Windows, posix gets imported rather than nt and winreg.
    • _io is imported because it is a builtin module explicitly imported during bootstrap, and hence not really an extra module.
    • marshal is imported because it is used to load/dump bytecode from/to .pyc files.
    • nt/posix is imported because it is used for operating system functions such as reading the current working directory.
    • winreg is imported because it is used to find modules declared in the windows registry.
  • As part of importing _frozen_importlib_external, the interpreter initialization then imports zipimport, presumably to allow for opening zip-format python archives
    • As part of importing zipimport, the only new dependency is the time module which is imported. The only use is time.mktime() to "convert the date/time values found in the Zip archive to a value that's compatible with the time stamp stored in .pyc files"
  • After _frozen_importlib_external (and thus after import keyword functionality is sorted), the interpreter initialization then imports encodings, presumably for decoding source text.
    • encodings.aliases is imported because it provides a dictionary of names to map to known encodings.
    • codecs is imported as it is a dependency of encodings
      • _codecs is imported, presumably because it is the C version of codecs?
    • encodings.utf_8 is then imported, presumably because it is the default encoding.
    • Because we are on Windows, encodings.cp1252 is also imported (encodings.latin_1 is imported instead if on Linux).
  • The interpreter initialization then imports _signal, presumably for the interpreter to deal with signal handling.
  • io is then fully imported, presumably to open source files?
    • abc is then imported as it is a dependency on io?
      • _abc is then imported, presumably because it is the C version of codecs?
  • (On Linux, readline is also imported)

And thus 22 extra 'modules' are loaded when using CPython.

llll00
Related