How to get a list of all the Python standard library modules?

Viewed 18480

I want something like sys.builtin_module_names except for the standard library. Other things that didn't work:

  • sys.modules - only shows modules that have already been loaded
  • sys.prefix - a path that would include non-standard library modules and doesn't seem to work inside a virtualenv.

The reason I want this list is so that I can pass it to the --ignore-module or --ignore-dir command line options of trace.

So ultimately, I want to know how to ignore all the standard library modules when using trace or sys.settrace.

10 Answers

Building on @Edmund's answer, this solution pulls the list from the official website:

def standard_libs(version=None, top_level_only=True):
    import re
    from urllib.request import urlopen
    if version is None:
        import sys
        version = sys.version_info
        version = f"{version.major}.{version.minor}"
    url = f"https://docs.python.org/{version}/py-modindex.html"
    with urlopen(url) as f:
        page = f.read()
    modules = set()
    for module in re.findall(r'#module-(.*?)[\'"]',
                             page.decode('ascii', 'replace')):
        if top_level_only:
            module = module.split(".")[0]
        modules.add(module)
    return modules

It returns a set. For example, here are the modules that were added between 3.5 and 3.10:

>>> standard_libs("3.10") - standard_libs("3.5")
{'contextvars', 'dataclasses', 'graphlib', 'secrets', 'zoneinfo'}

Since this is based on the official documentation, it doesn't include undocumented modules, such as:

  • Easter eggs, namely this and antigravity
  • Internal modules, such as genericpath, posixpath or ntpath, which are not supposed to be used directly (you should use os.path instead). Other internal modules: idlelib (which implements the IDLE editor), opcode, sre_constants, sre_compile, sre_parse, pyexpat, pydoc_data, nt.
  • All modules with a name starting with an underscore (which are also internal), except for __main__', '_thread', and '__future__ which are public and documented.

If you're concerned that the website may be down, you can just cache the list locally. For example, you can use the following function to create a small Python module containing all the module names:

def create_stdlib_module_names(module_name="stdlib_module_names", variable="stdlibs",
                               version=None, top_level_only=True):
    stdlibs = standard_libs(version=None, top_level_only=True)
    with open(f"{module_name}.py", "w") as f:
        f.write(f"{variable} = {stdlibs!r}\n")

Here's how to use it:

>>> create_stdlib_module_names()  # run this just once
>>> from stdlib_module_names import stdlibs
>>> len(stdlibs)
207
>>> "collections" in stdlibs
True
>>> "numpy" in stdlibs
False

This isn't perfect, but should get you pretty close if you can't run 3.10:

import os
import distutils.sysconfig

def get_stdlib_module_names():
    stdlib_dir = distutils.sysconfig.get_python_lib(standard_lib=True)
    return {f.replace(".py", "") for f in os.listdir(stdlib_dir)}

This misses some modules such as sys, math, time, and itertools.

My use case is logging which modules were imported during an app run, so having a rough filter for stdlib modules is fine. Also I return it as a set rather than a list so membership checks are faster.

This works on Anaconda on Windows, and I suspect it will work on Linux distros.

It goes to your Anaconda directory, e.g.: C:\Users\{user}\anaconda3\Lib, where standard libraries are installed. It then pulls folder names and filenames (dropping extensions).

import sys
import os

standard_libs = []
standard_lib_path = os.path.join(sys.prefix, "Lib")
for file in os.listdir(standard_lib_path):
    standard_libs.append(file.split(".py")[0].strip().lower())

NB: Builtins, viewable via print(dir(__builtins__)), are automatically loaded, whereas standard libs are not.

Related