How to re-export everything from a star (*) import in a python __init__.py file

Viewed 54

When I write a python library, I often end up with something like this:

/my_lib
   __init__.py
   module_a.py
   module_b.py
   ...

Now lets suppose that module_a and module_b are very big, and define a lot public and non-public methods. E.g.:

import ...
from ... import ...
... etc

__all__ = [
    "PublicClass",
    "public_fn",
    "PUBLIC_STATIC_VAR",
    "OtherPublicClass",
    ...
]

class _PrivateClass:
    ...

I put a lot of effort in curating a well thought out list of symbols to export in __all__. Now in my __init__.py I would like to do this (which does not work of course):

from my_lib.module_a import * as module_a
from my_lib.module_b import * as module_b
from third_party_lib import * as third_party
from other_lib import OtherA, OtherB

__all__ = [
    *module_a,
    *module_b,
    *third_party,
    "OtherA",
    "OtherB",
]

In essence, what I want is that, if someone imports my_lib, they can access all public symbols from module_a, module_b and third_party from the same namespace. I.e.:

import my_lib

obj1 = my_lib.pub_fn_from_module_a()
obj2 = my_lib.PubClassFromModuleB()
obj3 = my_lib.CONSTANT_FROM_THIRD_PARTY_LIB
obj4 = my_lib.OtherA()
obj5 = my_lib.OtherB()

Poor solutions:

1. Implicit __all__

I know I can do:

from my_lib.module_a import *
from my_lib.module_b import *
from third_party_lib import *
from other_lib import OtherA, OtherB

# __all__ is not defined in __init__.py

This will implicitly define __all__ and achieves the desired behavior, but it is not possible for static code analysis tools to distinguish this from dead code. For example pre-commit hooks or code formatting tools may delete these imports.

2. Re-type everything

I can also do:

from my_lib.module_a import (
    # <type all public symbols from module_a here>
)
from my_lib.module_b import (
    # <type all public symbols from module_b here>
)
from third_party_lib import (
    # <type all public symbols from third_party_lib here>
)
from other_lib import OtherA, OtherB

__all__ = [
    # <copy-paste the body of module_a.__all__ here>
    # <copy-paste the body of module_b.__all__ here>
    # <type all public symbols from third_party_lib here>
    "OtherA",
    "OtherB",
]

But now I basically have to retype the name of each symbol in module_a three times, once in module_a.__all__, once in the import of __init__.py and once in the __all__ definition in __init__.py. The same goes for module_b, I also need make sure al 3 remain in sync during development.

3. re-use __all__

This kind of works, but looks really awkward:

from my_lib.module_a import * as module_a
from my_lib.module_b import * as module_b
from my_lib import module_a, module_b
from third_party_lib import * as third_party
from other_lib import OtherA, OtherB

# only purpose of `_keep` is to fake the usage of a few symbols 
# such that the star imports are not flagged as dead code.
# It does nothing at runtime.
def _keep(x): pass

_keep(random_symbol_from_module_a)
_keep(random_symbol_from_module_b)
_keep(random_symbol_form_third_party)

__all__ = [
    *module_a.__all__,
    *module_b.__all__,
    *third_party_lib.__all__,
    "OtherA",
    "OtherB",
]

The star imports (from ... import *) are required because e.g. pycharm or other static code analysis tools can't infer what the *module_a.__all__ statement does. So without the star imports, static code analysis tools will report missing imports everywhere.

0 Answers
Related