When I write a python library, I often end up with something like this:
/my_lib
__init__.py
module_a.py
module_b.py
...
Now lets suppose that module_a and module_b are very big, and define a lot public and non-public methods. E.g.:
import ...
from ... import ...
... etc
__all__ = [
"PublicClass",
"public_fn",
"PUBLIC_STATIC_VAR",
"OtherPublicClass",
...
]
class _PrivateClass:
...
I put a lot of effort in curating a well thought out list of symbols to export in __all__. Now in my __init__.py I would like to do this (which does not work of course):
from my_lib.module_a import * as module_a
from my_lib.module_b import * as module_b
from third_party_lib import * as third_party
from other_lib import OtherA, OtherB
__all__ = [
*module_a,
*module_b,
*third_party,
"OtherA",
"OtherB",
]
In essence, what I want is that, if someone imports my_lib, they can access all public symbols from module_a, module_b and third_party from the same namespace. I.e.:
import my_lib
obj1 = my_lib.pub_fn_from_module_a()
obj2 = my_lib.PubClassFromModuleB()
obj3 = my_lib.CONSTANT_FROM_THIRD_PARTY_LIB
obj4 = my_lib.OtherA()
obj5 = my_lib.OtherB()
Poor solutions:
1. Implicit __all__
I know I can do:
from my_lib.module_a import *
from my_lib.module_b import *
from third_party_lib import *
from other_lib import OtherA, OtherB
# __all__ is not defined in __init__.py
This will implicitly define __all__ and achieves the desired behavior, but it is not possible for static code analysis tools to distinguish this from dead code. For example pre-commit hooks or code formatting tools may delete these imports.
2. Re-type everything
I can also do:
from my_lib.module_a import (
# <type all public symbols from module_a here>
)
from my_lib.module_b import (
# <type all public symbols from module_b here>
)
from third_party_lib import (
# <type all public symbols from third_party_lib here>
)
from other_lib import OtherA, OtherB
__all__ = [
# <copy-paste the body of module_a.__all__ here>
# <copy-paste the body of module_b.__all__ here>
# <type all public symbols from third_party_lib here>
"OtherA",
"OtherB",
]
But now I basically have to retype the name of each symbol in module_a three times, once in module_a.__all__, once in the import of __init__.py and once in the __all__ definition in __init__.py. The same goes for module_b, I also need make sure al 3 remain in sync during development.
3. re-use __all__
This kind of works, but looks really awkward:
from my_lib.module_a import * as module_a
from my_lib.module_b import * as module_b
from my_lib import module_a, module_b
from third_party_lib import * as third_party
from other_lib import OtherA, OtherB
# only purpose of `_keep` is to fake the usage of a few symbols
# such that the star imports are not flagged as dead code.
# It does nothing at runtime.
def _keep(x): pass
_keep(random_symbol_from_module_a)
_keep(random_symbol_from_module_b)
_keep(random_symbol_form_third_party)
__all__ = [
*module_a.__all__,
*module_b.__all__,
*third_party_lib.__all__,
"OtherA",
"OtherB",
]
The star imports (from ... import *) are required because e.g. pycharm or other static code analysis tools can't infer what the *module_a.__all__ statement does. So without the star imports, static code analysis tools will report missing imports everywhere.