Add a subdirectory to a python namespace

Question

I would like to be able to import a python module which is actually located in a subdirectory of another module.

I am developing a framework with plug-ins. Since I'm expecting to have a few thousands (there's currently >250 already) and I don't want one big directory containing >1000 files I have them ordered in directories like this, where they are grouped by the first letter of their name:

framework\
    __init__.py
    framework.py
    tools.py
    plugins\
        __init__.py
        a\
            __init__.py
            atlas.py
            ...
        b\
            __init__.py
            binary.py
            ...
        c\
            __init__.py
            cmake.py
        ...

Since I would not like to impose a burden on developers of other plugins, or people not needing as many as I have, I would like to put each plugin in the 'framework.plugins' namespace. This way someone adding a bunch of private plugins can just do so by adding them in the folder framework.plugins and there provide a __init__.py file containing:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

however, currently this setup is forcing them to also use the a-z subdirectories. Sometimes a plugin is extending another plugin, so now I have a

from framework.plugins.a import atlas

and I would like to have

from framework.pugins import atlas

Is there any way to declare a namespace where the full name space name actually doesn't map to a folder structure?

I am aware of the pkg_resources package, but this is only available via setuptools, and I'd rather not have an extra dependency.

import pkg_resources
pkg_resources.declare_namespace(__name__)

The solution should work in python 2.4-2.7.3

update: Combining the provided answers I tried to get a list of all plugins imported in the __init__.py from plugins. However, this fails due to dependencies. Since a plugin in the 'c' folder tries to import a plugin starting with 't', and this one has not been added yet.

plugins = [ x[0].find_module(x[1]).load_module(x[1]) for x in pkgutil.walk_packages([ os.path.join(framework.plugins.__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ],'framework.plugins.' ) ]

I'm not sure If I'm on the right track here, or just overcomplicating things and better write my own PEP302 importer. However, I can't seem to find any decent examples of how these should work.

Update: I tried to follow the suggesting of wrapping the __getattr__ function in my __init__.py, but this seems to no avail.

import pkgutil
import os
import sys
plugins = [x[1] for x in pkgutil.walk_packages([ os.path.join(__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ] )]
import types
class MyWrapper(types.ModuleType):
    def __init__(self, wrapped):
            self.wrapped = wrapped

    def __getattr__(self, name):
           if name in plugins:
                   askedattr =  name[0] + '.' + name
            else:
                    askedattr = name
            attr = getattr(self.wrapped, askedattr)
            return attr


sys.modules[__name__] = MyWrapper(sys.modules[__name__])

Add a subdirectory to a python namespace

4 Answers