Test discovery drops Python namespaces from relative imports?

Problem: Overlapping sys.path entries

The duplicate imports with different module names happen when you have overlapping sys.path entries: that is, when sys.path contains both a parent and child directory as separate entries. This situation is almost always an error: it will make Python see the child directory as a separate, unrelated root for imports, which leads surprising behaviour.

In your example:

$ python -m unittest discover src/namespace/
namespace.testcase.a
testcase.sub.b
testcase.a

This means that both src and src/namespace ended up in sys.path, so that:

• namespace.testcase.a was imported relative to src
• testcase.sub.b and testcase.a were imported relative to src/namespace

Why?

In this case, the overlapping sys.path entries happen because unittest discover is trying to be helpful: it defaults to assuming that the start directory for test discovery is also the top-level directory that your imports are relative to, and it will insert that top-level directory into sys.path if it's not already there, as a convenience. (…not so convenient, it turns out. ️)

Solution: Explicitly specify the correct top-level directory

You can explicitly specify the correct top-level directory with -t (--top-level-directory):

python -m unittest discover -t src -s src/namespace/

This will work as before, but won't treat src/namespace as a top-level directory to insert into sys.path.

Side note: The -s option prefix for src/namespace/ was implicit in the previous example: the above just makes it explicit. (unittest discover has weird positional argument handling: it treats its first three positional arguments as values for -s, -p, and -t, in that order.)

Details

The code responsible for this lives in unittest/loader.py:

class TestLoader(object):

    def discover(self, start_dir, pattern='test*.py', top_level_dir=None):

        ...

        if top_level_dir is None:
            set_implicit_top = True
            top_level_dir = start_dir

        top_level_dir = os.path.abspath(top_level_dir)

        if not top_level_dir in sys.path:
            # all test modules must be importable from the top level directory
            # should we *unconditionally* put the start directory in first
            # in sys.path to minimise likelihood of conflicts between installed
            # modules and development versions?
            sys.path.insert(0, top_level_dir)

        ...

Not sure exactly why unittest wouldn't respect your setup.py, but indeed often it does not (maybe a bug, or a difficulty in doing so for the implementers). Or perhaps unittest is by design very "low level" and does not come with any bells or whistles you'd expect from something like pytest.

What you need to do is help unittest out and tell it where your package starts, use the --top-level-directory option for that (or -t for short).

This should work as you expect:

python -m unittest discover -t src/ src/namespace/

The issue is that you probably have something like this in your setup.py:

    package_dir={"": "src"},

And unfortunately unittest is not "smart enough" to figure that out.

This is one example detail why I strongly prefer pytest to std-lib's unittest :)

pytest will go to greater lengths to "do the right thing", while not forcing you to be verbose in your test run invocation (for example: it auto-discovers recursively by default etc).

If you want to learn more about how unittest imports things, you can add this line to your a.py file:

assert __package__ == "namespace.testcase"

Then, run your test without the -t src/ as you originally did -> you will see exactly where unittest is crashing. If you open that code, you will see that all it does is try to simply __import__(name), where name is simply the thing it just found that could look like a test.

Tests are usually NOT in a package, a more strict project layout would be like:

src/namespace/ # -> your project or lib
tests/         # -> your tests

The above is "more strict" because it makes it harder to confuse your tests with your actual shipped code (ie: no oopsie import ..tests.foo from the actual code).

Now, given this, a lot of testing tools like unittest and pytest, will kind of assume that your tests don't really have a package, so they will import them as-if the package doesn't matter at all...

Ie: they won't necessarily try and import test_foo.py as-if it was under your main top-level name.

So, in theory you should (from my experience writing tests):

• use relative imports from within your actual code only (ie: any non-test submodule)
• use full absolute import from the tests (that simplifies quite a few things for testing tools + it allows to treat your code "less intimately" from the tests -> kinda forces you to import stuff from your namespace project like any other user would do)

Hope that helps. I don't have handy links to docs on this (and maybe it would be worth a good book). But consider this: if you write this from your test:

from ..sub import b

You are taking shortcuts a user of your library cannot do. Anyone who would pip install namespace for example would have to import b with an absolute import:

from namespace.sub import b

It is helpful I find to isolate tests from the code itself. I know many projects do just add a tests/ subfolder to their main code tree, but I do find that odd, since that ships the tests together with the published package, and one could technically import the tests just like the rest of the code... for example:

from namespace.testcase.tests import test_imports

An example of tests/ outside the main code tree is the requests package.

Followed the code, as this got me curious.

unittest discover looks for test cases, it finds testcase/ which looks like a test folder to it. So it simply does a "standalone" (ie: regardless of any "top-level" context) import testcase.

Then your test does this (all of these imports are simply cached in sys.modules, by name):

• from namespace.testcase import a, which triggers the import of a as a submodule of namespace.testcase as expected
• but then it calls from ..sub import b, now in unittest's context, this expands to testcase.sub.b, which then leads to the confusion.