Way to exclude directories when running make doctest

I have a python library which uses Sphinx to build documentation pages. The library structure is as follows:

Library-name
|- /docs
|  |- <lots of .rst files>
|  |- config.py
|
|- /src
|  |- <python files>
|
|- conftest.py

Some pages defined by .rst files have static Python code which should be tested and we recently converted the code-block directives to testcode directives. Running make doctest successfully tests the code in .rst files, but it also finds Python files under /src and tests the example code in docstrings. The docstring code is already covered by pytest and many of the examples fail with doctest since there are differences in test setup.

I've already tried setting doctest_global_setup in /docs/config.py to the code in conftest.py, but tests are still failing.

Is there a way to ignore certain directories when running make doctest so files in the source folder are ignored?