Modifying attribute access for the __main__ module (the details of name resolution)

Viewed 1240

Information collected from the docs

Concerning name resolution the documentation is not perfectly clear. It uses the terms scope and namespace but is not precise about how they come into effect and when exactly a NameError is raised:

When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block’s environment.

When a name is not found at all, a NameError exception is raised.

This however doesn't explain where exactly the name is searched for. Concerning namespaces we get the following information:

Names are resolved in the top-level namespace by searching the global namespace, i.e. the namespace of the module containing the code block, [...]

And further, regarding __main__:

The namespace for a module is automatically created the first time a module is imported. The main module for a script is always called __main__.

This part of the docs further states that

'__main__' is the name of the scope in which top-level code executes.

Relevant code

Combining the above statements, I suppose that whenever a name is to be resolved in the "top-level script environment" ("top-level namespace") this happens happens by checking sys.modules['__main__'] (similar to how attribute access for modules works and how it can modified, as pointed out by PEP 562). However the following snippet indicates that this is not the case:

import sys

class Wrapper:
    def __init__(self):
        self.main = sys.modules['__main__']

    def __getattr__(self, name):
        try:
            return getattr(self.main, name)
        except AttributeError:
            return 'Fallback for "{}"'.format(name)

sys.modules['__main__'] = Wrapper()
print(undefined)

which raises NameError: name 'undefined' is not defined.

On the other hand we can add names by modifying sys.modules['__main__'].__dict__ or using setattr:

import sys

# Either ...
sys.modules['__main__'].__dict__['undefined'] = 'not anymore'
# Or ...
setattr(sys.modules['__main__'], 'undefined', 'not anymore')

print(undefined)  # Works.

So I suspected that maybe it's the module's __dict__ attribute (or equivalently __builtins__.globals) which is checked directly, sidestepping getattr on the module object. Extending the above example however shows that this is not the case:

import sys

class Wrapper:
    def __init__(self):
        self.main = sys.modules['__main__']

    def __getattr__(self, name):
        try:
            return getattr(self.main, name)
        except AttributeError:
            return 'Fallback for "{}"'.format(name)

    @property
    def __dict__(self):
        class D:
            def __contains__(*args):
                return True

            def __getitem__(__, item):
                return getattr(self, item)

        return D()

sys.modules['__main__'] = Wrapper()
sys.modules['builtins'].globals = lambda: sys.modules['__main__'].__dict__
print(globals()['undefined'])  # Works.
print(undefined)               # Raises NameError.

Questions

  1. What is the exact definition of scopes and namespaces?
  2. How are names exactly resolved (what steps are taken and which resources are checked in order to determine if a name exists)?
  3. In what way does name resolution involve scopes and namespaces?
  4. Why does the above example using Wrapper fail (while it does work on "general" module attribute access, as per PEP 562)?
1 Answers

Your's is a very interesting question, since I don't have a clear answer let's run some experiments.

First let's change your code a bit:

# file main.py
import sys
print(sys.modules['__main__'])
class Wrapper:
    def __init__(self):
        self.main = sys.modules['__main__']

    def __getattr__(self, name):
        try:
            return getattr(self.main, name)
        except AttributeError:
            return 'Fallback for "{}"'.format(name)

sys.modules['__main__'] = Wrapper()
print(sys.modules['__main__'])
print(undefined)

It will print

<module '__main__' from 'main.py'>
<__main__.Wrapper object at 0x000001F87601BE48>
Traceback (most recent call last):
  File "main.py", line 15, in <module>
    print(undefined)
NameError: name 'undefined' is not defined

So we still have __main__ as a module here, and Wrapper class is inside it.

Docs says:

A module’s __name__ is set equal to __main__ when read from standard input, a script, or from an interactive prompt.

So that means our sys.modules['__main__'] = Wrapper() line is meant to subsitute an already loaded module, with something from inside that module (!!).

OTOH, importing main.py from the REPL (the other case where a __main__ module is created), totally messes with everything so some substitution is happening at that time.

Summarizing:

As far as I can see, it requires some deep dark magic to change __main__ from inside a running module, maybe if we use importlib.reload and mess with the cached modules?

Doing that from other module seems ok, but (the example) messes with things, and name resolution breaks, i.e the Wapper class doesn't resolve previous names as you believe it should.

PD.

Sorry if this is not the experienced answer you wanted and seems more like a commentary. I did it as an experiment to test your hypothesis and maybe find some results.

Related