how do exec and eval add __builtins__ to a given environment?

Question

I'm trying to understand how eval and exec treat the environment (globals and locals) that they are given, so I made a class "logdict" which behaves like a dict but logs most methods (__new__ is excluded):

from functools import wraps

class LogDict(dict):
    logs = {}
    def _make_wrapper(name):
        @wraps(getattr(dict, name))
        def wrapper(self, *args, **kwargs):
            LogDict.logs.setdefault(id(self), []).append({
                'name': name,
                'args': tuple(map(repr, args)),
                'kwargs': dict((key, repr(kwargs[key])) for key in kwargs)
                })
            return getattr(super(), name)(*args, **kwargs)
        return wrapper

    for attr in dir(dict):
        if callable(getattr(dict, attr)) and attr not in {'__new__',}:
            locals()[attr] = _make_wrapper(attr)

    def logrepr(self):
        return ''.join(
            "{fun}({rargs}{optsep}{rkwargs})
".format(
                fun = logitem['name'],
                rargs = ', '.join(logitem['args']),
                optsep = ', ' if len(logitem['kwargs'])>0 else '',
                rkwargs = ', '.join('{} = {}'\
                 .format(key, logitem['kwargs'][key]) for key in logitem['kwargs'])
                )
            for logitem in LogDict.logs[id(self)])

as an example, this code:

d = LogDict()
d['1'] = 3
d['1'] += .5
print('1' in d)
print('log:')
print(d.logrepr())

produces this output:

True
log:
__init__()
__setitem__('1', 3)
__getitem__('1')
__setitem__('1', 3.5)
__contains__('1')
__getattribute__('logrepr')

I tried feeding this to exec in order to understand how it was being used, but I can't see it accessing the dictionary beyond what makes sense:

print('	Testing exec(smth, logdict):')
d = LogDict()
exec('print("this line is inside the exec statement")', d)
print('the log is:')
print(d.logrepr(), end='')
print('the env now contains:')
print(d)

    Testing exec(smth, logdict):
this line is inside the exec statement
the log is:
__init__()
__getitem__('print')
__getattribute__('logrepr')
the env now contains:
[a dictionary containing __builtins__]

so the exec function didn't call any of the methods I'm logging except __getitem__ to see if 'print' was in it (__getattribute__ is called later when I print the log); how did it set the key '__builtins__' (or check that it wasn't already defined)? Am I just missing the method it's using, or is it doing something more low-level?

user_1405065 · Answer

The exec function uses low-level dictionary functions in the Python C API to insert the __builtins__ module into the global namespace dictionary. You can see the call in the CPython source code.

Because the call is to low level dict API, it doesn't look in your class to find your overridden __setitem__ method, it just directly writes into the underlying dictionary storage. The exec function requires that the global namespace passed in to it is a dict (or a dict subclass, but not some other mapping type), so this is always safe, at least in terms of not crashing the interpreter. But it does bypass your logging.

Unfortunately, I don't see any way to get logging added so that you can see __builtins__ get added to the global namespace. That probably means your attempt to directly observe exec's behavior is doomed. But perhaps reading the C source code is a suitable alternative, if you're just trying to understand what it does. One of the perks of using an open source programming language is that you can just go look up how the interpreter is programmed when you have questions like this. It does require reading C, rather than just Python, but the builtin_exec_impl function is straight forward enough (the actual code execution happens elsewhere and is surely much more complicated).

how do exec and eval add builtins to a given environment?

1 Answers