Python metaclasses handling double class creation with same `__classcell__` attribute

user_1953800 0
Viewed 65

I wrote a metaclass to handle the creation of two classes at class definition (with one accessible as an attribute of the other).

MRE with logic removed:

class MyMeta(type):

    def __new__(mcs, clsname, bases, namespace, **kwds):
        result = super().__new__(mcs, clsname, bases, namespace)
        
        # Omitted logic to prefix namespace

        print(namespace)
        # namespace.pop('__classcell__', None)
        disambiguated_cls = super().__new__(mcs, clsname + 'Prefixed', bases, namespace)

        result.disambiguated_cls = disambiguated_cls
        return result

    
class A(metaclass=MyMeta):
    
    @classmethod
    def f(cls, x):
        return x + 1

class B(A):
    
    @classmethod
    def f(cls, x):
        # return A.f(x)
        return super().f(x)

Anytime super is called (e.g. in class B definition) a __classcell__ object is added to the namespace. If that __classcell__ object is duplicated (e.g. in the creation of disambiguated_cls) the following exception is raised:

TypeError: __class__ set to <class '__main__.B'> defining 'B' as <class '__main__.B'>

Looks like there are two solutions (corresponding to the two commented lines in the example):

  • Don't call super (just directly call function on the superclass: A.f(x))
  • Pop the __classcell__ out of the namespace

The first means manual resolution of superclasses and I'm not sure what downstream effects the latter can have.

For reference the docs on __classcell__ in metaclasses state:

In CPython 3.6 and later, the __class__ cell is passed to the metaclass as a __classcell__ entry in the class namespace. If present, this must be propagated up to the type.__new__ call in order for the class to be initialised correctly. Failing to do so will result in a RuntimeError in Python 3.8.

Is there another option? What is the best way to do this?

python-3.8 metaclass python-3.x python
1 Answers

It feels like the simpler way is to partially create a new "blank" class with a __classcell__ and populate it with the namespace of the original class before calling type.__new__ for the second class.

If that works, it might be worth using it - otherwise, if there would be no multiple inheritance or complicated hierarchy involved, hard-coding the call to the superclass might work. However - it seems to be that hardcoding the call is impossible, since calling the hard-coded method inside the "disambiguated" class would call the outer ("result") class in the parent.

One point to take care is that all functions (including functions decorated with @classmethod) have their closures bound to the value provided in the original __classcell__. So, in order to have the same functions to participate as methods in a second class they have to be rebound to a fresh __classcell__ value, one created along the second class. Rebindind a function closure is not that straightforward, since functions are immutable. The way to do it is to create a new "function object" with the __code__ and other attributes of the original function, passing in a new closure. Once that is done, the rest is straightforward

from types import FunctionType

class MyMeta(type):

    def __new__(mcls, clsname, bases, namespace, **kwds):
        result = super().__new__(mcls, clsname, bases, namespace)
        
        # Omitted logic to prefix namespace


        print(namespace)

        # the upside of having "nested_process" here instead of
        # inside the other metaclass is transparent
        # read access to the attributes of the original class:
        def nested_process(stubname, stubbases, stub_namespace):
            new_namespace = namespace.copy()
            classcell = stub_namespace["__classcell__"]
            new_namespace["__classcell__"] = classcell
            for m_name, method in new_namespace.items():
                is_classmeth = False
                if isinstance(method, classmethod):
                    is_classmeth = True
                    method = method.__func__
                if not isinstance(method, FunctionType):
                    continue
                if not (free:=method.__code__.co_freevars) or free[0] != "__class__":
                    continue
                method = FunctionType(
                    method.__code__, method.__globals__, method.__name__,
                    method.__defaults__, 
                    closure=(classcell,)
                )
                if is_classmeth:
                    method = classmethod(method)
                new_namespace[m_name] = method

            # maybe modify namespace __qualname__. 
            # maybe rewrite "bases" to point to prefixed versions?
            return clsname + "Prefixed", bases, new_namespace

        class disambiguated_cls(metaclass=NestedMeta, clone_hook=nested_process):
            def _stub(self):
                _ = __class__  

        result.disambiguated_cls = disambiguated_cls
        return result

class NestedMeta(MyMeta):
    # the new syntethic class need this modified __new__. 
    # but as it also inherits from a class created by 
    # "MyMeta", this metaclass must inherit from it
    # otherwise "metaclass conflict" is raised.
    def __new__(mcls, clsname, bases, namespace, clone_hook):
        clsname, bases, namespace = clone_hook(clsname, bases, namespace)
        # surprise: we have to hardcode "type" here:
        return type.__new__(mcls, clsname, bases, namespace)
        # if "super()" is desired, than an extra KWD parameter
        # can be passed to "MyMeta.__new__" so it skip its nesting+prefixed logic

    # ensure a eventual __init__ from MyMeta is not called:
    __init__ = lambda *args, **kw: None


    
class A(metaclass=MyMeta):
    
    @classmethod
    def f(cls, x):
        return x + 1

class B(A):
    
    @classmethod
    def f(cls, x):
        return super().f(x)
user_1082050
Related