Undoing a decade of singleton pattern and class-level configuration

Question

Overview

I need to duplicate a whole inheritance tree of classes. Simply deep-copying the class objects does not work; a proper factory pattern involves a huge amount of code changes; I'm not sure how to use metaclasses to accomplish this.

Background

The software I work on implements support for specialized external hardware, connected to the host computer via USB. Many years ago, it was assumed that there would only ever be one type of hardware in use at a time. Consequently, the hardware object is used as a singleton. Along the years, secondary classes were configured based on the currently active hardware class.

At the moment, it is impossible to use this library with two types of hardware at the same time, since the classobjects cannot be configured for both hardware at the same time.

In recent years, we have avoided this issue by creating one python process for each hardware, but this is becoming untenable.

Here is an extremely simplified example of the architecture:

# ----------
# Hardware classes
class HwBase():
    def customizeComponent(self, compDict):
        compDict['ComponentBase'].hardware = self

class HwA(HwBase):
    def customizeComponent(self, compDict):
        super().customizeComponent(compDict)
        compDict['AnotherComponent'].prop.configure(1,2,3)

class HwB(HwBase):
    def customizeComponent(self, compDict):
        super().customizeComponent(compDict)
        compDict['AnotherComponent'].prop.configure(4,5,6)


# ----------
# Property classes
class SpecialProperty(property):
    def __init__(self, fvalidate):
        self.fvalidate = fvalidate
        # handle fset, fget, etc. here.
        # super().__init__()

# ----------
# Component classes
class ComponentBase():
    hardware = None

    def validateProp(self, val):
        return val < self.maxVal

    prop = SpecialProperty(fvalidate=validateProp)


class SomeComponent():
    """Users directly instantiate and use this compoent via an interactive shell.
    This component does complex operations with the hardware attribute"""

    def validateThing(self, val):
        return isinstance(val, ComponentBase)

    thing = SpecialProperty(fvalidate=validateThing)


class AnotherComponent():
    """Users directly instantiate and use this compoent via an interactive shell
    This component does complex operations with the hardware attribute"""
    maxVal = 15


# ----------
# Initialization



def initialize():
    """ This is only called once perppython instance."""
    #activeCls = HwA
    activeCls = HwB

    allComponents = {
            'ComponentBase': ComponentBase,
            'SomeComponent': SomeComponent,
            'AnotherComponent': AnotherComponent
    }

    hwInstance = activeCls()
    hwInstance.customizeComponent(allComponents)

    return allComponents

components = initialize()

# ----------
# User code goes here
someInstance1 = components['SomeComponent']()
someInstance2 = components['SomeComponent']()

someInstance1.prop = 10
someInstance2.prop = 10

The overarching goal would be to interact with both HwA and HwB at the same time. Since most interactions are done via components instead of the Hw objects themselves, I believe the solution involves having multiple versions of the components, e.g.: two separate inheritance trees, for a total of 6 final components, one tree/set configured for each hardware. This is what I need help with.

Potential solutions

Consider that I have around tens different hardware do configure for. Furthermore, there are hundreds of different leaf components classes, with many extra bases and mixin classes.

Move all configuration steps in the component's init method

Not possible due to the use of properties; these need to be set on the class.

Deepcopy the classobjects

Copy all classobjects, swap in the appropriate __bases__. Mutable class variables need to be carefully handled. However, I'm not sure how to deal with properties for this, since classbody references within the property objects (such as fvalidate) need to be updated to that of the copied class.

This requires a significant amount of manual intervention to work. Not impossible, but prone to breaking in the long term.

Factory pattern

Wrap all component definition in a factory function:

def ComponentBaseFactory(hw):
    class SomeComponent(cache[hw].ComponentBase):
        pass

and have some sort of component cache which would handle creating all classobjects during initialize()

This is what I consider the most architecturally-correct option available. Since the class body is re-executed on every factory call, the attributes of the properties will reference the appropriate class object.

Downside: huge code footprint. I am familiar with doing codebase-wide changes via sed or python scripts, but this would be quite a lot.

Add metaclasses on components

I am not sure how to proceed for this. Based on the python data model (py3.7), the following happens at class creation (which happens right after the class definition indentation ends):

MRO entries are resolved;
the appropriate metaclass is determined;
the class namespace is prepared;
the class body is executed;
the class object is created.

I would need to redo these steps after the class has been defined (like a factory function!), but i'm not sure how to redo step 4. Specifically, the python documentation states in section 3.3.3.5 that the class body is executed as with a "special?" form of the exec() builtin. How can I re-exec the class body with a different set of locals/globals? Even if I access the class body's code with inspect shenanigans, i'm not sure i'll be able to reproduce the module environment properly.

Even if I mess with __prepare__ and __new__, I don't see how I can fix the cross-references introduced in the class code block regarding the property instantiation.

Components as metaclasses

A metaclass is a class factory, just like a class is an object factory. SomeComponent and AnotherComponent could be declared as metaclasses, then get instantiated with the Hw object during initialize():

SomeComponent = SomeComponentMeta(hw)

This is similar to the factory pattern, but would also require quite a few code changes: a lot of class code would have to be moved to the metaclass __init__.

user_108205 · Answer

I'd have to spend a lot more of time here to proper understand what you need, but if your "TL;DR" of executing the class body with different globals/nonlocal variables is the bottom line, the factory approach is a very clean and readable way, as you had considered.

At first, I don't think a metaclass could be a good approach here - although it could be used to customize your special properties (in my first read, I could not figure out what they actually do, and how they should differ between your final classes). If the function as a class factory can specialize your properties, it would work nonetheless.

If what you need is that the properties are independent for Hwa and HwB like in accessing a different list object in HwA than is accessed in HwB, yes, a metaclass could take care of that, by automatically recreating any properties when creating a subclass (so that the property objects themselves are not shared with the supper-classes and across the hierarchy). If that i what you need, leave a comment, I can write some proof of concept code.

Anyway, it is possible to create a metaclass that, upon instantiating a subclass, will look upon the hierarchy for all SpecialProperty and create new-instances of those for the subclass - so that a base value set on a superclass remains valid for the subclasses, but when configuration runs, each class will have an independent configuration. (as it turns out, no metaclass is needed: we are covered by __init_subclass__ )

Another thing to take care of is that subclassses of property cannot be simply copies with Python's copy.copy (tested empirically), so we need a way to create reliable copies of those. I include one function bellow, but it might need to be improved to work with the actual SpecialProperty class.

from copy import copy

def copy_property(prop):
    cls = prop.__class__
    new_prop = cls.__new__(cls)
    # Initialize the attributes that can't be set from Python code, inplace:
    property.__init__(new_prop, prop.fget, prop.fset, prop.fdel) 
    if hasattr(prop, "__dict__"): # only exists for subclasses of property
        # Possible adaptation needed: it may be that for some attributes of
        # SpecialProperty, a deepcopy would be needed.
        # But for the given example attribute of "fvalidate" a simple copy is better:
        new_prop.__dict__ = copy(prop.__dict__)
    return new_prop



    
# Python 3.6 introduced `__init_subclass__` which is called at subclass _creation_
# time. With it, the logic can be inserted in ComponentBase and there is no need for
# a metaclass.
    
class ComponentBase():
    
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        for attrname in dir(cls):
            attr = getattr(cls, attrname)
            if not isinstance(attr, SpecialProperty):
                continue
            new_prop = copy_property(attr)
            setattr(cls, attrname, new_prop)
        
    
    
    hardware = None
    ...

As you see- theres some workarounds that had to be done because your project opted for subclassing property. I am leaving this remark here as a remainder that unless property fits one exact needs, it is more clean to write a new class implementing the Descriptor Protocol - just by implementing __set__, __get__ and __delete__ directly.