What are some (concrete) use-cases for metaclasses?

The purpose of metaclasses isn't to replace the class/object distinction with metaclass/class - it's to change the behaviour of class definitions (and thus their instances) in some way. Effectively it's to alter the behaviour of the class statement in ways that may be more useful for your particular domain than the default. The things I have used them for are:

• Tracking subclasses, usually to register handlers. This is handy when using a plugin style setup, where you wish to register a handler for a particular thing simply by subclassing and setting up a few class attributes. eg. suppose you write a handler for various music formats, where each class implements appropriate methods (play / get tags etc) for its type. Adding a handler for a new type becomes:

  class Mp3File(MusicFile):
      extensions = ['.mp3']  # Register this type as a handler for mp3 files
      ...
      # Implementation of mp3 methods go here
  

  The metaclass then maintains a dictionary of {'.mp3' : MP3File, ... } etc, and constructs an object of the appropriate type when you request a handler through a factory function.

• Changing behaviour. You may want to attach a special meaning to certain attributes, resulting in altered behaviour when they are present. For example, you may want to look for methods with the name _get_foo and _set_foo and transparently convert them to properties. As a real-world example, here's a recipe I wrote to give more C-like struct definitions. The metaclass is used to convert the declared items into a struct format string, handling inheritance etc, and produce a class capable of dealing with it.

  For other real-world examples, take a look at various ORMs, like sqlalchemy's ORM or sqlobject. Again, the purpose is to interpret defintions (here SQL column definitions) with a particular meaning.

I have a class that handles non-interactive plotting, as a frontend to Matplotlib. However, on occasion one wants to do interactive plotting. With only a couple functions I found that I was able to increment the figure count, call draw manually, etc, but I needed to do these before and after every plotting call. So to create both an interactive plotting wrapper and an offscreen plotting wrapper, I found it was more efficient to do this via metaclasses, wrapping the appropriate methods, than to do something like:

class PlottingInteractive:
    add_slice = wrap_pylab_newplot(add_slice)

This method doesn't keep up with API changes and so on, but one that iterates over the class attributes in __init__ before re-setting the class attributes is more efficient and keeps things up to date:

class _Interactify(type):
    def __init__(cls, name, bases, d):
        super(_Interactify, cls).__init__(name, bases, d)
        for base in bases:
            for attrname in dir(base):
                if attrname in d: continue # If overridden, don't reset
                attr = getattr(cls, attrname)
                if type(attr) == types.MethodType:
                    if attrname.startswith("add_"):
                        setattr(cls, attrname, wrap_pylab_newplot(attr))
                    elif attrname.startswith("set_"):
                        setattr(cls, attrname, wrap_pylab_show(attr))

Of course, there might be better ways to do this, but I've found this to be effective. Of course, this could also be done in __new__ or __init__, but this was the solution I found the most straightforward.

Let's start with Tim Peter's classic quote:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why). Tim Peters (c.l.p post 2002-12-22)

Having said that, I have (periodically) run across true uses of metaclasses. The one that comes to mind is in Django where all of your models inherit from models.Model. models.Model, in turn, does some serious magic to wrap your DB models with Django's ORM goodness. That magic happens by way of metaclasses. It creates all manner of exception classes, manager classes, etc. etc.

See django/db/models/base.py, class ModelBase() for the beginning of the story.

Metaclasses can be handy for construction of Domain Specific Languages in Python. Concrete examples are Django, SQLObject 's declarative syntax of database schemata.

A basic example from A Conservative Metaclass by Ian Bicking:

The metaclasses I've used have been primarily to support a sort of declarative style of programming. For instance, consider a validation schema:

class Registration(schema.Schema):
    first_name = validators.String(notEmpty=True)
    last_name = validators.String(notEmpty=True)
    mi = validators.MaxLength(1)
    class Numbers(foreach.ForEach):
        class Number(schema.Schema):
            type = validators.OneOf(['home', 'work'])
            phone_number = validators.PhoneNumber()

Some other techniques: Ingredients for Building a DSL in Python (pdf).

Edit (by Ali): An example of doing this using collections and instances is what I would prefer. The important fact is the instances, which give you more power, and eliminate reason to use metaclasses. Further worth noting that your example uses a mixture of classes and instances, which is surely an indication that you can't just do it all with metaclasses. And creates a truly non-uniform way of doing it.

number_validator = [
    v.OneOf('type', ['home', 'work']),
    v.PhoneNumber('phone_number'),
]

validators = [
    v.String('first_name', notEmpty=True),
    v.String('last_name', notEmpty=True),
    v.MaxLength('mi', 1),
    v.ForEach([number_validator,])
]

It's not perfect, but already there is almost zero magic, no need for metaclasses, and improved uniformity.

I was thinking the same thing just yesterday and completely agree. The complications in the code caused by attempts to make it more declarative generally make the codebase harder to maintain, harder to read and less pythonic in my opinion. It also normally requires a lot of copy.copy()ing (to maintain inheritance and to copy from class to instance) and means you have to look in many places to see whats going on (always looking from metaclass up) which goes against the python grain also. I have been picking through formencode and sqlalchemy code to see if such a declarative style was worth it and its clearly not. Such style should be left to descriptors (such as property and methods) and immutable data. Ruby has better support for such declarative styles and I am glad the core python language is not going down that route.

I can see their use for debugging, add a metaclass to all your base classes to get richer info. I also see their use only in (very) large projects to get rid of some boilerplate code (but at the loss of clarity). sqlalchemy for example does use them elsewhere, to add a particular custom method to all subclasses based on an attribute value in their class definition e.g a toy example

class test(baseclass_with_metaclass):
    method_maker_value = "hello"

could have a metaclass that generated a method in that class with special properties based on "hello" (say a method that added "hello" to the end of a string). It could be good for maintainability to make sure you did not have to write a method in every subclass you make instead all you have to define is method_maker_value.

The need for this is so rare though and only cuts down on a bit of typing that its not really worth considering unless you have a large enough codebase.

The only time I used metaclasses in Python was when writing a wrapper for the Flickr API.

My goal was to scrape flickr's api site and dynamically generate a complete class hierarchy to allow API access using Python objects:

# Both the photo type and the flickr.photos.search API method 
# are generated at "run-time"
for photo in flickr.photos.search(text=balloons):
    print photo.description

So in that example, because I generated the entire Python Flickr API from the website, I really don't know the class definitions at runtime. Being able to dynamically generate types was very useful.

Metaclasses aren't replacing programming! They're just a trick which can automate or make more elegant some tasks. A good example of this is Pygments syntax highlighting library. It has a class called RegexLexer which lets the user define a set of lexing rules as regular expressions on a class. A metaclass is used to turn the definitions into a useful parser.

They're like salt; it's easy to use too much.

You never absolutely need to use a metaclass, since you can always construct a class that does what you want using inheritance or aggregation of the class you want to modify.

That said, it can be very handy in Smalltalk and Ruby to be able to modify an existing class, but Python doesn't like to do that directly.

There's an excellent DeveloperWorks article on metaclassing in Python that might help. The Wikipedia article is also pretty good.

The way I used metaclasses was to provide some attributes to classes. Take for example:

class NameClass(type):
    def __init__(cls, *args, **kwargs):
       type.__init__(cls, *args, **kwargs)
       cls.name = cls.__name__

will put the name attribute on every class that will have the metaclass set to point to NameClass.

I recently had to use a metaclass to help declaratively define an SQLAlchemy model around a database table populated with U.S. Census data from http://census.ire.org/data/bulkdata.html

IRE provides database shells for the census data tables, which create integer columns following a naming convention from the Census Bureau of p012015, p012016, p012017, etc.

I wanted to a) be able to access these columns using a model_instance.p012017 syntax, b) be fairly explicit about what I was doing and c) not have to explicitly define dozens of fields on the model, so I subclassed SQLAlchemy's DeclarativeMeta to iterate through a range of the columns and automatically create model fields corresponding to the columns:

from sqlalchemy.ext.declarative.api import DeclarativeMeta

class CensusTableMeta(DeclarativeMeta):
    def __init__(cls, classname, bases, dict_):
        table = 'p012'
        for i in range(1, 49):
            fname = "%s%03d" % (table, i)
            dict_[fname] = Column(Integer)
            setattr(cls, fname, dict_[fname])

        super(CensusTableMeta, cls).__init__(classname, bases, dict_)

I could then use this metaclass for my model definition and access the automatically enumerated fields on the model:

CensusTableBase = declarative_base(metaclass=CensusTableMeta)

class P12Tract(CensusTableBase):
    __tablename__ = 'ire_p12'

    geoid = Column(String(12), primary_key=True)

    @property
    def male_under_5(self):
        return self.p012003

    ...

Pydantic is a library for data validation and settings management that enforces type hints at runtime and provides user friendly errors when data is invalid. It makes use of metaclasses for its BaseModel and for number range validation.

At work I encountered some code that had a process that had several stages defined by classes. The ordering of these steps was controlled by metaclasses that added the steps to a list as the classes were defined. This was thrown out and the order was set by adding them to a list.

the answer from @Dan Gittik is cool

the examples at the end could clarify many things,I changed it to python 3 and give some explanation:

class MetaMetaclass(type):
    def __new__(meta, name, bases, attrs):
        def __new__(meta, name, bases, attrs):
            cls = type.__new__(meta, name, bases, attrs)
            cls._label = 'Made in %s' % meta.__name__
            return cls

        attrs['__new__'] = __new__
        return type.__new__(meta, name, bases, attrs)

#China is metaclass and it's __new__ method would be changed by MetaMetaclass(metaclass)
class China(MetaMetaclass, metaclass=MetaMetaclass):
    __metaclass__ = MetaMetaclass

#Taiwan is metaclass and it's __new__ method would be changed by MetaMetaclass(metaclass)
class Taiwan(MetaMetaclass, metaclass=MetaMetaclass):
    __metaclass__ = MetaMetaclass

#A is a normal class and it's __new__ method would be changed by China(metaclass)
class A(metaclass=China):
    __metaclass__ = China

#B is a normal class and it's __new__ method would be changed by Taiwan(metaclass)
class B(metaclass=Taiwan):
    __metaclass__ = Taiwan


print(A._label)  # Made in China
print(B._label)  # Made in Taiwan

• everything is object,so class is object
• class object is created by metaclass
• all class inheritted from type is metaclass
• metaclass could control class creating
• metaclass could control metaclass creating too(so it could loop for ever)
• this's metaprograming...you could control the type system at running time
• again,everything is object,this's a uniform system,type create type,and type create instance

Another use case is when you want to be able to modify class-level attributes and be sure that it only affects the object at hand. In practice, this implies "merging" the phases of metaclasses and classes instantiations, thus leading you to deal only with class instances of their own (unique) kind.

I also had to do that when (for concerns of readibility and polymorphism) we wanted to dynamically define propertys which returned values (may) result from calculations based on (often changing) instance-level attributes, which can only be done at the class level, i.e. after the metaclass instantiation and before the class instantiation.

I know this is an old question But here is a use case that is really invaluable if wanting to create only a single instance of a class based on the parameters passed to the constructor.

Instance singletons I use this code for creating a singleton instance of a device on a Z-Wave network. No matter how many times I create an instance if the same values are passed to the constructor if an instance with the exact same values exists then that is what gets returned.

import inspect


class SingletonMeta(type):
    # only here to make IDE happy
    _instances = {}

    def __init__(cls, name, bases, dct):
        super(SingletonMeta, cls).__init__(name, bases, dct)
        cls._instances = {}

    def __call__(cls, *args, **kwargs):
        sig = inspect.signature(cls.__init__)
        keywords = {}

        for i, param in enumerate(list(sig.parameters.values())[1:]):
            if len(args) > i:
                keywords[param.name] = args[i]
            elif param.name not in kwargs and param.default != param.empty:
                keywords[param.name] = param.default
            elif param.name in kwargs:
                keywords[param.name] = kwargs[param.name]
        key = []
        for k in sorted(list(keywords.keys())):
            key.append(keywords[k])
        key = tuple(key)

        if key not in cls._instances:
            cls._instances[key] = (
                super(SingletonMeta, cls).__call__(*args, **kwargs)
            )

        return cls._instances[key]


class Test1(metaclass=SingletonMeta):

    def __init__(self, param1, param2='test'):
        pass


class Test2(metaclass=SingletonMeta):

    def __init__(self, param3='test1', param4='test2'):
        pass


test1 = Test1('test1')
test2 = Test1('test1', 'test2')
test3 = Test1('test1', 'test')

test4 = Test2()
test5 = Test2(param4='test1')
test6 = Test2('test2', 'test1')
test7 = Test2('test1')

print('test1 == test2:', test1 == test2)
print('test2 == test3:', test2 == test3)
print('test1 == test3:', test1 == test3)
print('test4 == test2:', test4 == test2)
print('test7 == test3:', test7 == test3)
print('test6 == test4:', test6 == test4)
print('test7 == test4:', test7 == test4)
print('test5 == test6:', test5 == test6)

print('number of Test1 instances:', len(Test1._instances))
print('number of Test2 instances:', len(Test2._instances))

output

test1 == test2: False
test2 == test3: False
test1 == test3: True
test4 == test2: False
test7 == test3: False
test6 == test4: False
test7 == test4: True
test5 == test6: False
number of Test1 instances: 2
number of Test2 instances: 3

Now someone might say it can be done without the use of a metaclass and I know it can be done if the __init__ method is decorated. I do not know of another way to do it. The code below while it will return a similiar instance that contains all of the same data it is not a singleton instance, a new instance gets created. Because it creates a new instance with the same data there wuld need to be additional steps taken to check equality of instances. I n the end it consumes more memory then using a metaclass and with the meta class no additional steps need to be taken to check equality.

class Singleton(object):
    _instances = {}

    def __init__(self, param1, param2='test'):
        key = (param1, param2)
        if key in self._instances:
            self.__dict__.update(self._instances[key].__dict__)
        else:
            self.param1 = param1
            self.param2 = param2
            self._instances[key] = self


test1 = Singleton('test1', 'test2')
test2 = Singleton('test')
test3 = Singleton('test', 'test')

print('test1 == test2:', test1 == test2)
print('test2 == test3:', test2 == test3)
print('test1 == test3:', test1 == test3)

print('test1 params', test1.param1, test1.param2)
print('test2 params', test2.param1, test2.param2)
print('test3 params', test3.param1, test3.param2)

print('number of Singleton instances:', len(Singleton._instances))

output

test1 == test2: False
test2 == test3: False
test1 == test3: False
test1 params test1 test2
test2 params test test
test3 params test test
number of Singleton instances: 2

The metaclass approach is really nice to use if needing to check for the removal or addition of a new instance as well.

    import inspect


class SingletonMeta(type):
    # only here to make IDE happy
    _instances = {}

    def __init__(cls, name, bases, dct):
        super(SingletonMeta, cls).__init__(name, bases, dct)
        cls._instances = {}

    def __call__(cls, *args, **kwargs):
        sig = inspect.signature(cls.__init__)
        keywords = {}

        for i, param in enumerate(list(sig.parameters.values())[1:]):
            if len(args) > i:
                keywords[param.name] = args[i]
            elif param.name not in kwargs and param.default != param.empty:
                keywords[param.name] = param.default
            elif param.name in kwargs:
                keywords[param.name] = kwargs[param.name]
        key = []
        for k in sorted(list(keywords.keys())):
            key.append(keywords[k])
        key = tuple(key)

        if key not in cls._instances:
            cls._instances[key] = (
                super(SingletonMeta, cls).__call__(*args, **kwargs)
            )

        return cls._instances[key]


class Test(metaclass=SingletonMeta):

    def __init__(self, param1, param2='test'):
        pass


instances = []

instances.append(Test('test1', 'test2'))
instances.append(Test('test1', 'test'))

print('number of instances:', len(instances))

instance = Test('test2', 'test3')
if instance not in instances:
    instances.append(instance)

instance = Test('test1', 'test2')
if instance not in instances:
    instances.append(instance)

print('number of instances:', len(instances))

output

number of instances: 2
number of instances: 3

Here is a way to remove an instance that has been created after the instance is no longer in use.

    import inspect
import weakref


class SingletonMeta(type):
    # only here to make IDE happy
    _instances = {}

    def __init__(cls, name, bases, dct):
        super(SingletonMeta, cls).__init__(name, bases, dct)

        def remove_instance(c, ref):
            for k, v in list(c._instances.items())[:]:
                if v == ref:
                    del cls._instances[k]
                    break
                    
        cls.remove_instance = classmethod(remove_instance)
        cls._instances = {}

    def __call__(cls, *args, **kwargs):
        sig = inspect.signature(cls.__init__)
        keywords = {}

        for i, param in enumerate(list(sig.parameters.values())[1:]):
            if len(args) > i:
                keywords[param.name] = args[i]
            elif param.name not in kwargs and param.default != param.empty:
                keywords[param.name] = param.default
            elif param.name in kwargs:
                keywords[param.name] = kwargs[param.name]
        key = []
        for k in sorted(list(keywords.keys())):
            key.append(keywords[k])
        key = tuple(key)

        if key not in cls._instances:
            instance = super(SingletonMeta, cls).__call__(*args, **kwargs)

            cls._instances[key] = weakref.ref(
                instance,
                instance.remove_instance
            )

        return cls._instances[key]()


class Test1(metaclass=SingletonMeta):

    def __init__(self, param1, param2='test'):
        pass


class Test2(metaclass=SingletonMeta):

    def __init__(self, param3='test1', param4='test2'):
        pass


test1 = Test1('test1')
test2 = Test1('test1', 'test2')
test3 = Test1('test1', 'test')

test4 = Test2()
test5 = Test2(param4='test1')
test6 = Test2('test2', 'test1')
test7 = Test2('test1')

print('test1 == test2:', test1 == test2)
print('test2 == test3:', test2 == test3)
print('test1 == test3:', test1 == test3)
print('test4 == test2:', test4 == test2)
print('test7 == test3:', test7 == test3)
print('test6 == test4:', test6 == test4)
print('test7 == test4:', test7 == test4)
print('test5 == test6:', test5 == test6)

print('number of Test1 instances:', len(Test1._instances))
print('number of Test2 instances:', len(Test2._instances))


print()
del test1
del test5
del test6

print('number of Test1 instances:', len(Test1._instances))
print('number of Test2 instances:', len(Test2._instances))

output

test1 == test2: False
test2 == test3: False
test1 == test3: True
test4 == test2: False
test7 == test3: False
test6 == test4: False
test7 == test4: True
test5 == test6: False
number of Test1 instances: 2
number of Test2 instances: 3

number of Test1 instances: 2
number of Test2 instances: 1

if you look at the output you will notice that the number of Test1 instances has not changed. That is because test1 and test3 are the same instance and I only deleted test1 so there is still a reference to the test1 instance in the code and as a result of that the test1 instance does not get removed.

Another nice feature of this is if the instance uses only the supplied parameters to do whatever it is tasked to do then you can use the metaclass to facilitate remote creations of the instance either on a different computer entirely or in a different process on the same machine. the parameters can simply be passed over a socket or a named pipe and a replica of the class can be created on the receiving end.