Understanding Metaclass Interaction with __init_subclass__ and Class Decorators

Question

I'm trying to wrap my head around the precise order of operations and interactions when a Python class definition involves a metaclass, a init_subclass method, and a class decorator.

class Meta(type):
    def __new__(mcs, name, bases, dct):
        print(f"Meta.__new__ called for {name}")
        # Perform some modification to the class dictionary
        dct['meta_attr'] = f"Value from {mcs.__name__}"
        return super().__new__(mcs, name, bases, dct)

    def __init__(cls, name, bases, dct):
        print(f"Meta.__init__ called for {name}")
        super().__init__(cls, name, bases, dct)
        cls.meta_init_attr = f"Initialized by {cls.__class__.__name__}"


def class_decorator(cls):
    print(f"Class decorator called for {cls.__name__}")
    cls.decorated_attr = "Decorated!"
    return cls

@class_decorator
class MyBase(metaclass=Meta):
    def __init_subclass__(cls, **kwargs):
        print(f"MyBase.__init_subclass__ called for {cls.__name__}")
        super().__init_subclass__(**kwargs)
        cls.base_subclass_attr = "From MyBase __init_subclass__"

    def __init__(self):
        print("MyBase instance __init__ called")
        self.instance_attr = "Instance created"

print("\nDefining MyDerived:")
class MyDerived(MyBase):
    def __init_subclass__(cls, **kwargs):
        print(f"MyDerived.__init_subclass__ called for {cls.__name__}")
        super().__init_subclass__(**kwargs)
        cls.derived_subclass_attr = "From MyDerived __init_subclass__"

    def __init__(self):
        print("MyDerived instance __init__ called")
        super().__init__()

print("\nCreating MyDerived instance:")
instance = MyDerived()

print("\nAttributes on MyDerived class:")
print(f"MyDerived.meta_attr: {hasattr(MyDerived, 'meta_attr')}")
print(f"MyDerived.meta_init_attr: {hasattr(MyDerived, 'meta_init_attr')}")
print(f"MyDerived.decorated_attr: {hasattr(MyDerived, 'decorated_attr')}")
print(f"MyDerived.base_subclass_attr: {hasattr(MyDerived, 'base_subclass_attr')}")
print(f"MyDerived.derived_subclass_attr: {hasattr(MyDerived, 'derived_subclass_attr')}")

print("\nAttributes on instance:")
print(f"instance.instance_attr: {hasattr(instance, 'instance_attr')}")

Which of the listed attributes (meta_attr, meta_init_attr, decorated_attr, base_subclass_attr, derived_subclass_attr, instance_attr) will be present on the MyDerived class itself, and which on the instance of MyDerived?

Answer 1

Note that while it is nice to understand how it works, coding Python in the day-to-day does not require one to know this by heart. I believe the only important part for almost everyone is to know that the decorator is applied last, when creating a class, and __init__ of the class runs when creating instances.

Both class decorators (IIRC Python 2.4) and __init_subclass__ where introduced with the intent to customize class creation without needing a custom metaclass - and there are few things nowadays that would actually require one.

Now, for the full journey:

When there is a class statement, Python will first: determine the metaclass, based on the bases of the class - then, it will call the metaclass's __prepare__ method (if any).

After that, it will execute the class body: any statement in the class body itself (including the declaration of methods) is executed: the result of executing the class body is passed to the metaclass' __new__ method, along with the class name, bases, and the resulting namespace after processing the class body (usually a dictionary containing the class-level attributes and methods.

(Any annotations on the class body are set on that namespace __annotations__ key before calling the metaclass' __new__, along with other automatic entries, like __module__ and __qualname__)

The metaclass's __new__ resolve: at a certain point it will have to call type.__new__ for the actual class creation: at that point, any __init_subclass__ present in a superclass will run. If type.__new__ is not called, a new class is not created: that is the only way to actually create classes in pure Python code: a custom metaclass can run code before and/or after calling it - and it is in that code that __init_subclass__ on superclasses is executed.

After the metaclass' __new__, the metaclass' __init__ (if any) is called. After which, the class is created and ready to use - but if a class decorator is present, it is then called - it receives the new class as an argument and whatever it returns is placed in the current scope as the result of the class statement body.

Breaking that into the order given in your example, when instantiating MyBase , this will run:

1. dct['meta_attr'] = f"Value from {mcs.__name__}" and on the return statement of Meta.__new__ you call super().__new__(...) - which will execute type.__new__: at that point any __init_subclass__ would run -but there are none for MyBase.

2. Meta.__init__ is called with a fresh MyBase class, and the print(f"Meta.__init__ called for {name}") line runs.

3. class_decorator is called with the new MyBase as parameter and print(f"Class decorator called for {cls.__name__}") is executed.

4. Both meta_attr, meta_init_attr and decorated_attr are set on the fully initialized MyBase

5. MyDerived class statement is met, as one of the bases (MyBase) has a custom metaclass, that metaclass is picked as the metaclass to use ( Meta).

6. MyDerived class body executes, and Meta.__new__ is called, printing dct['meta_attr'] = f"Value from {mcs.__name__}"

7. Before Meta.__new__ for MyDerived returns, MyBase.__init_subclass__ is called as an effect of the super call in super().__new__(mcs, name, bases, dct). This prints print(f"MyBase.__init_subclass__ called for {cls.__name__}") and sets base_subclass_attr.

8. Meta.__init__ is executed for MyDerived as in step 2 above

9. class_decorator is executed for MyDerived as in step 3 above.

10. Upon running instance = MyDerived(), MyDerived.__init__ is called, printing print("MyDerived instance __init__ called") and in the sequence

11. MyBase.__init__ is called by the super() call in MyDerived.__init__ executing print("MyBase instance __init__ called") and setting self.instance_attr = "Instance created"

And finally, answering:

Which of the listed attributes (meta_attr, meta_init_attr, decorated_attr, base_subclass_attr, derived_subclass_attr, instance_attr) will be present on the MyDerived class itself, and which on the instance of MyDerived?

meta_attr, meta_init_attr, decorated_attr and base_subclass_attr will be present in the both the MyDerived class, and through it, on the instance, since class attributes are retrievied as fallbacks when one attribute is not present in an instance. The instance_attr will be set solely on the instance of MyDerived.