1

I recently stumbled across metaclasses in Python and decided to use them in order to simplify some features. (Using Python 3.5)

In a nutshell, I'm writing a module defining classes such as "components" that must be registered and initialized (I mean I need to initialize the actual class, not an instance).

I can register the class easily:

class MetaComponent(type):
    def __init__(cls, *args, **kargs):
        super().__init__(*args, **kargs)
        RegisterComponent(cls)

class BaseComponent(metaclass=MetaComponent):
    pass

class Component(BaseComponent):
    """This is the actual class to use when writing components"""

In this case, I'm registering the class of the component, as it allows me to refer to them later, without their actual reference.

But classes lack the ability to initialize themselves (at least in Python 3.5), and can cause some problems, like such:

class Manager(Component):
    SubManagers = []

    @classmethod
    def ListSubManagers(cls):
        for manager in cls.SubManagers:
            print(manager)

    @classmethod
    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

@Manager1.RegisterSubManager
class Manager2(Manager):
    pass

@Manager2.RegisterSubManager
class Manager3(Manager):
    pass

# Now the fun:
Manager1.ListSubManagers()

# Displays:
# > Manager2
# > Manager3

Now, this is a problem, because the idea was to have a unique list of sub-managers per manager. But the SubManager field is shared across each subclasses... So adding to one's list adds to every's. RIP.

So the next idea was to implement a kind of initializator:

class BaseManager(Component):
    @classmethod
    def classinit(cls):
        cls.SubManagers = []

But now, I need a way to call this method after the class creation. So lets do this with metaclasses again:

class MetaComponent(type):
    def __init__(cls, *args, **kargs):
        super().__init__(*args, **kargs):
        cls.classinit(**kargs)
        RegisterComponent(cls)

class BaseComponent(metaclass=MetaComponent):
    @classmethod
    def classinit(cls, **kargs):
        print('BASE', cls)

class Component(BaseComponent):
    @classmethod
    def classinit(cls, **kargs):
        super().classinit(**kargs) # Being able to use super() is the goal
        print('COMPONENT', cls)

I would consider myself done with this. Somewhat of an elegant way to do it IMO. The classinit() would be called from each class being created (unlike the 3.6 __init_subclass__ that is called on the parent). At least I liked it, until Python 3.5 cried in RuntimeError: super(): empty __class__ cell...

I read it was because I was calling a method from the metaclass's __init__ method and that although the class was created (hence my will to put code in __init__, to initialize something already created), it lacks this __class__ cell, at least at that moment...

I tried running the exact same code in Python 3.6, and it worked, so I guess something was wrong but got fixed...

My real questions are:

  • Can we truly initialize classes with metaclasses in Python 3.5 ?
  • Is there a way to avoid the use-restriction of super() in the initialization procedure?
  • Why is it working in 3.6?
  • If everything should fail, what would be the best course of action to still provide the class initialization, and allowing for super(...) calls? (Like do I need to refer to the super class explicitly ?)

Thanks for your help in advance.

EDIT:

The goal is to be able to derive components and be able to initialize each one's class relative to its parent, in an "easy" way:

class Manager(Component):
    def classinit(cls, **kargs):
        cls.SubManagers = []

    @classmethod
    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

@Manager.RegisterSubManager
class EventManager(Manager):
    def classinit(cls, **kargs):
        super().classinit(**kargs) # keep the old behaviour
        cls.Events = []

    # ...

@EventManager.RegisterSubManager
class InputManager(EventManager):
    def classinit(cls, **kargs):
        super().classinit(**kargs) # again, keep old behaviour
        cls.Inputs = []

    # use parts of EventManager, but define specialized methods
    # for input management

Managers are one concern, I have multiple concepts that depend on components and their ability to initialize their class.

2
  • I don't understand how @Manager2.RegisterSubManager can possibly work, since Manager2 isn't a subclass of Manager1. Commented Aug 25, 2017 at 4:42
  • My mistake, fixed Commented Aug 25, 2017 at 20:40

3 Answers 3

1

TL;DR - you will indeed have the RuntimeError: super(): empty __class__ cell... if you try to use an empty call to super from the metaclass __new__ or __init__ methods: at this stage the implicit "magic" variable __class__ that is internally used by super has not been created yet. (On verifying this, I just found out this has been fixed in Python 3.6 - that is: classmethods using parameterless super can be called from the metaclass's __init__ in Python 3.6, but yield this error in 3.5)

If that is the only thing in your way by now, just hardcode the call to the superclass method, like it was needed prior to the creation of super in Python. (Using the verbose form of super won't work as well).

--

Your second to last idea, of using classmethods as class decorators for registering, can be made to work by automatic creating a SubManagers attribute with the metaclass by using the a simple Python name-mangling to automatically create each manager class unique SubManagers attribute by inspecting one class' own namespace in its __dict__ (and it can be done without a metaclass as well)

Using metaclasses, just add these 2 lines at the end of your metaclass __init__:

if getattr(cls, "SubManagers") and not "SubManagers" in cls.__dict__:
    cls.SubManagers = []

If your class-decorator approach otherwise preclude metaclasses, you don't need to use a metaclass just for that - change your register method to perform the above "own" submanager list creation:

@classmethod
def RegisterSubManager(cls, manager):
   if not "SubManagers" in cls.__dict__:
       cls.SubManagers = []
   cls.SubManagers.append(manager)
Sign up to request clarification or add additional context in comments.

3 Comments

Thank you for your response. I was looking more for a workaround, if any, to allow for an automatic super()-like resolution in subclasses, but I guess I'll use the explicit parent class call in the subclasses. On the other hand, the registration decorator is only a thing for managers, not all components: different components work as singleton-like classes and might need different initializations (writing it down almost sounds like not such a bright idea using classes as I do, but in a perfect world such as 3.6, it is working... just not in 3.5 :( ).
Maybe you can do a simple class decorator just to call the initclass method - the decorator is applied after the metaclass has run, and super() should work then.
Was doing that, tryied to avoid it using the metaclasses :p. At least now the component (not manager) registration works without decorators. But I'm a bit depressed about this limitation about running methods in the metaclass.__init__ thing... And also about the fact that in the next version of Python (3.6) it was fixed... I really thought some pure Python workarounds were possible, even tho the closest approach I found was a bit... Hardcore: stackoverflow.com/a/4885951/7983255. Would there be a way from this link ? (I'm a bit overthrown I must say)
1

If you want extra behavior for your Manager types, perhaps you want them to have their own, more refined metaclass, rather than just using the one they've inherited from Component. Try writing a MetaManager metaclass that inherits from MetaComponent. You can even move the class methods from Manager1 into the metaclass (where they become normal methods):

class MetaManager(MetaComponent):
    def __init__(cls, *args, **kwargs):
        super().__init__(*args, **kwargs)
        cls.SubManagers = [] # each class gets its own SubManagers list

    def ListSubManagers(cls):
        for manager in cls.SubManagers:
            print(manager)

    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

class Manager(Component, metaclass=MetaManager): # inherit from this to get the metaclass
    pass

class Manager1(Manager):
    pass

@Manager1.RegisterSubManager
class Manager2(Manager):
    pass

@Manager2.RegisterSubManager
class Manager3(Manager):
    pass

5 Comments

I like the idea, but I wanted the end developper to be able to write its own managers and their own initialization (while refering to super().__init__ so that the base cls.SubManagers is initialized too). Your idea does the job, but it does not provide the modularity I wanted, more specialy it does complicate things a bit :). Thank you for your time looking with me for an answer ! I'm still digging too, we never know what we may find ! (I'm actually losing a bit of time with this, but once it is done, I will have covered a fairly interesting point about Python)
Although I never thought about placing class methods in the metaclass, I find this pretty cool, it feels "natural" indeed ! (But I would reserve this definition style for classes that would absolutly require metaclasses, I don't feel like Managers would in my case...)
I'm not sure I understand your concern. Why do you think you can't use super somewhere? You don't need to fudge a classinit method, the metaclass's __init__ already handles initializing the list of sub-managers.
In a nutshell, I have manager classes that need to be initialized, but each sub class (class SubManagerX(Manager)) might want to initialize itself like a vanilla manager, plus some extra initialization for itself. So in order to be able to initialize submanagers while conserving the old behaviour, I needed to be able to call super().classinit() from submanagers classinit(), because managers can derive from each others. But if I only use metaclasses for the initialization, everytime someone will want to subclass it it would be a bit of a pain...
Well, at a certain point perhaps you should reconsider using classes and metaclasses instead of using classes and instances. It's easier for users to customize the behavior of instances than it is to customize class creation (unless they're comfortable writing their own metaclasses).
0

Ok, so after some experiments, I managed to provide a "fix" in order to allow for class initialization, allowing the use of super().

First off, the module to "fix" the initialization method:

# ./PythonFix.py
import inspect
import types

def IsCellEmpty(cell):
    """Lets keep going, deeper !"""
    try:
        cell.cell_contents
        return False
    except ValueError:
        return True

def ClosureFix(cls, functionContainer):
    """This is where madness happens.
    I didn't want to come here. But hey, lets get mad.
    Had to do this to correct a closure problem occuring in
     Python < 3.6, joy.
    Huge thanks: https://stackoverflow.com/a/4885951/7983255
    """

    # Is the class decorated with @classmethod somehow
    isclassmethod = inspect.ismethod(functionContainer) and functionContainer.__self__ is cls
    if isclassmethod:
        function = functionContainer.__func__
    else:
        function = functionContainer

    # Get cells and prepare a cell holding ref to __class__
    ClosureCells = function.__closure__ or ()
    ClassCell_Fix = (lambda: cls).__closure__[0]

    # Shortcut
    c = function.__code__
    HasClassFreevar = '__class__' in c.co_freevars
    HasEmptyCells = any(IsCellEmpty(cell) for cell in ClosureCells)
    if HasClassFreevar and not HasEmptyCells: # No fix required.
        return classmethod(function)

    Freevars_Fixed = c.co_freevars
    Closure_Fixed = ClosureCells

    if not HasClassFreevar:
        Freevars_Fixed += ('__class__',)
        Closure_Fixed += (ClassCell_Fix,)

    elif HasEmptyCells: # This is silly, but for what I'm doing its ok.
        Closure_Fixed = tuple(ClassCell_Fix if IsCellEmpty(cell) else cell for cell in ClosureCells)

    # Now the real fun begins
    PyCode_fixedFreevars = types.CodeType(
        c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
        c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
        c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
        c.co_lnotab, Freevars_Fixed, c.co_cellvars
    )

    # Lets fetch the last closure to add our __class__ fix
    FixedFunction = types.FunctionType(
        PyCode_fixedFreevars, function.__globals__, function.__name__,
        function.__defaults__, Closure_Fixed
    )

    # Lets rewrap it so it is an actual classmethod (which it should be):
    return classmethod(FixedFunction)

And now, the component code:

class MetaComponent(type):
    def __init__(cls:type, *args, **kargs) -> None:
        super().__init__(*args, **kargs)
        if hasattr(cls, 'classinit'):
            cls.classinit = PythonFix.ClosureFix(cls, cls.classinit)
            cls.classinit(**kargs)
        RegisterComponent(cls)

    def classinit(cls:type, **kargs) -> None:
        """The default classinit method."""
        pass

class Component(metaclass=MetaComponent):
    """This class self registers, inherit from this"""

To be fair, I'm happy with it being done. Hope this helps someone wanting to initialize classes too (in a pre-Python3.6 env at least...).

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.