4

i need to do something like this:

class Base1:
   def __init__(self, uniform_params):
       pass

class Base2:
   def __init__(self, uniform_params):
       pass

class DynamicDerive(self, dynamicspec, uniform_params):
     kls = dynamicspec.kls
     kls.__init__self, uniform_params)

spec = SomeSpecificationObject()
spec.kls = Base1

x = DynamicDerive(spec, uniform_params)

spec2 = SomeSpecificationObject()
spec2.kls = Base2

y = DynamicDerive(spec2, some_more_uniform_params)

the parameters to Base1 and Base2 are uniform and consistent. the requirement is to pass in the class that DynamicDerive is to derive from at instance creation time.

the alternative is "simple": create DynamicDerive1(Base1), DynamicDerive2(Base2), except unfortunately:

  1. the class DynamicDerive is used in hundreds of places.
  2. there is no way to predict what future users will pass in, here. users may create a Base3, Base4 etc.

so a cut/paste option of creating entire swathes of hundreds of identical classes, which merely change the name of the base class, is just not an option.

hypothetically this could be solved through a "redirection" API, where a special class does this:

class Base:
    def __init__(self, redirectorkls, uniform_args):
        self.redir = redirectorkls(uniformargs)
    def fn1(self, *args, **kwargs):
        return self.redir.fn1(*args, **kwargs)
    def fn2(self, *args, **kwargs):
        return self.redir.fn2(*args, **kwargs)
    ...
    ...

however although it will work, that entirely defeats the object of the exercise. there has to be a way to do this that involves meta-programming (meta-classes).

looking up metaclass programming tutorials, unfortunately, all show how to create classes from outside of the constructor, whereas what is needed above is for the metaclass to be created inside the constructor.

anyone have any clues?

[update] - i need to then be able to further derive from DynamicDerive. GreenCloakGuy kindly answered by providing a function that would do the task, however it is not possible to derive classes from functions.

class DerivedFromDynamicDerive(DynamicDerive):
    def __init__(self, dynamicspec, nonuniformparams, uniform_params):
        self.nonuniformparams = nonuniformparams
        DynamicDerive.__init__(self, dynamicspec, uniform_params)

(note: as this is actual libre code, the place where this is required is here: https://git.libre-riscv.org/?p=ieee754fpu.git;a=blob;f=src/ieee754/fpadd/addstages.py;h=2bc23df0dabf89f8a4e194d5e573a88d5d740d0e;hb=78cbe8c5131a84426a3cad4b0b3ed4ab7da49844#l19

SimpleHandShake needs to be dynamically replaced in around 40 places, where users of this IEEE754 compliant RTL may specify the class that they wish to use. this is just of over 50 classes that need this capability).

7
  • Why do you need a specification object spec with a class-valued attribute, instead of simply passing the class itself to DynamicDerive? Commented Jul 30, 2019 at 14:08
  • (note: the original question - and this answer - are unaffected - not related to - the question that you ask, chepner. if the class BaseN was passed in instead of spec it would make no difference to the original question) answer: because there are other parameters inside that spec object, and the number of parameters kept on increasing. with a complex set of classes it was quickly becoming absolute hell to keep on adding yet more and more parameters, where some class instances used only a few of those parameters and others use them all. solution: a SINGLE pspec object. Commented Jul 30, 2019 at 14:23
  • Whether you find it "elegant" or not, your "redirection API" - which is known as "composition/delegation" BTW - seems to be the simple, obvious, readable, maintainable and perfectly sane solution. Commented Jul 30, 2019 at 14:41
  • i just tried it: there's a unique problem associated with the redirection, in that the class derivation chain critically relies on objects being set up by (in) a higher up constructor (the inheriting class) in order for other functions to work. because the "redirector" object is a different object that is no longer part of the inheritance chain, that no longer works and it would require a massive redesign of complex code that took 6 months to write. Commented Jul 30, 2019 at 14:46
  • I've provided an answer bellow that answers to the needs you've posted here. If this is to be used in a "massive redesign" of a 6-month-to-write codebase however, you should really take your time to understand what is going on. If you need further help, just get in touch (e-mail is at my profile) Commented Jul 30, 2019 at 19:44

2 Answers 2

1

This thing would be cleaner if you could just use multiple inheritance, and your configuring parameters as Mixin classes -to the point no special metaclass or action at class creation time would be needed.

And, of course, if one won't need neither an issubclass check nor to have subclasses of DynamicDerive, a factory function, that would take in the bases, keep a registry as a cache, and just instantiate the new object would also not require any special code.

But, if you need the parametric bases to be higher up in the MRO than "DerivedClass", as you are asking for, then, the way to customize the instance class at class-instantiation time, is to override the __call__ method of the metaclass. (This is what Python run, type.__call__, that will ultimately call the class' __new__ and __init__ methods).

This thing worked here for what I tried - see if it suits you:


import threading

class M(type):
    registry = {} 
    recursing = threading.local()
    recursing.check = False
    mlock = threading.Lock()

    def __call__(cls, *args, **kw):
        mcls = cls.__class__
        if mcls.recursing.check:
            return super().__call__(*args, **kw)
        spec = args[0]
        base = spec.kls


        if (cls, base) not in mcls.registry:
            mcls.registry[cls, base] = type(
                cls.__name__,
                (cls, base) + cls.__bases__[1:],
                {}
            )
        real_cls = mcls.registry[cls, base]

        with mcls.mlock:
            mcls.recursing.check = True
            instance = real_cls.__class__.__call__(real_cls, *args, **kw)
            mcls.recursing.check = False
        return instance 

I imported this, and run this snippet in a Python session:


In [54]: class DynamicDerive(metaclass=M): 
...:     def __init__(self, spec): 
...:         super().__init__(spec) 
...:          
...:  
...: class Base1: 
...:    def __init__(self, uniform_params): 
...:        print("at base 1") 
...:  
...: class Base2: 
...:    def __init__(self, uniform_params): 
...:        print("at base 2") 
...:  
...:  
...: SomeSpec = type("SomeSpec", (), {}) 
...:  
...: spec1 = SomeSpec() 
...: spec1.kls = Base1 
...:  
...: spec2 = SomeSpec() 
...: spec2.kls = Base2 
...:  
...: a1 = DynamicDerive(spec1) 
...: a2 = DynamicDerive(spec2) 


at base 1
at base 2

In [55]: isinstance(a1, DynamicDerive)                                                                                             
Out[55]: True

In [56]: isinstance(a2, DynamicDerive)                                                                                             
Out[56]: True

In [57]: class D2(DynamicDerive): pass                                                                                             

In [58]: b1 = D2(spec1)                                                                                                            
at base 1

In [59]: b1.__class__.__mro__                                                                                                      
Out[59]: (__main__.D2, __main__.D2, __main__.DynamicDerive, __main__.Base1, object)

Sign up to request clarification or add additional context in comments.

10 Comments

jsbueno: yes that works perfectly, including inheriting from DynamicDerive. the recursion trick was the utterly obscure piece that i'd missed, after noticing that overriding __call__ resulted in two calls to __call__, i had no idea what to do with that, so thank you. more on the mixin idea in a separate comment
regarding the mixin idea: i've been using python since 2000, i've used SocketServer extensively: the approach is just not practical in this particular case. there's an entire suite of classes (using nmigen) that cover IEEE754 FP hardware for ADD, MUL, DIV, SQRT, R-SQRT, INT2FP, FP2INT, all of which have something like 5 to 6 uses of a "mixin" class that needs replacing with a class that the user of this API chooses. they (and we) would need to duplicate and maintain a whopping 50 classes (minimum) using the mixin "pattern": it's just not practical in this case. so, thank you!
interesting, i'm running into this: stackoverflow.com/questions/11276037/…
Another clean design here would be to have objects of DynamicDerive to wrap the Base classes and proxy to the methods they should cover there. In a critical codebase, I'd prefer this approach - though it requires some careful wiring of the proxing mechanism.
As for the metaclass conflict, it is easily fixable - in this case, just make the metaclass here (M) inherit from the other metaclass in use. Among my answers here there is an example that creates a suitable metaclass by having + (__add__) on the meta-meta-class as well: stackoverflow.com/questions/476586/… (that is overkill, of course, the intent was to show a "metametaclass example")
|
0

One thing you might be able to do is to make either a method or a static class that creates the necessary class dynamically and returns an instance of it:

def DynamicDerive(dynamicspec, uniform_params):
    superclass = dynamicspec.kls
    class DynamicDerive(superclass):
        def __init__(self, uniform_params):
            print(superclass)
            superclass.__init__(self, uniform_params)
        pass
    return DynamicDerive(uniform_params)

This works because the interior code of a function isn't evaluated until it's called, so you can determine the base class of DynamicDerive dynamically, and then do whatever modifications you need to its definition. The downside of this is it's significantly slower, as you have to actually go through the trouble of redefining the class every time. But here's an example of this in action:

>>> x = DynamicDerive(spec1, None)
<class '__main__.Base1'>
>>> y = DynamicDerive(spec2, None)
<class '__main__.Base2'>
>>> z = DynamicDerive(spec1, None)
<class '__main__.Base1'>
>>> x.__class__ == y.__class__
False
>>> x.__class__ == z.__class__
False
>>> str(x.__class__) == str(y.__class__)
True
>>> x.__class__
<class '__main__.DynamicDerive.<locals>.DynamicDerive'>

If you wanted to be really ambitious you could maybe change the __class__ variable in the new class's __init__ (after all, you have the superclass right there if you want it).


Before implementing this properly, you'll want to think very carefully about exactly what use cases it's supposed to cover, and whether you can do that in a less hacky - and more explicit - way. Having a bunch of classes that look identical but are not is very confusing to maintain, and could lead to very easy-to-make coding errors.

It would probably be better practice to define separate classes in every individual case where they're necessary, and tailor them to the appropriate use case. Python is duck typed (if it looks like a duck and quacks like a duck, then we may as well assume it's a duck), so anything besides the code that's instantiating a class shouldn't actually care about its type, only about what it can do.

4 Comments

it's almost there: the problem is, i need to derive from this, rather than "use" it. DynamicDerive therefore itself has to be a class. see usage example here and apologies that i am restricted to a 512-character comment with no proper edit capabilities git.libre-riscv.org/?p=ieee754fpu.git;a=blob;f=src/ieee754/…
"Having a bunch of classes that look identical but are not is very confusing to maintain, and could lead to very easy-to-make coding errors." - a way round that is to rename the class (using type()'s first argument). the problem is that to explicitly do inheritance "manually", it results in MASSIVE duplication of 50+ classes throughout a huge inheritance tree, all identical in every way except for Base1/Base2/Base3/Base4...
You could maybe follow the same paradigm I'm already using, and make another method to return the dynamically-generated class instead of an instance of it (which you could then use for inheritance/further derivation: class NewClass(DynamicDeriveClass(dynamicspec)). Otherwise, you'd need to use a different paradigm to solve the problem.
hmmm hmm... maybe overload __call__ there? this is how metaclasses are supposed to work, except the problem here is: dynamicspec is not a global variable... oh, i think i see: combine the two techniques. let me think how that would work

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.