This is the last part of a three parts-series on adding functionality to existing code. In The voodoo of reflection I shortly described how this could be achieved by using reflection in java. In Interludium pro magus nigris we’ve changed the access to object-attributes (such as methods) to achieve similar results (not as mighty, though).

I’ve mentioned something I called minimal invasive changes which feels a bit like surgery and in a way it is. What I meant was to change the behavior of existing code without changing much (if any) of it. In part I and II of this series there was quite some change to the existing code, though. One may wonder whether not changing existing code is at all possible. It is. In comes the magic of metaclasses.

What are metaclasses?

Briefly, metaclasses are to classes what classes are to objects. Eh? The instances of metaclasses are classes. Several classes can be instances of the same metaclass. You can define behavior of these classes through that metaclass.

That sounds complicated. It is, sort of. On the other hand, in python those metaclasses behave just like classes. It’s just that their instances are classes, not objects. For example, the constructor (__init__) of a metaclass gets a class-object, not a class-instance (an object). The __new__-method of a metaclass gets called each time the definition of a class is finished. It’s arguments are the class-object (as instance of the metaclass), the name of the class, it’s bases and it’s dictionary (Matching names of class-attributes and methods to the respective objects, also called the class’s namespace.):

class Meta(type):
    def __new__(cls, name, bases, dct):
        #should call the super-implementation. You cannot create classes all by yourself.
        return super(Meta, cls).__new__(cls, name, bases, dct)

class SomeClass(object):
    def __init__(self):
        pass
#here __new__ of the metaclass of SomeClass gets called
#then, __init__ of the metaclass is called with the
#  SomeClass-classobject created in __new__

obj = SomeClass()
#now __init__ of SomeClass is called with an instance of SomeClass as argument.

The difference between __new__ and __init__ is (in metaclasses and classes) that __new__ is called to create a new instance of the class (or metaclass) while __init__ initializes the instance created by __new__. If __new__ does not return an instance (which is possible), __init__ will not be called.

The example is not quite correct. If a metaclass other than the default-metaclass (which is type) should be used it must be declared. It is possible to declare a global metaclass from which all classes defined afterwards are instances. Another way is to specify the metaclass in the class itself:

__metaclass__ = Meta
#declaring a global metaclass

class SomeClass(object):
    __metaclass__ = Meta
    #declaring an alternative metaclass just for this class

Looks like implementing some metaclass somehow and declaring it globally may be quite helpful in doing some minimal invasive behavioral change to all the classes. But how would that somehow look like? The idea is the following:

• Define some wrapper-function that adds the debugging-functionality to its argument (which is another function or method).
• Define a metaclass which puts the wrapper-function around all methods defined in instances of it (i.e. classes). It’s also possible to be selective on what to wrap. One could, e.g. just wrap methods following some naming-convention.
• Inject the metaclass into the code. If that happens globally no already existing code must be touched. One could be selective here, too, using the metaclass only on certain classes. Those would have to be touched, therefore.

Putting the plan into action, here comes the wrapper-function:

def wrapper(func):
  def wrap_intern(*args):
    print '  enter func "%s" with args ' % (func.__name__), args
    result = func(*args)
    print '  exit func "%s"n' % func.__name__
    return result
  # making the wrapper appear as the wrapped function
  wrap_intern.__name__ = func.__name__ #renaming the wrapper so that it looks like the wrapped function
  wrap_intern.__doc__  = func.__doc__  # also taking the documentation-string with it.

  return wrap_intern

wrapper defines another function wrap_intern internally. That function can access wrapper’s func-argument (which is the method to be wrapped). The arguments to wrap_intern will be the arguments to the actual function func. wrap_intern simply adds some output before and after calling func with the given arguments. The result of the call is stored and then returned. The last lines of code make wrap_intern look like the original function func, copying its name and documentation. That is not necessary but may be helpful.

And here is the metaclass that uses wrapper:

class Meta(type):
  def __new__(cls, name, bases, dct):
    print 'creating class with __new__: "%s"' % name
    for attr, value in dct.items():
      if callable(value):
        dct[attr] = wrapper(value)
    #calling the super-definition of __new__ to finish instantiation.
    return super(Meta, cls).__new__(cls, name, bases, dct)

We are overwriting __new__, inherited from type. We could also overwrite __init__. Almost the same method-body could be used, withouth returning something, though. I will give an example later. dct is a dictionary (hashtable) of all the class-attributes (variables and methods). For every entry is checked whether it can be called (Which roughly is equal to being a method. Objects can be callable, too, by declaring __call__, though. In production-code some attention is necessary.). If the value of the entry can be called, it is wrapped with the wrapper-function. To finish the creation of a new class (remember, we are talking meta), the super-implementation of __new__ is called.

Here now another metaclass implementation, this time using __init__:

class Meta2(type):
  def __init__(cls, name, bases, dct):
    super(Meta, cls).__init__(cls, name, bases, dct) #for completeness
    print 'creating class with __init__: "%s"' % name
    for attr, value in dct.items():
      if callable(value) and not attr == '__metaclass__':  # metaclasses may be callable
        setattr(cls, attr, wrapper_renaming(value))

There are two differences noteworthy: In Meta the super-implementation is called last and is mandatory. In Meta2 the super-implementation is called first and is optional. Another difference is the use of direct dictionary-access in Meta (dct[attr]), while in Meta2 setattr is used. I am actually not quite sure why it is not possible to use the dictionary-access in Meta2. Perhaps some of my readers can clarify that?

The metaclass is easily injected into the code:

__metaclass__ = Meta

All classes defined after that statement will use Meta as their metaclass. Simple, eh?

Conclusion

Using Python’s metaclasses it is rather easy to add functionality to existing classes without touching their code. That’s what I call minimal invasive.

I’ve collected some links to pages with more, detailed, information: MetaProgramming. Links specific to the current entry are:

• Callable types
• metaclass something similar: displaying logging information.

The code from the snippets above is put together in loggingWithMeta.py.

Aspect Oriented Programming

All this metaclass-thingy is somehow reminiscient of Aspect Oriented Programming (AOP). Displaying debugging-information like we did above is called an aspect in AOP. More general an aspect is some functionality that gets attached to some list of methods. This is done by preprocessing existing code. So AOP is minimal invasive, too. It’s more general than metaprogramming as described above, though. The following example shall be sufficient, the inspired reader may start investigating AOP from here.

Example

Let’s assume we have some classes A, B, C, and D. We would like to write debugging-information of some methods of A, B, and D to a file. Some other information on A, C, and D should be displayed on the screen. Note that class D participates on both extensions (ascpects). It’s difficult to implement these overlapping extensions via metaclasses. The more aspects are implemented the more difficult it will get to do it using metaclasses. With AOP this will be no problem. This overlapping functionality is called crosscutting in AOP. Since I am a newbie in AOP myself I won’t go into greater detail but refer to pages linked above.

Have fun.

Verwandte Artikel

• Interludium pro magus nigris (1)
• Developing large scale applications with dynamic languages (0)
• Calling hotshot, a python profiler, from command line (1)
• Being a tool-guy (0)
• A new year, a new language (2)
• Using rsync together with Subversion – Part II (5)
• Using rsync together with Subversion – a final part of the story (0)
• Using rsync together with Subversion (4)