The Metaprogramming In Production On Python Part 1

Many people think that metaprogramming in Python unnecessarily complicates the code, but if you use it correctly, you can quickly and elegantly implement complex design patterns. In addition, well-known Python frameworks such as Django, DRF, and SQLAlchemy use metaclasses to provide easy extensibility and simple code reuse.

In this article, I’ll tell you why you shouldn’t be afraid to use metaprogramming in your projects and show you what tasks it is best for. You can learn more about metaprogramming capabilities in the Advanced Python course.

For first, let’s recall the basics of metaprogramming in Python. It is not superfluous to add that all that is written below refers to the version of Python 3.5 and higher.

A brief excursion into the Python data model

So, we all know that everything in Python is an object, and it is no secret that for each object there is a certain class by which it was generated, for example:

>>> def f(): pass
>>> type(f)
<class 'function'>

The type of the object or the class by which the object was spawned can be determined using the built-in type function, which has a rather interesting call signature (it will be discussed later). The same effect can be achieved if you display the attribute __class__ from any object.

So, to create functions is a kind of built-in function. Let’s see what we can do with it. To do this, take the blank from the built-in types module:

>>> from types import FunctionType
>>> FunctionType
<class 'function'>
>>> help(FunctionType)

class function(object)
| function(code, globals[, name[, argdefs[, closure]]])
|
| Create a function object from a code object and a dictionary.
| The optional name string overrides the name from the code object.
| The optional argdefs tuple specifies the default argument values.
| The optional closure tuple supplies the bindings for free variables.

As we can see, any function in Python is an instance of the class described above. Let’s now try to create a new function without resorting to its declaration via def. To do this, we need to learn how to create code objects using the built-in interpreter function compile:

# create a code object that prints the string "Hello, world!"
>>> code = compile ('print ("Hello, world!")', '<repl>', 'eval')
>>> code
<code object <module> at 0xdeadbeef, file "<repl>", line 1>
# create a function by passing the code object to the constructor,
# global variables and function name
>>> func = FunctionType (code, globals (), 'greetings')
>>> func
<function <module> at 0xcafefeed>
>>> func .__ name__
'greetings'
>>> func ()
Hello, world!

Fine! With the help of meta-tools, we learned to create functions on the fly, but in practice, this knowledge is rarely used. Now let’s take a look at how class objects and instance objects of these classes are created:

>>> class User: pass
>>> user = User()
>>> type(user)
<class '__main__.User'>
>>> type(User)
<class 'type'>

Obviously, the User class is used to create an instance of the user, it is much more interesting to look at the type class, which is used to create the User class itself. Here we turn to the second variant of calling the built-in type function, which is also a metaclass for any class in Python. A metaclass is, by definition, a class whose instance is another class. Metaclasses allow us to customize the process of creating a class and partially manage the process of creating an instance of a class.

According to the documentation, the second variant of the type (name, bases, attrs) returns a new data type or, if simple, a new class, and the name attribute becomes the __name__ attribute of the returned class, bases – the list of parent classes will be available as __bases__ and attrs, a dict-like object containing all attributes and methods of the class, will become __dict__. The principle of operation of the function can be described as a simple pseudocode in Python:

type(name, bases, attrs)
~
class name(bases):
attrs

Let’s see how we can, using only the type call, construct a completely new class:

>>> User = type('User', (), {})
>>> User
<class '__main__.User'>

As you can see, we do not need to use the class keyword to create a new class, the type function copes without it, now let’s consider an example more complicated:

class User:
def __init __ (self, name):
self.name = name

class SuperUser (User):
"" "Encapsulate domain logic to work with super users" ""
group_name = 'admin'

@property
def login (self):
return f '{self.group_name} / {self.name}'. lower ()

# Now create an analogue of the class SuperUser "dynamically"
CustomSuperUser = type (
# Class name
'SuperUser',
# List of classes from which the new class is inherited
(User),
# Attributes and methods of the new class in the form of a dictionary
{
'__doc__': 'Encapsulate the domain logic to work with super users',
'group_name': 'admin',
'login': property (lambda self: f '{self.group_name} / {self.name}'. lower ()),
}
)

assert SuperUser .__ doc__ == CustomSuperUser .__ doc__
assert SuperUser ('Vladimir'). login == CustomSuperUser ('Vladimir'). login

As can be seen from the examples above, the description of classes and functions using the class and def keywords is just syntactic sugar, and any types of objects can be created with ordinary calls to built-in functions. And now, finally, let’s talk about how you can use dynamic class creation in real projects.

Dynamic creation of forms and validators

Sometimes we need to validate information from the user or from other external sources according to a previously known data scheme. For example, we want to change the user login form from the admin panel — remove and add fields, change their validation strategy, etc.

To illustrate this, we will try to dynamically create a Django form, the description of the scheme of which is stored in the following json format:

{
"fist_name": { "type": "str", "max_length": 25 },
"last_name": { "type": "str", "max_length": 30 },
"age": { "type": "int", "min_value": 18, "max_value": 99 }
}

Now, based on the description above, we will create a set of fields and a new form using the type of function we already know:

import json
from django import forms

fields_type_map = {
'str': forms.CharField,
'int': forms.IntegerField,
}

# form_description - our json format description
deserialized_form_description: dict = json.loads (form_description)
form_attrs = {}

# select the class of the object of the field in the form, depending on its type
for field_name, field_description in deserialized_form_description.items ():
field_class = fields_type_map [field_description.pop ('type')]
form_attrs [field_name] = field_class (** field_description)

user_form_class = type ('DynamicForm', (forms.Form,), form_attrs)

>>> form = user_form_class ({'age': 101})
>>> form
<DynamicForm bound = True, valid = Unknown, fields = (fist_name; last_name; age)>
>>> form.is_valid ()
False
>>> form.errors
{'fist_name': ['This field is required.'],
'last_name': ['This field is required.'],
'age': ['Ensure this value is less than or equal to 99.']}

Super! Now you can transfer the created form to a template and render it to the user. The same approach can be used with other frameworks for validation and presentation of data (DRF Serializers, marshmallow, and others).

Configuring the creation of a new class through the metaclass

Above, we looked at the “ready” metaclass type, but most often in the code, you will create your own metaclasses and use them to configure the creation of new classes and their instances. In general, the “blank” metaclass looks like this:

class MetaClass (type):
"" "
Description of accepted parameters:

mcs is a metaclass object, for example <__ main __. MetaClass>
name - string, the name of the class for which it is used
This metaclass, for example "User"
bases - a tuple of parent classes, for example (SomeMixin, AbstractUser)
attrs - a dict-like object that stores the values of attributes and methods of a class.
cls - created class, for example <__ main __. User>
extra_kwargs - additional keyword arguments passed to the class signature
args and kwargs - the arguments passed to the class constructor
when creating a new instance
"" "
def __new __ (mcs, name, bases, attrs, ** extra_kwargs):
return super () .__ new __ (mcs, name, bases, attrs)

def __init __ (cls, name, bases, attrs, ** extra_kwargs):
super () .__ init __ (cls)

@classmethod
def __prepare __ (mcs, cls, bases, ** extra_kwargs):
return super () .__ prepare __ (mcs, cls, bases, ** kwargs)

def __call __ (cls, * args, ** kwargs):
return super () .__ call __ (* args, ** kwargs)

To use this metaclass to configure the User class, use the following syntax:

class User (metaclass = MetaClass):

def __new __ (cls, name):
return super () .__ new __ (cls)

def __init __ (self, name):
self.name = name

The most interesting thing is the order in which the Python interpreter invokes the metamethods of the metaclass at the time of the creation of the class itself:

1. The interpreter identifies and finds parent classes for the current class (if any).
2. The interpreter defines the metaclass (MetaClass in our case).
3. The method is called MetaClass.__ prepare__ – it must return a dict-like object in which the attributes and methods of the class will be written. After that, the object will be passed to the method MetaClass.__ new__ through the argument attrs. We will talk about the practical use of this method a little later in the examples.
4. The interpreter reads the body of the User class and forms the parameters for transferring them to the MetaClass.
5. The method is called MetaClass.__ new__ is a constructor method, returns the created class object. We have already met with the arguments name, bases and attrs when we passed them to the type function, and we will talk about the parameter ** extra_kwargs a bit later. If the type of the attrs argument has been changed using __prepare__, then it must be converted to dict before being passed to the super () method call.
6. The method is invoked MetaClass.__ init__ – an initializer method with which you can add additional attributes and methods to a class object. In practice, it is used in cases when metaclasses are inherited from other metaclasses, otherwise, everything that can be done in __init__, it is better to do in __new__. For example, the __slots__ parameter can be set only in the __new__ method by writing it to the attrs object.
7. At this step, the class is considered to be created.

And now let’s create an instance of our User class and look at the call chain:

 

1. At the moment User (…) is called, the interpreter calls the method MetaClass.__call __ (name = ‘Alyosha’), where it passes the class object and the arguments passed.
2. MetaClass .__ call__ calls User .__ new __ (name = ‘Alyosha’) – a constructor method that creates and returns an instance of the User class
3. Next, MetaClass .__ call__ calls User .__ init __ (name = ‘Alyosha’) – an initialization method that adds new attributes to the created instance.
4. MetaClass .__ call__ returns the created and initialized instance of the User class.
5. At this point, an instance of the class is considered to be created.

This description, of course, does not cover all the nuances of the use of metaclasses, but it is enough to start applying metaprogramming to implement some architectural patterns. Forward to the examples!

Abstract classes

And the very first example can be found in the standard library: ABCMeta – the metaclass allows you to declare any of our classes to be abstract and force all of its heirs to implement predefined methods, properties, and attributes, look at this:

from abc import ABCMeta, abstractmethod

class BasePlugin (metaclass = ABCMeta):
"" "
The supported_formats class attribute and the run method must be implemented
in the heirs of this class
"" "
@property
@abstractmethod
def supported_formats (self) -> list:
pass

@abstractmethod
def run (self, input_data: dict):
pass

If all abstract methods and attributes are not implemented in the heir, then when we try to create an instance of the heir class, we will get a TypeError:

class VideoPlugin(BasePlugin):

def run(self):
print('Processing video...')

plugin = VideoPlugin()
# TypeError: Can't instantiate abstract class VideoPlugin
# with abstract methods supported_formats

Using abstract classes helps to fix the interface of the base class immediately and to avoid errors in future inheritance, for example, typos in the name of the overridden method.

Plugin system with automatic registration

Quite often, metaprogramming is used to implement various design patterns. Almost any known framework uses metaclasses to create registry objects. Such objects store references to other objects and allows them to be quickly received anywhere in the program. Consider a simple example of auto-registration of plug-ins for playing media files of various formats.

Metaclass implementation:

class RegistryMeta (ABCMeta):
"" "
The metaclass that creates the registry from the classes of heirs.
The registry stores links like "file format" -> "plugin class"
"" "
_registry_formats = {}

def __new __ (mcs, name, bases, attrs):
cls: 'BasePlugin' = super () .__ new __ (mcs, name, bases, attrs)

# do not handle abstract classes (BasePlugin)
if inspect.isabstract (cls):
return cls

for media_format in cls.supported_formats:
if media_format in mcs._registry_formats:
raise ValueError (f'Format {media_format} is already registered ')

# save link to plugin in registry
mcs._registry_formats [media_format] = cls

return cls

@classmethod
def get_plugin (mcs, media_format: str):
try:
return mcs._registry_formats [media_format]
except KeyError:
raise RuntimeError (f'Plugin is not defined for {media_format} ')

@classmethod
def show_registry (mcs):
from pprint import pprint
pprint (mcs._registry_formats)

And here are the plugins themselves, let’s take the BasePlugin implementation from the previous example:

class BasePlugin(metaclass=RegistryMeta):
...

class VideoPlugin(BasePlugin):
supported_formats = ['mpg', 'mov']
def run(self): ...

class AudioPlugin(BasePlugin):
supported_formats = ['mp3', 'flac']
def run(self): ...

After the interpreter executes this code, 4 formats and 2 plug-ins that can process these formats will be registered in our registry:

>>> RegistryMeta.show_registry()
{'flac': <class '__main__.AudioPlugin'>,
'mov': <class '__main__.VideoPlugin'>,
'mp3': <class '__main__.AudioPlugin'>,
'mpg': <class '__main__.VideoPlugin'>}
>>> plugin_class = RegistryMeta.get_plugin('mov')
>>> plugin_class
<class '__main__.VideoPlugin'>
>>> plugin_class().run()
Processing video...

Here it is worth noting another interesting nuance of working with metaclasses, thanks to the unobvious method resolution order, we can call the show_registry method not only on the RegistyMeta class but on any other class whose metaclass is:

>>> AudioPlugin.get_plugin('avi')
# RuntimeError: Plugin is not found for avi

Using attribute names as metadata

Using metaclasses, you can use class attribute names as metadata for other objects. Nothing is clear? But I’m sure you’ve already seen this approach many times, for example, the declarative declaration of model fields in Django:

class Book(models.Model):
title = models.Charfield(max_length=250)

In the example above, the title is the name of the Python identifier; it is also used for the title of the column in the book table, although we have not explicitly indicated this anywhere. Yes, such “magic” can be implemented using metaprogramming. Let us, for example, implement an application error transfer system on the front-end, so that each message has readable code that can be used to translate a message into another language. So, we have a message object that can be converted to json:

class Message:
def __init__(self, text, code=None):
self.text = text
self.code = code

def to_json(self):
return json.dumps({'text': self.text, 'code': self.code})

All our error messages will be stored in a separate “namespace”:

class Messages:
not_found = Message('Resource not found')
bad_request = Message('Request body is invalid')
...

>>> Messages.not_found.to_json()
{"text": "Resource not found", "code": null}

Now we want the code to become not null,  but not_found, for this we will write the following metaclass:

class MetaMessage (type):

def __new __ (mcs, name, bases, attrs):
for attr, value in attrs.items ():
# pass through all the attributes described in the class with the Message type
# and replace the code field with the attribute name
# (if code is not set in advance)
if isinstance (value, Message) and value.code is None:
value.code = attr

return super () .__ new __ (mcs, name, bases, attrs)

class Messages (metaclass = MetaMessage):
...

Let’s see how our messages look now:

>>> Messages.not_found.to_json()
{"text": "Resource not found", "code": "not_found"}
>>> Messages.bad_request.to_json()
{"text": "Request body is invalid", "code": "bad_request"}

What you need! Now you know what to do so that by the format of the data you can easily find the code that processes them.

Caching class metadata and its heirs

Another frequent case is caching of any static data at the stage of class creation, in order not to waste time on their calculation while the application is running. In addition, some data can be updated when creating new instances of classes, for example, the count of the number of objects created.

How can this be used? Suppose you are developing a framework for building reports and tables, and you have such an object:

class Row (metaclass = MetaRow):
name: str
age: int
...
def __init __ (self, ** kwargs):
self.counter = None
for attr, value in kwargs.items ():
setattr (self, attr, value)

def __str __ (self):
out = [self.counter]

# attribute __header__ will be dynamically added in the metaclass
for name in self .__ header __ [1:]:
out.append (getattr (self, name, 'N / A'))

return '| '.join (map (str, out))

We want to save and increase the counter when creating a new series, and also want to generate the header of the resulting table in advance. Metaclass to the rescue!

class MetaRow (type):
# global counter of all rows created
row_count = 0

def __new __ (mcs, name, bases, attrs):
cls = super () .__ new __ (mcs, name, bases, attrs)

# Cache a list of all fields in a row sorted alphabetically
cls .__ header__ = ['No.'] + sorted (attrs ['__ annotations __']. keys ())
return cls

def __call __ (cls, * args, ** kwargs):
# creating a new series takes place here
row: 'Row' = super () .__ call __ (* args, ** kwargs)
# increment global counter
cls.row_count + = 1

# set the current row number
row.counter = cls.row_count
return row

Here you need to clarify 2 things:

• The Row class has no class attributes with the names name and age — these are type annotations, so they are not in the attrs dictionary keys, and to get the list of fields, we use the class attribute __annotations__.
• The operation cls.row_count + = 1 should have misled you: how is that? After all, cls is a class Row it does not have an attribute row_count. That’s right, but as I explained above – if the created class does not have an attribute or method that they are trying to call, then the interpreter goes further along the chain of base classes – if there is no one, the metaclass is searched. In such cases, in order not to confuse anyone, it is better to use another record: MetaRow.row_count + = 1.

See how elegantly you can now display the entire table:

rows = [
Row(name='Valentin', age=25),
Row(name='Sergey', age=33),
Row(name='Gosha'),
]

print(' | '.join(Row.__header__))
for row in rows:
print(row)

№ | age | name
1 | 25 | Valentin
2 | 33 | Sergey
3 | N/A | Gosha

By the way, the display and work with the table can be encapsulated in any separate class Sheet.

To be continued…

In the next part of this article, I will explain how to use metaclasses to debug your application code, how to parameterize the creation of a metaclass and show the main examples of using the __prepare__ method. Stay tuned!

In more detail, about metaclasses and descriptors in Python, I will talk in the framework of Advanced Python intensive.