Type Annotate an existing Python Django Codebase with MonkeyType

Why Add Types?

There has been quite a buzz around type annotations in the recent past, so you may ask yourself if type checking is something you should do. This article is not a general introduction to type annotations and type checks, I recommend Hypermodern Python: Typing to get an overview of the background.

In a Nutshell citing the above article: "Type annotations are a way to annotate functions and variables with types. Combined with tooling that understands them, they can make your programs easier to understand, debug, and maintain. A static type checker can use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed." or Dropbox puts it like this: "If you aren’t using type checking in your large-scale Python project, now is a good time to get started — nobody who has made the jump I’ve talked to has regretted it. It really makes Python a much better language for large projects."

In addition to the above the tooling around type annotations is growing continuously and helps to keep your code DRY. There are libraries like Desert or Typical to help with data validation, Typeguard or strongtyping to check types at runtime, sphinx-autodoc-typehints to help build documentation, Typer to build CLIs, FastAPI a web framework for building APIs, and even transpilers to compile your python code with Mypy to Python C or Rust

Do Not Let Legacy Code Become Technical Debt

As Dropbox describes in Our journey to type checking 4 million lines of Python it is quite daunting to type annotate an existing codebase. The article describes in depth how you can approach gradual improvements with type annotations to your codebase.

Enter MonkeyType

But why do I have to write my annotations manually? After all Python knows what type a Variable is at runtime. The folks at Instagram thought the same and created MonkeyType. There is also a similar project PyAnnotate created by Dropbox.

MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

Have a look at the Introduction to MonkeyType for an overview of its functionality

Setup MonkeyType For a Django Project With PyTest

While you can run MonkeyType in a production like environment and get real world data on how you code is called, the most common use case is probably to get this data out of test runs. If you do not use pytest with django Testing Your Django App With Pytest should get you started.

Install the required dependencies by creating a file monkeytype.txt with the contents

mypy
mypy-extensions
MonkeyType
pytest-monkeytype
django-stubs
djangorestframework-stubs

and install the dependencies with pip install -r monkeytype.txt

Configure mypy with mypy.ini e.g.

[mypy]
disallow_any_generics = True
disallow_untyped_calls = True
disallow_untyped_defs = True
disallow_untyped_decorators = True
ignore_errors = False
ignore_missing_imports = True
implicit_reexport = False
strict_optional = True
strict_equality = True
no_implicit_optional = True
warn_unused_ignores = True
warn_redundant_casts = True
warn_unused_configs = True
warn_unreachable = True
warn_no_return = True
warn_return_any = True

plugins =
    mypy_django_plugin.main,
    mypy_drf_plugin.main

[mypy.plugins.django-stubs]
django_settings_module = "myproject.settings"

[mypy-accounts.tests.*]
ignore_errors = True

[mypy-api_client.tests.*]
ignore_errors = True

[mypy-core_app.tests.*]
ignore_errors = True

[mypy-core_app.management.*]
ignore_errors = True

[mypy-core_app.migrations.*]
ignore_errors = True

# yet to be annoted:
[mypy-myproject.*]
ignore_errors = True

#External libraries without type annotation
[mypy-celery.*]
ignore_missing_imports = True

[mypy-django_extensions.*]
ignore_missing_imports = True

Create a monkeytype_config.py file:

# Standard Library
import os
from contextlib import contextmanager

# 3rd-party
from monkeytype.config import DefaultConfig


class MonkeyConfig(DefaultConfig):
    @contextmanager
    def cli_context(self, command):
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
        import django

        django.setup()
        yield


CONFIG = MonkeyConfig()

Change the DJANGO_SETTINGS_MODULE to the one that is used in tests. Now you can run your tests as usual and MonkeyType will collect runtime types of function arguments and return values. By default, this will dump call traces into a SQLite database in the file monkeytype.sqlite3 in the current working directory.

If for any reason (NFS) SQLite cannot use the working directory you can instruct MonkeyType to use another location with e.g. pytest --monkeytype-output=/tmp/monkeytype.sqlite3 and export MT_DB_PATH /tmp/monkeytype.sqlite3 on Bash, or setenv MT_DB_PATH /tmp/monkeytype.sqlite3 on C Shell.

Check which modules MonkeyType has collected infomation for with monkeytype list-modules.

You can then use the monkeytype stub some.module command to generate a stub file for a module, or apply the type annotations directly to your code with monkeytype apply some.module.

Remember that MonkeyType’s annotations are an informative first draft, to be checked and corrected by a developer. Start with small modules that have few (or even better no) dependencies on other parts of your code, apply the types, check the correctness of the annotations with mypy some/module.py fix the typing information, and move on to the next module.

Little Helpers

You will want to keep your type information clean, readable and consistent. To achieve this there are some plugins for flake8.

flake8-annotations-complexity reports on too complex, hard to read type annotations. Complex type annotations often means bad annotation usage, wrong code decomposition or improper data structure choice.

flake8-annotations-coverage reports on files with a lot of code without type annotations. This is mostly useful when you add type annotations to existing large codebase and want to know if new code in annotated modules is annotated.

flake8-type-annotations is used to validate type annotations syntax as it was originally proposed

flake8-annotations detects the absence of function annotations. What this won't do: Check variable annotations respect stub files, or replace mypy.