Python: sharing common tests in unittest

Example code

In all the examples below, we’ll be testing these two classes which have a common interface:

class Armadillo:
    def speak(self) -> str:
        return "Hrrr!"


class Okapi:
    def speak(self) -> str:
        return "Gronk!"

With a deleted base class

This approach creates a base class, uses it, and then hides the base class with del:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi


del BaseAnimalTests  # Hide base class from test discovery

The del is needed to prevent unittest from collecting and running BaseAnimalTests itself. Without it, unittest would run it, and it would fail because it does not define the required animal_class attribute.

This approach is my preferred one. It might be a little surprising for readers to find a del, but I think the comment is explanation enough.

Running the tests, we see just the two concrete test classes executing:

$ python -m unittest -v
test_speak (tests.ArmadilloTests.test_speak) ... ok
test_speak (tests.OkapiTests.test_speak) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

With a test class mixin

This approach puts the common tests in a mixin class that does not inherit from unittest.TestCase:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests:
    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests, TestCase):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests, TestCase):
    animal_class = Okapi

No del is needed here: unittest will not collect BaseAnimalTests because it does not inherit from unittest.TestCase. But this approach has at least a couple of drawbacks:

1. If any base class forgets to inherit from both BaseAnimalTests and unittest.TestCase, it will not be collected by unittest, and its tests will not run. This may be hard to notice. Even a defence like enforcing 100% coverage on your tests, as Ned Batchelder implores we use, may not help, since subclasses may not contain any tests of their own.

2. Type checkers will raise errors about undefined methods in the base class, for example:

   $ mypy --check-untyped-defs tests.py
   tests.py:11: error: "BaseAnimalTests" has no attribute "assertIsInstance"  [attr-defined]
   tests.py:12: error: "BaseAnimalTests" has no attribute "assertGreater"  [attr-defined]
   Found 2 errors in 1 file (checked 1 source file)
   

   I previously blogged about writing type-checked mixin classes, with the conclusion being that they should inherit from their intended base class. That means returning to the first approach, where the base class is a subclass of unittest.TestCase.

With pytest: using the __test__ attribute

If you use pytest to run your unittest classes, you can use a __test__ attribute to prevent collection of a specific class:

from unittest import TestCase

from example import Armadillo, Okapi


class BaseAnimalTests(TestCase):
    __test__ = False  # Hide from test discovery

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.__test__ = True  # Enable test discovery for subclasses

    animal_class: type  # To be defined in subclasses

    def test_speak(self):
        sound = self.animal_class().speak()
        self.assertIsInstance(sound, str)
        self.assertGreater(len(sound), 0)


class ArmadilloTests(BaseAnimalTests):
    animal_class = Armadillo


class OkapiTests(BaseAnimalTests):
    animal_class = Okapi

__test__ controls whether pytest should collect a class, or not. pytest respects it as a lightly-documented feature, originally copied from the historical nose runner. The approach above hides the base class but exposes the subclasses, through some automatic configuration in __init_subclass__.

Pytest collects and runs only the subclasses, as expected:

$ pytest -v
===== test session starts ======
...
collected 2 items

test_example.py::ArmadilloTests::test_speak PASSED                                                                                                                                     [ 50%]
test_example.py::OkapiTests::test_speak PASSED                                                                                                                                         [100%]

====== 2 passed in 0.00s =======

This approach is a little bit more complex than the previous two, and it only works with pytest. Still it is nice that pytest gives you that control. If you have a lot of common test classes, the technique could be wrapped up into a class decorator.

Fin

I think it would be neat if unittest gained some functionality here, like a TestCase decorator to prevent collection of a specific class but not its subclasses. Until that hypothetical future, though, I think del is the way to go.

May your common tests work towards the common good,

—Adam