Test driven development

When writing code you want to make sure the code works as intended, previously identified bugs stay fixed, and the code being written can solve the desired research question. Test driven development is a way to systematically achieve these goals.

What are tests

The basic function of a test is to pass a given input to your code that has a known output and check that the output matches your expectations. The key here is the known output you are testing against comes from some source that is not your code (e.g. an analytic solution, work it out by hand, some other code you know works, etc…).

These test cases are written as functions within your code and can be run at any time to ensure any changes made in the code have not broken these test cases. Using CI you can set up your code repository to run these tests any time a new PR is made and block the PR from merging if any of the tests fail.

When bugs arise in your code, it is best practice to first write a new test that reproduces the bug before fixing the bug. That way you can be sure the bug is identified, fixed, and changes made in the future will not reintroduce the same bug into your code.

Once you become comfortable with writing tests you may find it easier to write your tests first and the code second. Doing this encourages you to think about the main goals of the code you are about to write, and when the tests pass you know the code is finished. This process of writing the test first is know as test driven development.

Python’s built in unittest library

Python comes with a built in testing library called unittest that includes full object mocking capabilities. Unittest organizes tests into classes (subclassed from unittest.TestCase) where each test is a method on the class that has a name starting with test_. There are also special setUp and tearDown methods that, if defined, will run before and after each test method. This class also provides various assert functions such as self.assertEqual, self.assertDictEqual, and self.assertGreater (see API documentation for full list) that provide more useful error messages when they fail than a plain assert statement does. All the examples in this workshop use unittest’s framework and syntax.

Python will auto-discover any tests that are in your current directory by looking for any files that follow the pattern test*.py by default (this search pattern can be changed when the unittest command is run). The test file must be importable from the top level directory of the project (i.e. a blank __init__.py is in the folders containing any tests). Test can be run with the command:

python -m unittest discover

If this command does not find your tests you can specify various command line options to help it out, see the discover documentation for more information.

Other python testing libraries

The other commonly used python testing library is pytest. It has its own syntax for writing tests that is more compact than unittest, but can come at the cost of clarity for more complex tests. Pytest is also fully compatible with tests written in unittest syntax.

Note

VScode uses pytest to run tests inside the editor, if you are using unittest’s self.subTest syntax you will need to install the pytest-subtests package for them to run as expected.

Warning

You might also see nosetest used by some older packages, this testing framework is no longer in active development and should not be considered for new projects.

Types of tests

There are several flavors of tests that can be written for code. The main three I have come across are:

  • Unit tests

  • Integration tests

  • Validation tests

While these are not the only three types of tests, they are a good starting point for writing your own tests.

Note

These are the names that make the most sense to me, but others might use different names for the same kinds of tests.

Unit tests

The goal of unit tests are to test a “unit” of code (e.g. a single function definition or a single class method) in isolation from all the other units of code that make up the software. These test are mostly for checking if the unit of code works as written (i.e. the algorithm was coded correctly). Here is an example of a simple unit test:

def func(a, N=1):
    return a + N

import unittest
from package.file import func


class TestFunc(unittest.TestCase):
    def test_func_default_N(self):
        '''Test func with default N value'''
        expected = 3
        result = func(2)
        self.assertEqual(result, expected)
    
    def test_func_with_N(self):
        '''Test func with N keyword set'''
        expected = 4
        result = func(2, N=2)
        self.assertEqual(result, expected)

Integration tests

The goal of integration tests are to test if multiple units of code work together correctly. For example if you had a function who’s only job is to call two other functions in a particular order and pass the results from one function into the next, an integration test would check exactly these things. It would not check that the two sub-functions return the correct values (that is the job of a unit test for those sub-functions), just that that values passed back are sent to the correct place.

def func_1(a):
    return a + 1


def func_2(b):
    return b + 2


def run_all(a):
    b = func_1(a)
    c = func_2(b)
    return c

import unittest
from unittest.mock import patch
from package.file import run_all


class TestRunAll(unittest.TestCase):
    def test_run_all_value(self):
        '''Test run_all give expected value'''
        # a traditional unit test to check an known input gives a known output
        expected = 6
        result = run_all(3)
        self.assertEqual(result, expected)

    @patch('package.file.func_2')
    @patch('package.file.func_1')
    def test_run_all_calls(self, mock_func_1, mock_func_2):
        '''Test run_all calls the inner functions with the expected values'''
        # an integration test to check data is being pass around as expected
        
        # we can input any value, the `result` will not change because of the mocks
        input_value = 3
        result = run_all(input_value)
        
        # test func_1 called with the argument provided to run_all
        mock_func_1.assert_called_once_with(input_value)

        # test func_2 called with the return value of func_1
        mock_func_2.assert_called_once_with(mock_func_1.return_value)

        # test run_all returns the func_2 return value
        self.assertEqual(result, mock_func_2.return_value)

Warning

When writing integration tests (or any tests using mocks) it is very easy to make a test that is too strongly coupled with the code. Any adjustments to the source code would require the test to be adjusted as well. The unit test in this example is not strongly coupled, the integration test is.

To help avoid this, when you have finished writing a test read it over and ask yourself “what is the main goal of the code being tests and do these test check only that? Is a future refactor likely to break this test?”

If the main goal run_all is to get the expected result out the other end, the unit test is sufficient. If the main goal of run_all is to compose func_1 and func_2 the integration test is sufficient.

Validation tests

The goal of validation tests are to check if the code written is appropriate for problem you are trying to solve (i.e. is the code using the correct algorithm). For example, can you get away with using a 2nd order ODE integrator or do you need to using a 4th order one to keep the desired level of accuracy. Often these are just a flavor of an integration test or unit test, but with “science quality” data being passed in.

Test coverage

Test coverage is a way of tracking what lines of your source code were run when each of your tests ran. When writing tests for a function you goal should be to have as much coverage as possible. This typically means making sure you have at least one test for every branch in your code (i.e. one test for each if branch in the code). Sometimes when writing your tests you will realize that there are branches in your code that can never be reached and can be safely removed from the code.

To check test coverage you can use the coverage package, once installed it will wrap around your chosen test runner and check how many lines of you package’s code were run by your tests. After the tests are run you can generate a report to see how many lines were covered. The basic usage is:

coverage run --source data_transforms --omit *test* -m unittest discover
coverage report --show-missing

The options passed to run are:

  • --source data_transforms: base folder for the package

  • --omit *test*: pattern of files not to include in coverage percent

  • -m unittest discover: the test runner command

The option passed to report is:

  • --show-missing: for each file list the line number(s) not covered by any tests

These extra command line options can be included in your project’s pyproject.toml file (already done in this workshop’s repo) to simplify these commands to:

coverage run
coverage report

This leaves you with an easy to remember command for running your tests with all the project specific options placed in a single configuration file.

Mocks

In the integration tests section above the example makes use of what are called “function mocks.” These are special classes that are used to fully replace another function or class. This mock will keep track of how many times it is called, what arguments and keywords it was called with, and what it should return when called.

The easiest way to use a mock is with the @patch decorator, this takes the full import path of the object to be mocked, and an optional return_value specifying what value should be returned when the mock is called. By default any method or return value accessed on a mock is also a mock, so there is no need to know the full object spec beforehand.

Note

The import path should be an absolute path to the object being mocked starting at the top level of your package.

The main use of mocks are:

  • testing pipeline code passes data around correctly

  • replace functions that take a long time to run with an object that returns instantly

  • ensuring that bugs in sub-functions don’t make other functions’ tests fail (i.e. only testing the code you wrote and not the code coming from other packages).

  • avoid having to read/write data to disk (unittest.mock.mock_open is its own function that helps when you need to test code that opens files)

Example, we want to mock away a class that has several methods (we will use a with block this time instead of the decorator syntax):

from unittest.mock import patch, call

# make an instance of a class we will mock
thing = object()

# Set what the mocked `method_1` and `method_2` should return
config = {
    'method_1.return_value': 5,
    'method_2.side_effect': [1, 2, 3]
}


with patch('__main__.thing', **config) as mock_MyClass:
    # check the mock applied
    assert thing is mock_MyClass

    # check method_1 returns what we set
    assert thing.method_1() == 5

    # how to test it is called exactly once
    mock_MyClass.method_1.assert_called_once()

    # check method_2 returns each values set in turn
    assert thing.method_2(a=5) == 1
    assert thing.method_2(a=6) == 2
    assert thing.method_2(a=7) == 3

    # how to test it was called three times with specific arguments/keywords each time
    expected_calls = [
        call(a=5),
        call(a=6),
        call(a=7)
    ]
    mock_MyClass.method_2.assert_has_calls(expected_calls, any_order=False)

    # how to test it was called three times (without specifying the explicit calls)
    assert mock_MyClass.method_2.call_count == 3

    # any method and return value generates a new mock that tracks calls
    thing.method_3().sub_method(a=1, b=2)

    # test this sub_method was called once with specific keywords
    mock_MyClass.method_3.return_value.sub_method.assert_called_once_with(a=1, b=2)

    # define new return_values on the mock object directly rather than passing them
    # into **config
    mock_MyClass.method_4.return_value = 'new value'

    assert thing.method_4() == 'new value'

# check the mock is no longer applied outside the with block
assert thing is not mock_MyClass

See https://docs.python.org/3/library/unittest.mock.html for more examples of mocks and their uses.