The particular awkwardness of testing predicates

Sunday 15 January 2023 14:44

Predicates are awkward to test.

Or, to be more precise, predicates are awkward to test such that the test will reliably fail if the behaviour under test stops working.

To see why, let's look at an example: a permission check. Suppose I'm writing a system where only admins should have permission to publish articles. I might write the following predicate:

function hasPermissionArticlePublish(user: User): boolean {
    return user.isAdmin;
}

test("admins have permission to publish articles", () => {
    const user = {isAdmin: true};

    const result = hasPermissionArticlePublish(user);

    assert.isTrue(result);
});

test("users that aren't admins don't have permission to publish articles", () => {
    const user = {isAdmin: false};

    const result = hasPermissionArticlePublish(user);

    assert.isFalse(result);
});

I then realise that only active admins should have permission to publish articles, so I update the function (eliding the additional tests for brevity):

function hasPermissionArticlePublish(user: User): boolean {
    return user.isActive && user.isAdmin;
}

Since I'm using TypeScript, I'll need to update any existing tests to include the new field to keep the compiler happy:

test("users that aren't admins don't have permission to publish articles", () => {
    const user = {isActive: false, isAdmin: false};

    const result = hasPermissionArticlePublish(user);

    assert.isFalse(result);
});

Whoops! Although the test still passes, it's actually broken: since isActive is false, it'll pass regardless of the value of isAdmin. How can we prevent this sort of mistake?

Solution #1: Consistent construction of test data

The idea here is that we have a set of inputs that we know will make our unit under test behave one way, and then we make some minimal change to that set of inputs to make the unit under test behave another way.

In our example, we'd have a test for a user that does have permission:

test("active admins have permission to publish articles", () => {
    const user = {isActive: true, isAdmin: true};

    const result = hasPermissionArticlePublish(user);

    assert.isTrue(result);
});

We can extract the user from this test into a constant, and then make a minimal change to the user to cause the permission check to fail:

const userWithPermission = {isActive: true, isAdmin: true};

test("active admins have permission to publish articles", () => {
    const result = hasPermissionArticlePublish(userWithPermission);

    assert.isTrue(result);
});

test("users that aren't admins don't have permission to publish articles", () => {
    const user = {...userWithPermission, isAdmin: false};

    const result = hasPermissionArticlePublish(user);

    assert.isFalse(result);
});

This is a fairly unintrusive solution: the changes we needed to make were fairly small. The downside is that we've effectively coupled two of our tests together: our test that non-admins can't publish articles relies on another test to check that userWithPermission can indeed publish an article. As the predicate and the data become more complicated, maintaining the tests and the relationships between them becomes more awkward.

Solution #2: Testing the counter-case

To break the decoupling between test cases that our first solution introduced, we can instead test both of the cases we care about in a single test:

test("users that aren't admins don't have permission to publish articles", () => {
    const admin = {isActive: true, isAdmin: true};
    const nonAdmin = {...admin, isAdmin: false};

    const adminResult = hasPermissionArticlePublish(admin);
    const nonAdminResult = hasPermissionArticlePublish(nonAdmin);

    assert.isTrue(adminResult);
    assert.isFalse(nonAdminResult);
});

If we made the same mistake as before by setting isActive to false, then our first assertion would fail. Much like our first solution, we can now be more confident that we are indeed testing how the function under test behaves as we vary the isAdmin property, except that our confidence in this test no longer relies on another test.

Both this approach and the previous approach work less well when the predicate itself is more complex. When the predicate is checking that a set of conditions are all true, it's easy enough to take a user that satisfies all conditions and change one property so that a condition is no longer satisfied. When there are more complicated interactions between the inputs, this approach becomes trickier to use.

Solution #3: Returning more information from the function under test

A fundamental issue here is that there are many reasons why the permission check might fail, but we only get out a true or a false. In other words, the return value of the function doesn't give us enough information.

So, another solution would be to extract a function that returns the information we want, which can then be used both by the original permission check function and our tests.

function hasPermissionArticlePublish(user: User): boolean {
    return checkPermissionArticlePublish(user) === "SUCCESS";
}

type PermissionCheckResult =
    | "SUCCESS"
    | "FAILURE_USER_NOT_ACTIVE"
    | "FAILURE_USER_NOT_ADMIN";

function checkPermissionArticlePublish(user: User): PermissionCheckResult {
    if (!user.isActive) {
        return "FAILURE_USER_NOT_ACTIVE";
    } else if (!user.isAdmin) {
        return "FAILURE_USER_NOT_ADMIN";
    } else {
        return "SUCCESS";
    }
}

test("users that aren't admins don't have permission to publish articles", () => {
    const user = {isActive: true, isAdmin: false};

    const result = checkPermissionArticlePublish(user);

    assert.isEqual(result, "FAILURE_NOT_ADMIN");
});

This requires changing the code under test, but it allows us to get the answer we want (why did the permission check fail?) directly, rather than having to infer it in the tests. You'd probably also want to have a couple of test cases against hasPermissionArticlePublish directly, just to check it's using the result of checkPermissionArticlePublish correctly.

In this simple case, the extra code might not seem worth it, but being able to extract this sort of information can be increasingly useful as the condition becomes more complex. It's also a case where we might be willing to make our production code a little more complex in exchange for simplifying and having more confidence in our tests.

Conclusion

I've used them all of these techniques successfully in the past, and often switched between them as the problem being solved changes.

There are certainly other solutions -- for instance, property-based testing -- but hopefully the ones I've described give some food for thought if you find yourself faced with a similar challenge.

It's also worth noting that this problem isn't really specific to predicates: it happens any time that the function under test returns less information than would you would like to assert on in your test.

Interestingly, despite having faced this very problem many times, I haven't really seen anybody write about these specific techniques or the problem in general. It's probably just the case that I've been looking in the wrong places and my search skills are poor: pointers to other writing on the topic would be much appreciated!

Topics: Testing

Thoughts? Comments? Feel free to drop me an email at hello@zwobble.org. You can also find me on Twitter as @zwobble.

Mike's corner of the web.