Mutation Testing in CI

[tob-scott-a]

mull is a mutation testing framework for C. It can take a long time (depending on the project), and you have to set ulimit -n 1024 before any mull-runner commands on Mac OS X or else you get a report full of useless errors.

However, mutation testing is generally useful at finding gaps in test coverage. How it works is simple: It runs your test suite and identifies test coverage, then it changes your source code, AST, or binary. This is called a mutant. If the tests that touched that part of the code still pass, the mutant is said to escape. That indicates it was either a false positive (equivalent statements) or the tests didn't adequately cover it.

From my latest run before the changes in my recent PRs:

Mutation testing (Mull 0.29.0, LLVM 18) was run against libsodium (commit 148d2d60). After deduplication: 4,918 mutants, 83.6% mutation score, 92.5% adjusted (excluding false positives). 251 survived mutants indicate genuine missing test coverage across 82 functions.

This is a respectable percentage for any project that wasn't already running mutation testing in CI.

Mutation testing complements fuzzing. I think it's worth considering as an addition. Happy to help with this, of course.