Mutation testing is fundamentally about measuring the quality of your tests, not just their coverage.
Imagine you have a function that adds two numbers:
// src/math.js
function add(a, b) {
return a + b;
}
module.exports = { add };
And a test for it:
// src/math.test.js
const { add } = require('./math');
test('adds 1 + 2 to equal 3', () => {
expect(add(1, 2)).toBe(3);
});
This test has 100% line coverage. But what if add was accidentally changed to subtract?
// src/math.js (broken)
function add(a, b) {
return a - b; // Oops, subtraction!
}
module.exports = { add };
Your test suite would still pass. This is where mutation testing comes in. Stryker, a popular mutation testing tool, will modify (mutate) your source code in small ways, like changing + to -, > to >=, or true to false. It then runs your test suite against these mutated versions of your code. If your tests fail for a mutated version, that mutation is considered "killed," meaning your tests successfully detected the change. If your tests pass for a mutated version, that mutation "survived," indicating a potential weakness in your test suite.
Let’s set up Stryker with Jest.
First, install Stryker and its Jest mutator:
npm install --save-dev stryker stryker-jest-runner
# or
yarn add --dev stryker stryker-jest-runner
Next, create a Stryker configuration file named stryker.conf.js in your project root:
// stryker.conf.js
module.exports = {
mutator: {
name: 'javascript', // Use the default JavaScript mutator
// You can configure specific mutation operators here if needed.
// For example, to disable certain operators:
// excludedMutators: ['StringLiteral']
},
packageManager: 'npm', // or 'yarn'
reporters: ['clear-text', 'progress'], // Use clear-text for readable output and progress for real-time updates
testRunner: 'jest', // Specify Jest as the test runner
// The path to your test files. Stryker will find them based on your Jest config.
// If Jest is configured to find tests in `__tests__` or `*.test.js`, Stryker will too.
// You might need to explicitly point to your source files if they are not discoverable by Jest.
// For example:
// files: ['src/**/*.js', 'test/**/*.js'],
// Or if you have a specific test file pattern:
// testFilePatterns: ['test/**/*.spec.js'],
// It's often best to let Stryker infer from Jest's config.
// If you have specific Jest configurations (e.g., in package.json or jest.config.js),
// Stryker should pick them up.
jest: {
// You can pass Jest options here if needed,
// but it's usually better to have a separate jest.config.js or jest section in package.json
// For example:
// config: {
// testEnvironment: 'node'
// }
},
// Where to find your source code. This is crucial for Stryker to mutate.
// This should generally match what your test runner expects.
// If you have a 'src' directory:
// mutate: ['src/**/*.js'],
// If your source files are elsewhere, adjust accordingly.
// If you are using TypeScript, you'll need a different mutator and potentially a build step.
mutate: ['src/**/*.js'], // Adjust this to match your project structure
// The threshold for mutant survival. If more than 10% survive, the build will fail.
// This is a good starting point; you'll want to aim for 0% survival.
thresholds: {
high: 90, // Percentage of mutants that *must* be killed
low: 80, // Percentage below which the build will definitely fail
break: 80, // Percentage below which the build will fail
},
// Optional: If you have a build step before running tests (e.g., Babel, TypeScript)
// you might need to configure Stryker to run it.
// For example, for TypeScript:
// transpile: true,
// babelrcFile: '.babelrc', // or your babel config file
};
Now, add a script to your package.json:
// package.json
{
// ... other fields
"scripts": {
"test": "jest",
"mutation-test": "stryker run"
}
// ...
}
Run Stryker:
npm run mutation-test
# or
yarn mutation-test
Stryker will analyze your code, mutate it, run Jest against each mutation, and report the results.
Here’s what a typical output might look like:
Stryker
--------------------------------------------------------------------------------------------------
Stryker version: 5.x.x
...
INFO Starting Stryker...
INFO Running tests on 100 mutants...
INFO [JestTestRunner]: Running tests with Jest...
INFO [JestTestRunner]: Jest found 1 test suite.
INFO [JestTestRunner]: Jest command: jest --json --outputFile stryker.json
INFO [JestTestRunner]: Jest exited with code 0.
INFO [ClearTextReporter]: Mutation testing report:
--------------------------------------------------------------------------------------------------
| Status | # of mutants | Percentage |
--------------------------------------------------------------------------------------------------
| Killed | 95 | 95% |
|Survived | 5 | 5% |
|No Coverage | 0 | 0% |
|Timeout | 0 | 0% |
|Unhandled | 0 | 0% |
--------------------------------------------------------------------------------------------------
INFO [ClearTextReporter]: All tests passed for 95 mutants.
INFO [ClearTextReporter]: Tests failed for 5 mutants.
INFO [ClearTextReporter]: 0 mutants were not covered by tests.
INFO [ClearTextReporter]: 0 mutants timed out.
INFO [ClearTextReporter]: 0 mutants caused unhandled errors.
--------------------------------------------------------------------------------------------------
Total mutants: 100
Score: 95%
--------------------------------------------------------------------------------------------------
INFO Stryker finished.
In this example, 95 out of 100 mutants were killed. The 5 survivors indicate potential gaps in your tests. Stryker will often provide a detailed report (e.g., in an HTML format if you add html to reporters) showing exactly which mutants survived and where they occurred in your code.
To improve the score, you need to write new tests or modify existing ones to catch the surviving mutations. For instance, if the add function was mutated to return a - b; and it survived, you’d add a test like this:
// src/math.test.js (updated)
const { add } = require('./math');
test('adds 1 + 2 to equal 3', () => {
expect(add(1, 2)).toBe(3);
});
test('subtracts 5 - 2 to equal 3', () => { // New test to catch subtraction mutation
expect(add(5, 2)).toBe(3); // This test fails if add returns a-b
});
Running Stryker again would then show a higher score.
The most surprising thing about mutation testing is how it forces you to think about edge cases and the intent of your code, rather than just its surface-level behavior. A test like expect(add(1, 2)).toBe(3) is brittle; it only checks one specific input-output pair. Mutation testing pushes you to create tests that would break if the logic of the function changes, not just if a specific output is wrong for a specific input.
Consider a scenario where your add function also handles nulls, and you have a test for it:
// src/math.js
function add(a, b) {
if (a === null || b === null) {
return null;
}
return a + b;
}
module.exports = { add };
// src/math.test.js
test('adds null and 5 to return null', () => {
expect(add(null, 5)).toBeNull();
});
Stryker might mutate the if condition. For example, it could change a === null to a !== null. If your test suite passes, it means your existing tests aren’t sensitive enough to detect this change in null handling logic. You’d need to add more specific tests, perhaps covering different combinations of null and non-null inputs, to ensure the if condition is robustly tested.
The real power comes when you integrate this into your CI/CD pipeline. A failing mutation test run can block a merge, ensuring that new code or refactors don’t degrade the test suite’s ability to detect bugs. The goal is to reach a 100% mutation score, meaning every possible small change in your code is caught by at least one test.
When your Stryker score is high, but not 100%, and you’re struggling to find the surviving mutants, look at your mutate configuration. Sometimes, you might have files in your mutate array that aren’t actually part of your application’s runtime logic—like configuration files or utility scripts that are never directly executed. Stryker will try to mutate them, but if they aren’t run by your tests, they’ll appear as "No Coverage" or "Survived." Ensure your mutate array precisely targets your application’s source code.
After achieving a high mutation score, the next challenge is often managing the increased test suite execution time and refining the mutation operators to focus on the most critical aspects of your application’s logic.