Mutation Testing
Measure test quality with Stryker mutation testing
Mutation Testing
Mutation testing measures the quality of your tests by introducing small bugs (mutations) into your code and checking if your tests catch them. High code coverage doesn't mean good tests—mutation testing tells you if your tests actually work.
The Problem with Code Coverage
// This has 100% code coverage...
function add(a: number, b: number): number {
return a + b;
}
test('add function works', () => {
add(2, 3); // Runs the function, but...
});
// But this test doesn't verify anything!
// It would still pass if add() returned 0, null, or even a - bCode Coverage ≠ Test Quality
Mutation testing solves this by:
- Creating mutants (changing
+to-,>to<, etc.) - Running your tests against mutants
- Tests should kill mutants (fail when code is wrong)
What is a Mutant?
// Original code
function isAdult(age: number): boolean {
return age >= 18; // ← This is the original
}
// Mutant 1: Change >= to >
function isAdult(age: number): boolean {
return age > 18; // Age 18 is now false!
}
// Mutant 2: Change >= to <=
function isAdult(age: number): boolean {
return age <= 18; // Inverted logic!
}
// Mutant 3: Change 18 to 19
function isAdult(age: number): boolean {
return age >= 19; // Wrong threshold!
}
// Good tests should kill all these mutantsStryker Setup
Installation
npm install --save-dev @stryker-mutator/core
npx stryker initConfiguration
// stryker.config.mjs
/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
const config = {
packageManager: 'npm',
reporters: ['html', 'clear-text', 'progress'],
testRunner: 'jest',
coverageAnalysis: 'perTest',
// Which files to mutate
mutate: [
'src/**/*.ts',
'src/**/*.tsx',
'!src/**/*.test.ts',
'!src/**/*.test.tsx',
'!src/**/*.spec.ts',
],
// Thresholds
thresholds: {
high: 80,
low: 60,
break: 75, // Fail if mutation score < 75%
},
// Ignore specific files
mutator: {
excludedMutations: [
'StringLiteral', // Don't mutate string literals
'ObjectLiteral', // Don't mutate object literals
],
},
// Timeout
timeoutMS: 60000,
timeoutFactor: 1.5,
};
export default config;Running Stryker
// package.json
{
"scripts": {
"test:mutation": "stryker run"
}
}npm run test:mutationUnderstanding Results
Mutation Score
Mutation Score = (Killed Mutants / Total Mutants) × 100%Example Output:
All tests: 18 passed
Mutants: 50 tested
- Killed: 40 (80%)
- Survived: 8 (16%)
- Timeout: 2 (4%)
Mutation Score: 80%Mutant Status
Killed ✅: Test failed when mutant was introduced (good!)
Survived ❌: All tests passed with mutant (bad!)
Timeout ⏱️: Tests took too long (infinite loop?)
No Coverage ⚠️: No tests run this code
Runtime Error 💥: Mutant broke the code (neutral)
Example: Improving Tests
Original Code
// utils/validation.ts
export function isValidEmail(email: string): boolean {
return email.includes('@') && email.length > 5;
}Weak Test (Survivors)
// ❌ BAD: Test with poor assertions
test('validates email', () => {
isValidEmail('test@example.com');
// Doesn't check return value!
});
// Stryker creates mutants:
// - Mutant 1: email.includes('@') → !email.includes('@')
// - Mutant 2: email.length > 5 → email.length < 5
// - Mutant 3: && → ||
// All these mutants SURVIVE because we don't assert!Strong Test (Kills Mutants)
// ✅ GOOD: Test with proper assertions
test('validates email correctly', () => {
// Test true cases
expect(isValidEmail('test@example.com')).toBe(true);
expect(isValidEmail('a@b.co')).toBe(true);
// Test false cases (kills boundary mutants)
expect(isValidEmail('test')).toBe(false); // No @
expect(isValidEmail('t@e')).toBe(false); // Too short
expect(isValidEmail('@test')).toBe(false); // Just @
expect(isValidEmail('')).toBe(false); // Empty
});
// Now all mutants are KILLED!Common Mutators
Arithmetic Mutator
// Original → Mutant
a + b → a - b
a - b → a + b
a * b → a / b
a / b → a * b
a % b → a * bComparison Mutator
a > b → a >= b, a < b, a <= b
a >= b → a > b, a < b, a <= b
a < b → a <= b, a > b, a >= b
a <= b → a < b, a > b, a >= b
a === b → a !== b
a !== b → a === bLogical Mutator
a && b → a || b, true, false
a || b → a && b, true, false
!a → aConditional Mutator
if (condition) { } → if (true) { }, if (false) { }
while (condition) { } → while (false) { }String/Number Mutator
"hello" → ""
123 → 0, 1
true → falseReal-World Example
Code Under Test
// services/cart.ts
export class ShoppingCart {
private items: CartItem[] = [];
addItem(product: Product, quantity: number): void {
if (quantity <= 0) {
throw new Error('Quantity must be positive');
}
const existingItem = this.items.find(item => item.product.id === product.id);
if (existingItem) {
existingItem.quantity += quantity;
} else {
this.items.push({ product, quantity });
}
}
getTotal(): number {
return this.items.reduce((sum, item) => {
return sum + (item.product.price * item.quantity);
}, 0);
}
isEmpty(): boolean {
return this.items.length === 0;
}
}Comprehensive Tests
// services/cart.test.ts
describe('ShoppingCart', () => {
let cart: ShoppingCart;
beforeEach(() => {
cart = new ShoppingCart();
});
describe('addItem', () => {
it('adds new item to cart', () => {
const product = { id: 1, name: 'Product 1', price: 10 };
cart.addItem(product, 2);
expect(cart.getTotal()).toBe(20);
});
it('increases quantity if item already exists', () => {
const product = { id: 1, name: 'Product 1', price: 10 };
cart.addItem(product, 2);
cart.addItem(product, 3);
expect(cart.getTotal()).toBe(50); // 5 items × $10
});
it('throws error for zero quantity', () => {
const product = { id: 1, name: 'Product 1', price: 10 };
expect(() => cart.addItem(product, 0)).toThrow('Quantity must be positive');
});
it('throws error for negative quantity', () => {
const product = { id: 1, name: 'Product 1', price: 10 };
expect(() => cart.addItem(product, -1)).toThrow('Quantity must be positive');
});
});
describe('getTotal', () => {
it('returns 0 for empty cart', () => {
expect(cart.getTotal()).toBe(0);
});
it('calculates total correctly', () => {
cart.addItem({ id: 1, name: 'P1', price: 10 }, 2);
cart.addItem({ id: 2, name: 'P2', price: 5 }, 3);
expect(cart.getTotal()).toBe(35); // (10×2) + (5×3)
});
});
describe('isEmpty', () => {
it('returns true for new cart', () => {
expect(cart.isEmpty()).toBe(true);
});
it('returns false when items added', () => {
cart.addItem({ id: 1, name: 'P1', price: 10 }, 1);
expect(cart.isEmpty()).toBe(false);
});
});
});
// This test suite should have high mutation score (90%+)CI/CD Integration
# .github/workflows/mutation-testing.yml
name: Mutation Testing
on:
pull_request:
branches: [main]
schedule:
- cron: '0 2 * * 0' # Weekly on Sunday
jobs:
mutation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Run mutation tests
run: npm run test:mutation
- name: Upload report
if: always()
uses: actions/upload-artifact@v3
with:
name: mutation-report
path: reports/mutation/
- name: Comment on PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
const report = JSON.parse(fs.readFileSync('reports/mutation/mutation.json'));
const score = report.mutationScore;
const emoji = score >= 80 ? '✅' : score >= 60 ? '⚠️' : '❌';
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## ${emoji} Mutation Testing Results\n\n**Mutation Score: ${score}%**\n\n- Killed: ${report.killed}\n- Survived: ${report.survived}\n- Timeout: ${report.timeout}`
});Optimization Tips
1. Incremental Testing
// stryker.config.mjs
export default {
// Only test changed files
mutate: [
'src/**/*.ts',
'!src/**/*.test.ts',
],
// Use incremental mode
incremental: true,
incrementalFile: '.stryker-tmp/incremental.json',
};2. Ignore Test Files
mutate: [
'src/**/*.ts',
'!src/**/*.test.ts', // Don't mutate tests
'!src/**/*.spec.ts', // Don't mutate specs
'!src/test-utils/**', // Don't mutate test utilities
]3. Parallel Execution
concurrency: 4, // Run 4 mutation tests in parallelBest Practices
- Start Small: Run on critical modules first
- Set Reasonable Thresholds: 70-80% is good, 100% is unrealistic
- Run Regularly: Weekly or before major releases
- Focus on Survivors: Fix tests for survived mutants
- Don't Over-Optimize: Some mutants are equivalent (okay to survive)
- Document Exceptions: Explain why certain code isn't mutated
Common Pitfalls
❌ Mutating everything: Too slow, too many false positives
✅ Focus on critical business logic
❌ Aiming for 100%: Unrealistic and time-consuming
✅ Target 75-85% mutation score
❌ Ignoring timeouts: They indicate problems
✅ Investigate and fix timeout causes
❌ Not excluding test files: Wastes time
✅ Only mutate production code
Mutation Score Guidelines
| Score | Quality | Action |
|---|---|---|
| < 40% | Poor | Major test improvements needed |
| 40-60% | Fair | Add more assertions |
| 60-75% | Good | Keep improving |
| 75-85% | Excellent | Maintain this level |
| > 85% | Outstanding | Diminishing returns |
When to Use Mutation Testing
✅ Use mutation testing for:
- Critical business logic
- Complex algorithms
- Security-sensitive code
- Financial calculations
- Code with high test coverage but uncertain quality
❌ Skip mutation testing for:
- UI components (visual regression better)
- Simple CRUD operations
- Configuration files
- Third-party integrations
- Prototype/MVP code
Mutation testing is the ultimate test of your tests—if your tests can't catch intentional bugs, they won't catch real ones either.