Front-end Engineering Lab
PatternsTesting Strategies

Mutation Testing

Measure test quality with Stryker mutation testing

Mutation Testing

Mutation testing measures the quality of your tests by introducing small bugs (mutations) into your code and checking if your tests catch them. High code coverage doesn't mean good tests—mutation testing tells you if your tests actually work.

The Problem with Code Coverage

// This has 100% code coverage...
function add(a: number, b: number): number {
  return a + b;
}

test('add function works', () => {
  add(2, 3); // Runs the function, but...
});

// But this test doesn't verify anything!
// It would still pass if add() returned 0, null, or even a - b

Code CoverageTest Quality

Mutation testing solves this by:

  1. Creating mutants (changing + to -, > to <, etc.)
  2. Running your tests against mutants
  3. Tests should kill mutants (fail when code is wrong)

What is a Mutant?

// Original code
function isAdult(age: number): boolean {
  return age >= 18; // ← This is the original
}

// Mutant 1: Change >= to >
function isAdult(age: number): boolean {
  return age > 18; // Age 18 is now false!
}

// Mutant 2: Change >= to <=
function isAdult(age: number): boolean {
  return age <= 18; // Inverted logic!
}

// Mutant 3: Change 18 to 19
function isAdult(age: number): boolean {
  return age >= 19; // Wrong threshold!
}

// Good tests should kill all these mutants

Stryker Setup

Installation

npm install --save-dev @stryker-mutator/core
npx stryker init

Configuration

// stryker.config.mjs
/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
const config = {
  packageManager: 'npm',
  reporters: ['html', 'clear-text', 'progress'],
  testRunner: 'jest',
  coverageAnalysis: 'perTest',
  
  // Which files to mutate
  mutate: [
    'src/**/*.ts',
    'src/**/*.tsx',
    '!src/**/*.test.ts',
    '!src/**/*.test.tsx',
    '!src/**/*.spec.ts',
  ],
  
  // Thresholds
  thresholds: {
    high: 80,
    low: 60,
    break: 75, // Fail if mutation score < 75%
  },
  
  // Ignore specific files
  mutator: {
    excludedMutations: [
      'StringLiteral', // Don't mutate string literals
      'ObjectLiteral', // Don't mutate object literals
    ],
  },
  
  // Timeout
  timeoutMS: 60000,
  timeoutFactor: 1.5,
};

export default config;

Running Stryker

// package.json
{
  "scripts": {
    "test:mutation": "stryker run"
  }
}
npm run test:mutation

Understanding Results

Mutation Score

Mutation Score = (Killed Mutants / Total Mutants) × 100%

Example Output:

All tests: 18 passed
Mutants: 50 tested
  - Killed: 40 (80%)
  - Survived: 8 (16%)
  - Timeout: 2 (4%)

Mutation Score: 80%

Mutant Status

Killed ✅: Test failed when mutant was introduced (good!)
Survived ❌: All tests passed with mutant (bad!)
Timeout ⏱️: Tests took too long (infinite loop?)
No Coverage ⚠️: No tests run this code
Runtime Error 💥: Mutant broke the code (neutral)

Example: Improving Tests

Original Code

// utils/validation.ts
export function isValidEmail(email: string): boolean {
  return email.includes('@') && email.length > 5;
}

Weak Test (Survivors)

// ❌ BAD: Test with poor assertions
test('validates email', () => {
  isValidEmail('test@example.com');
  // Doesn't check return value!
});

// Stryker creates mutants:
// - Mutant 1: email.includes('@') → !email.includes('@')
// - Mutant 2: email.length > 5 → email.length < 5
// - Mutant 3: && → ||

// All these mutants SURVIVE because we don't assert!

Strong Test (Kills Mutants)

// ✅ GOOD: Test with proper assertions
test('validates email correctly', () => {
  // Test true cases
  expect(isValidEmail('test@example.com')).toBe(true);
  expect(isValidEmail('a@b.co')).toBe(true);
  
  // Test false cases (kills boundary mutants)
  expect(isValidEmail('test')).toBe(false); // No @
  expect(isValidEmail('t@e')).toBe(false);  // Too short
  expect(isValidEmail('@test')).toBe(false); // Just @
  expect(isValidEmail('')).toBe(false);      // Empty
});

// Now all mutants are KILLED!

Common Mutators

Arithmetic Mutator

// Original → Mutant
a + b  →  a - b
a - b  →  a + b
a * b  →  a / b
a / b  →  a * b
a % b  →  a * b

Comparison Mutator

a > b   →  a >= b, a < b, a <= b
a >= b  →  a > b, a < b, a <= b
a < b   →  a <= b, a > b, a >= b
a <= b  →  a < b, a > b, a >= b
a === b →  a !== b
a !== b →  a === b

Logical Mutator

a && b  →  a || b, true, false
a || b  →  a && b, true, false
!a      →  a

Conditional Mutator

if (condition) { } → if (true) { }, if (false) { }
while (condition) { } → while (false) { }

String/Number Mutator

"hello"""
1230, 1
truefalse

Real-World Example

Code Under Test

// services/cart.ts
export class ShoppingCart {
  private items: CartItem[] = [];

  addItem(product: Product, quantity: number): void {
    if (quantity <= 0) {
      throw new Error('Quantity must be positive');
    }

    const existingItem = this.items.find(item => item.product.id === product.id);

    if (existingItem) {
      existingItem.quantity += quantity;
    } else {
      this.items.push({ product, quantity });
    }
  }

  getTotal(): number {
    return this.items.reduce((sum, item) => {
      return sum + (item.product.price * item.quantity);
    }, 0);
  }

  isEmpty(): boolean {
    return this.items.length === 0;
  }
}

Comprehensive Tests

// services/cart.test.ts
describe('ShoppingCart', () => {
  let cart: ShoppingCart;

  beforeEach(() => {
    cart = new ShoppingCart();
  });

  describe('addItem', () => {
    it('adds new item to cart', () => {
      const product = { id: 1, name: 'Product 1', price: 10 };
      
      cart.addItem(product, 2);
      
      expect(cart.getTotal()).toBe(20);
    });

    it('increases quantity if item already exists', () => {
      const product = { id: 1, name: 'Product 1', price: 10 };
      
      cart.addItem(product, 2);
      cart.addItem(product, 3);
      
      expect(cart.getTotal()).toBe(50); // 5 items × $10
    });

    it('throws error for zero quantity', () => {
      const product = { id: 1, name: 'Product 1', price: 10 };
      
      expect(() => cart.addItem(product, 0)).toThrow('Quantity must be positive');
    });

    it('throws error for negative quantity', () => {
      const product = { id: 1, name: 'Product 1', price: 10 };
      
      expect(() => cart.addItem(product, -1)).toThrow('Quantity must be positive');
    });
  });

  describe('getTotal', () => {
    it('returns 0 for empty cart', () => {
      expect(cart.getTotal()).toBe(0);
    });

    it('calculates total correctly', () => {
      cart.addItem({ id: 1, name: 'P1', price: 10 }, 2);
      cart.addItem({ id: 2, name: 'P2', price: 5 }, 3);
      
      expect(cart.getTotal()).toBe(35); // (10×2) + (5×3)
    });
  });

  describe('isEmpty', () => {
    it('returns true for new cart', () => {
      expect(cart.isEmpty()).toBe(true);
    });

    it('returns false when items added', () => {
      cart.addItem({ id: 1, name: 'P1', price: 10 }, 1);
      
      expect(cart.isEmpty()).toBe(false);
    });
  });
});

// This test suite should have high mutation score (90%+)

CI/CD Integration

# .github/workflows/mutation-testing.yml
name: Mutation Testing

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 0' # Weekly on Sunday

jobs:
  mutation:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run mutation tests
        run: npm run test:mutation
      
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: mutation-report
          path: reports/mutation/
      
      - name: Comment on PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const report = JSON.parse(fs.readFileSync('reports/mutation/mutation.json'));
            
            const score = report.mutationScore;
            const emoji = score >= 80 ? '✅' : score >= 60 ? '⚠️' : '❌';
            
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## ${emoji} Mutation Testing Results\n\n**Mutation Score: ${score}%**\n\n- Killed: ${report.killed}\n- Survived: ${report.survived}\n- Timeout: ${report.timeout}`
            });

Optimization Tips

1. Incremental Testing

// stryker.config.mjs
export default {
  // Only test changed files
  mutate: [
    'src/**/*.ts',
    '!src/**/*.test.ts',
  ],
  
  // Use incremental mode
  incremental: true,
  incrementalFile: '.stryker-tmp/incremental.json',
};

2. Ignore Test Files

mutate: [
  'src/**/*.ts',
  '!src/**/*.test.ts',     // Don't mutate tests
  '!src/**/*.spec.ts',     // Don't mutate specs
  '!src/test-utils/**',    // Don't mutate test utilities
]

3. Parallel Execution

concurrency: 4, // Run 4 mutation tests in parallel

Best Practices

  1. Start Small: Run on critical modules first
  2. Set Reasonable Thresholds: 70-80% is good, 100% is unrealistic
  3. Run Regularly: Weekly or before major releases
  4. Focus on Survivors: Fix tests for survived mutants
  5. Don't Over-Optimize: Some mutants are equivalent (okay to survive)
  6. Document Exceptions: Explain why certain code isn't mutated

Common Pitfalls

Mutating everything: Too slow, too many false positives
Focus on critical business logic

Aiming for 100%: Unrealistic and time-consuming
Target 75-85% mutation score

Ignoring timeouts: They indicate problems
Investigate and fix timeout causes

Not excluding test files: Wastes time
Only mutate production code

Mutation Score Guidelines

ScoreQualityAction
< 40%PoorMajor test improvements needed
40-60%FairAdd more assertions
60-75%GoodKeep improving
75-85%ExcellentMaintain this level
> 85%OutstandingDiminishing returns

When to Use Mutation Testing

Use mutation testing for:

  • Critical business logic
  • Complex algorithms
  • Security-sensitive code
  • Financial calculations
  • Code with high test coverage but uncertain quality

Skip mutation testing for:

  • UI components (visual regression better)
  • Simple CRUD operations
  • Configuration files
  • Third-party integrations
  • Prototype/MVP code

Mutation testing is the ultimate test of your tests—if your tests can't catch intentional bugs, they won't catch real ones either.

On this page