AI-Powered Mutation Testing

Target: $ARGUMENTS

If no argument provided, default to files changed in the current branch (via git diff ).

This skill implements the mutation testing workflow: generate targeted mutants, filter unproductive ones, run tests to find survivors, and harden the test suite by strengthening or creating tests that kill surviving mutants.

Step 1: Gather Context

1a. Identify Target Files

Determine what to mutate based on $ARGUMENTS :

• File path — Mutate the specified source file

• Directory — Mutate source files in the directory (exclude test files)

• No argument — Use git diff to find changed source files: git diff --name-only $(git merge-base HEAD main)...HEAD -- '.ts' '.tsx' ':!.spec.' ':!.test.'

If no target files found, notify the user and stop.

1b. Gather Supporting Context

For each target source file, collect:

• Source code — Read the full file contents

• Existing tests — Find corresponding test files (*.spec.ts , *.test.ts ) via naming convention or import analysis

• Test coverage — Run tests with coverage for the target file to identify uncovered lines: bun run test -- --coverage --collectCoverageFrom='<target-file>' --silent 2>&1 | tail -20

• Recent git history — Check for recent defects or frequent changes: git log --oneline -10 -- <target-file>

• Risk factors — Assess which risk factors apply to each file:

Risk Factor Indicators

Data security / compliance Handles PII, auth, authorization, encryption, tokens

Integrations External API calls, HTTP clients, webhooks, message queues

Code vs data model Database queries, ORM entities, schema validation

Historic defects Frequent bug-fix commits, complex conditional logic

Change impact Exported utility functions, shared modules, base classes

Performance Loops over large datasets, recursive calls, caching logic

1c. Determine Test Runner

Detect the project's test framework from package.json scripts and devDependencies:

• Jest: bun run test

• Vitest: bun run test

• Other: adapt accordingly

Step 2: Generate Mutants

2a. Create Experimental Branch

git stash --include-untracked || true git checkout -b mutation-testing/$(date +%Y-%m-%d-%H%M%S) git stash pop || true

2b. Generate Risk-Factor-Guided Mutants

For each target source file, generate 3-5 mutants that target the identified risk factors. Apply these mutation operators:

Operator Type Examples

Decision mutations Change > to >= , && to || , === to !==

Value mutations Change constants, swap string literals, alter numeric values

Statement mutations Remove guard clauses, delete error handling, skip validation

Integration mutations Remove timeout handling, skip error code checks, bypass retry logic

Security mutations Remove auth checks, bypass input validation, expose sensitive data in logs

For each mutant, produce:

• Mutant ID — Sequential identifier (M001, M002, etc.)

• File and line — Exact location of the change

• Mutation description — What was changed and why

• Risk factor — Which risk factor this targets

• The code change — A minimal, atomic edit to introduce the defect

Mutant quality criteria — Each mutant must be:

• Non-trivial — Not just whitespace, comments, or formatting

• Buildable — The project must still compile/typecheck

• Realistic — Simulates a plausible programming error

• Targeted — Addresses a specific risk factor

• Atomic — Exactly one logical change per mutant

2c. Apply and Validate Each Mutant

For each generated mutant, one at a time:

• Apply the mutation — Edit the source file to introduce the defect

• Build check — Run bun run typecheck to verify the project still compiles

• If build fails: discard the mutant, revert the change, log failure reason, continue to next

• Commit the mutant — git commit -am "mutant: M00X - <description>"

Step 3: Filter Mutants

3a. Run Tests Against Each Mutant

Process mutants in reverse order (LIFO — latest commit first):

• Run relevant tests against the mutant: bun run test -- <test-file-path> --silent 2>&1

• Evaluate result:

• Test fails (mutant killed) → Revert the mutant commit. Log as killed. Proceed to next mutant.

• Test passes (mutant survived) → The mutant reveals a test gap. Keep it for Step 4.

3b. Equivalence Detection for Surviving Mutants

For each surviving mutant, verify it is not semantically equivalent to the original:

• AST-level comparison — Compare the diff. If the change is purely syntactic (reordering independent statements, renaming to equivalent aliases), discard it.

• Behavioral analysis — Reason about whether any input could distinguish the mutant from the original:

• If no distinguishing input exists → Discard as equivalent

• If a distinguishing input exists → Keep as a unique actionable mutant

• If uncertain → Keep and flag for human review

3c. Record Mutant Metadata

For each mutant, maintain a record:

{ "mutant_id": "M001", "file": "<source-file>", "line": 42, "risk_factor": "Data security / compliance", "description": "Removed authentication check before data access", "status": "survived|killed|equivalent|discarded", "commit_hash": "<hash>", "tests_run": ["test-file.spec.ts"], "equivalence_result": "not_equivalent|equivalent|uncertain" }

Print a summary table after filtering:

Step 4: Harden Test Suite

For each surviving, non-equivalent mutant:

4a. Determine Test Strategy

• Existing test covers the mutated region → Strengthen the existing test

• No test covers the mutated region → Generate a new test

4b. Strengthen Existing Test or Generate New Test

When strengthening an existing test, modify the test to:

• Add assertions that detect the behavioral difference introduced by the mutant

• Add or modify test inputs that expose the defect in the mutant code

• Preserve the test method name, signature, and overall structure

• Avoid adding trivial or unrelated checks

When generating a new test, create a test that:

• Specifically exercises the behavioral difference between original and mutant code

• Uses strong, precise assertions that detect the exact defect

• Focuses on the identified risk factor

• Includes realistic test data that exposes the mutant's flawed behavior

• Follows existing test naming conventions and structure from the codebase

4c. Validate Each Test (3-attempt limit)

For each improved or new test, validate with up to 3 attempts:

• Revert the mutant — Switch to original code

• Build check — Ensure the test compiles

• Run on original code — Test must PASS on unmodified code

• Re-apply the mutant — Switch back to mutated code

• Run on mutant code — Test must FAIL on the mutant

If validation fails, refine the test (up to 3 total attempts). If still failing after 3 attempts, flag for manual review.

4d. Apply Decision Matrix

Classify each test result:

Original Code Mutant Code Signal Action

Builds + Passes Builds + Fails Strong detection Keep — alert developer

Builds + Passes Builds + Passes Weak assertion Refine — strengthen assertion

Builds + Fails Builds + Passes Bad test Discard — generate new test

Doesn't build Any Broken test Discard — generate new test

Builds + Passes Doesn't build Untestable mutant Discard — notify developer

Builds + Fails Builds + Fails Ambiguous signal Discard — generate new test

4e. Finalize Tests

For each validated test:

• Add an inline comment above the test referencing the mutant: // Test hardened to kill mutant M001 (Risk Factor: Data security / compliance)

• Commit the test improvement to the experimental branch

Step 5: Cleanup and Report

5a. Revert All Mutants

Remove all mutant commits from the experimental branch, keeping only test improvements:

Revert mutant commits (identified by "mutant:" prefix in commit message)

git log --oneline | grep "^.* mutant:" | awk '{print $1}' | while read hash; do git revert --no-commit "$hash" done git commit -m "chore: revert all mutants after test hardening"

5b. Validate Clean State

• Run full test suite on the clean code with test improvements: bun run test 2>&1

• All tests must pass. If any fail, investigate and fix.

5c. Report Results

Print a comprehensive mutation testing report:

Mutation Testing Report

Target: <files tested> Date: <timestamp> Branch: <experimental branch name>

Summary

• Total mutants generated: X
• Killed by existing tests: X
• Survived (test gaps found): X
• Equivalent (discarded): X
• Build failures (discarded): X
• Mutation Score: X% (killed / (total - equivalent))

Test Improvements

• Tests strengthened: X
• New tests generated: X
• Tests validated successfully: X
• Tests requiring manual review: X

Surviving Mutants (Unresolved)

Oracle Gap

• Code coverage: X%
• Mutation score: X%
• Oracle gap: X% (coverage - mutation score)

Risk Factor Coverage

5d. Present Options

Ask the user:

• Merge test improvements — Cherry-pick test commits to the original branch, delete experimental branch

• Keep experimental branch — For further review before merging

• Discard everything — Delete the experimental branch, no changes kept

If the user chooses to merge:

git checkout <original-branch> git cherry-pick <test-improvement-commits> git branch -D <experimental-branch>

Never

• Modify test files to make them pass on mutants (defeats the purpose)

• Generate mutants in test files (only mutate source code)

• Skip the build check after generating a mutant

• Commit mutants to protected branches (dev, staging, main)

• Leave mutant code in the codebase after completion

• Generate more than 5 mutants per file (diminishing returns)

• Skip equivalence detection for surviving mutants

• Mark a test as valid without running it on both original and mutant code

• Use --no-verify with any git command