Test Quality Analysis

Expert knowledge for analyzing and improving test quality - detecting test smells, overmocking, insufficient coverage, and testing anti-patterns.

Core Dimensions

• Correctness: Tests verify the right behavior

• Reliability: Tests are deterministic, not flaky

• Maintainability: Tests are easy to understand

• Performance: Tests run quickly

• Coverage: Tests cover critical code paths

• Isolation: Tests don't depend on external state

Test Smells

Overmocking

Problem: Mocking too many dependencies makes tests fragile.

// ❌ BAD: Overmocked test('calculate total', () => { const mockAdd = vi.fn(() => 10) const mockMultiply = vi.fn(() => 20) // Testing implementation, not behavior })

// ✅ GOOD: Mock only external dependencies test('calculate order total', () => { const mockPricingAPI = vi.fn(() => ({ tax: 0.1 })) const total = calculateTotal(order, mockPricingAPI) expect(total).toBe(38) })

Detection: More than 3-4 mocks, mocking pure functions, complex mock setup.

Fix: Mock only I/O boundaries (APIs, databases, filesystem).

Fragile Tests

Problem: Tests break with unrelated code changes.

// ❌ BAD: Tests implementation details await page.locator('.form-container > div:nth-child(2) > button').click()

// ✅ GOOD: Semantic selector await page.getByRole('button', { name: 'Submit' }).click()

Flaky Tests

Problem: Tests pass or fail non-deterministically.

// ❌ BAD: Race condition test('loads data', async () => { fetchData() await new Promise(resolve => setTimeout(resolve, 1000)) expect(data).toBeDefined() })

// ✅ GOOD: Proper async handling test('loads data', async () => { const data = await fetchData() expect(data).toBeDefined() })

Poor Assertions

// ❌ BAD: Weak assertion test('returns users', async () => { const users = await getUsers() expect(users).toBeDefined() // Too vague! })

// ✅ GOOD: Strong, specific assertions test('creates user with correct attributes', async () => { const user = await createUser({ name: 'John' }) expect(user).toMatchObject({ id: expect.any(Number), name: 'John', }) })

Analysis Tools

Vitest coverage (prefer bun)

bun test --coverage open coverage/index.html

Check thresholds

bun test --coverage --coverage.thresholds.lines=80

pytest-cov (Python)

uv run pytest --cov --cov-report=html open htmlcov/index.html

Best Practices Checklist

Unit Test Quality (FIRST)

• Fast: Tests run in milliseconds

• Isolated: No dependencies between tests

• Repeatable: Same results every time

• Self-validating: Clear pass/fail

• Timely: Written alongside code

Mock Guidelines

• Mock only external dependencies

• Don't mock business logic or pure functions

• Use real implementations when possible

• Limit to 3-4 mocks per test maximum

Coverage Goals

• 80%+ line coverage for business logic

• 100% for critical paths (auth, payment)

• All error paths tested

• Boundary conditions tested

Test Structure (AAA Pattern)

test('user registration', async () => { // Arrange const userData = { email: 'user@example.com' }

// Act const user = await registerUser(userData)

// Assert expect(user.email).toBe('user@example.com') })

Code Review Checklist

• Tests verify behavior, not implementation

• Assertions are specific and meaningful

• No flaky tests (timing, ordering issues)

• Proper async/await usage

• Test names clearly describe behavior

• Minimal code duplication

• Critical paths have tests

• Both happy path and error cases covered

Common Anti-Patterns

Testing Implementation Details

// ❌ BAD const spy = vi.spyOn(Math, 'sqrt') calculateDistance() expect(spy).toHaveBeenCalled() // Testing how, not what

// ✅ GOOD const distance = calculateDistance({ x: 0, y: 0 }, { x: 3, y: 4 }) expect(distance).toBe(5) // Testing output

Mocking Too Much

// ❌ BAD const mockAdd = vi.fn((a, b) => a + b)

// ✅ GOOD: Use real implementations import { add } from './utils' // Only mock external services const mockPaymentGateway = vi.fn()

See Also

• vitest-testing

• TypeScript/JavaScript testing

• playwright-testing

• E2E testing

• mutation-testing

• Validate test effectiveness