Test Suite Validation Skill

This skill helps you efficiently validate code changes by running the appropriate subset of the test suite. It uses scripts/run-tests to intelligently discover affected tests and run only what's necessary for validation.

When to Use This Skill

Use this skill when you have:

• Made changes to source code files and want to validate they work

• Fixed a bug and want to verify the fix

• Added a feature and need test coverage

• Modified test infrastructure or configuration

• Want to verify changes don't break existing functionality

Key Principles

• Always use the run-tests skill when testing code changes - it's optimized for intelligent suite discovery

• Never run pytest directly - bypasses the project's test infrastructure (use scripts/run-tests or riot via scripts/ddtest )

• Minimal venvs for iteration - run 1-2 venvs initially, expand only if needed

• Use --dry-run first - see what would run before executing

• Follow official docs - docs/contributing-testing.rst is the source of truth for testing procedures

How This Skill Works

Step 1: Identify Changed Files

First, determine which files were modified:

• If you have pending edits, I'll identify the changed files from the current session

• I'll look at git status to find staged, unstaged, and untracked changes

• You can also specify files explicitly if working on specific changes

Step 2: Discover Available Test Suites

I'll use the scripts/run-tests script to discover what test suites match your changes:

scripts/run-tests --list <edited-files>

This outputs JSON showing:

• Available test suites that match your changed files

• All venvs (Python versions + package combinations) available for each suite

• Their hashes, Python versions, and package versions

Step 3: Intelligently Select Venvs

Rather than running ALL available venvs (which could take hours), I'll select the minimal set needed to validate your changes:

For Core/Tracing Changes (Broad Impact)

When you modify files like:

• ddtrace/internal/core/* , ddtrace/_trace/* , ddtrace/trace/*

• ddtrace/_monkey.py , ddtrace/settings/*

• ddtrace/constants.py

Strategy: Run core tracer + internal tests with 1 venv each

• Example: tracer suite with latest Python + internal suite with latest Python

• This validates broad-reaching changes without excessive overhead

• Skip integration suites unless the change directly affects integration code

For Integration/Contrib Changes (Targeted Impact)

When you modify files like:

• ddtrace/contrib/flask/* , ddtrace/contrib/django/* , etc.

• ddtrace/contrib/*/patch.py or integration-specific code

Strategy: Run ONLY the affected integration suite with 1-2 venvs

• Example: Flask changes → run contrib::flask suite with latest Python

• If change involves multiple versions (e.g., Django 3.x and 4.x), pick 1 venv per major version

• Skip unrelated integrations

For Test-Only Changes

When you modify tests/ files (but not test infrastructure):

• Run only the specific test files/functions modified

• Use pytest args: -- -k test_name or direct test file paths

For Test Infrastructure Changes

When you modify:

• tests/conftest.py , tests/suitespec.yml , scripts/run-tests , riotfile.py

Strategy: Run a quick smoke test suite

• Example: internal suite with 1 venv as a sanity check

• Or run small existing test suites to verify harness changes

Step 4: Execute Selected Venvs

I'll run the selected venvs using:

scripts/run-tests --venv <hash1> --venv <hash2> ...

This will:

• Start required Docker services (redis, postgres, etc.)

• Run tests in the specified venvs sequentially

• Stop services after completion

• Show real-time output and status

Step 5: Handle Results

If tests pass: ✅ Your changes are validated!

If tests fail: 🔴 I'll:

• Show you the failure details

• Identify which venv failed

• Ask clarifying questions to understand the issue

• Offer to run specific failing tests with more verbosity

• Help iterate on fixes and re-run

For re-running specific tests:

scripts/run-tests --venv <hash> -- -vv -k test_name

When Tests Fail

When you encounter test failures, follow this systematic approach:

• Review the failure details carefully - Don't just skim the error, understand what's actually failing

• Understand what's failing - Don't blindly re-run; analyze the root cause

• Make code changes - Fix the underlying issue

• Re-run with more verbosity if needed - Use -vv or -vvv for detailed output

• Iterate until tests pass - Repeat the process with each fix

Venv Selection Strategy in Detail

Understanding Venv Hashes

From scripts/run-tests --list , you'll see output like:

{ "suites": [ { "name": "tracer", "venvs": [ { "hash": "abc123", "python_version": "3.8", "packages": "..." }, { "hash": "def456", "python_version": "3.11", "packages": "..." } ] } ] }

Selection Rules

Latest Python version is your default choice

• Unless your change specifically targets an older Python version

• Example: if fixing Python 3.8 compatibility, also test 3.8

One venv per suite is usually enough for iteration

• Only run multiple venvs per suite if:

• Change impacts multiple Python versions differently

• Testing package compatibility variations (e.g., Django 3.x vs 4.x)

• Initial validation passed and you want broader coverage

Minimize total venvs

• 1-2 venvs total for small targeted changes

• 3-4 venvs maximum for broader changes

• Never run 10+ venvs for initial validation (save that for CI)

Consider test runtime

• Each venv can take 5-30 minutes depending on suite

• With 2 venvs you're looking at 10-60 minutes for iteration

• With 5 venvs you're looking at 25-150 minutes

• Scale appropriately for your patience and deadline

Using --venv Directly

When you have a specific venv hash you want to run, you can use it directly without specifying file paths:

scripts/run-tests --venv e06abee

The --venv flag automatically searches all available venvs across all suites, so it works regardless of what files you have locally changed. This is useful when:

• You know exactly which venv you want to test

• You have unrelated local changes that would otherwise limit suite matching

• You want to quickly re-run a specific venv without file path arguments

Examples

Example 1: Fixing a Flask Integration Bug

Changed file: ddtrace/contrib/internal/flask/patch.py

scripts/run-tests --list ddtrace/contrib/internal/flask/patch.py

Output shows: contrib::flask suite available

Select output (latest Python):

Suite: contrib::flask

Venv: hash=e06abee, Python 3.13, flask

Run with --venv directly (searches all venvs automatically)

scripts/run-tests --venv e06abee

Runs just Flask integration tests

Example 2: Fixing a Core Tracing Issue

Changed file: ddtrace/_trace/tracer.py

scripts/run-tests --list ddtrace/_trace/tracer.py

Output shows: tracer suite, internal suite available

Select strategy:

- tracer: latest Python (e.g., abc123)

- internal: latest Python (e.g., def456)

Run with --venv directly (searches all venvs automatically)

scripts/run-tests --venv abc123 --venv def456

Validates core tracer and internal components

Example 3: Fixing a Test-Specific Bug

Changed file: tests/contrib/flask/test_views.py

scripts/run-tests --list tests/contrib/flask/test_views.py

Output shows: contrib::flask suite

Run just the specific test:

scripts/run-tests --venv flask_py311 -- -vv tests/contrib/flask/test_views.py

Example 4: Iterating on a Failing Test

First run shows one test failing:

scripts/run-tests --venv flask_py311 -- -vv -k test_view_called_twice

Focused on the specific failing test with verbose output

Best Practices

DO ✅

• Start small: Run 1 venv first, expand only if needed

• Be specific: Use pytest -k filter when re-running failures

• Check git: Verify you're testing the right files with git status

• Read errors: Take time to understand test failures before re-running

• Ask for help: When unclear what tests to run, ask me to analyze the changes

DON'T ❌

• Run all venvs initially: That's what CI is for

• Skip the minimal set guidance: It's designed to save you time

• Ignore service requirements: Some suites need Docker services up

• Run tests without changes saved: Make sure edits are saved first

• Iterate blindly: Understand what's failing before re-running

Additional Testing Resources

For comprehensive testing guidance, refer to the contributing documentation:

docs/contributing-testing.rst - Detailed testing guidelines

• What kind of tests to write (unit tests, integration tests, e2e tests)

• When to write tests (feature development, bug fixes)

• Where to put tests in the repository

• Prerequisites (Docker, uv)

• Complete scripts/run-tests usage examples

• Riot environment management details

• Running specific test files and functions

• Test debugging strategies

docs/contributing.rst - PR and testing requirements

• All changes need tests or documented testing strategy

• How tests fit into the PR review process

• Testing expectations for different types of changes

docs/contributing-design.rst - Test architecture context

• How products, integrations, and core interact

• Where different types of tests should live

• Testing patterns for each library component

When to reference these docs:

• First time writing tests for this project → Read contributing-testing.rst

• Understanding test requirements for PRs → Read contributing.rst

• Need context on test architecture → Read contributing-design.rst

Troubleshooting

Docker services won't start

Manually check/stop services:

docker compose ps docker compose down

Can't find matching suites

• Verify the file path is correct

• Check tests/suitespec.yml to understand suite patterns

• Your file might not be covered by any suite pattern yet

Test takes too long

• You may have selected too many venvs

• Try running with just 1 venv

• Use pytest -k to run subset of tests

Technical Details

Architecture

The scripts/run-tests system:

• Maps source files to test suites using patterns in tests/suitespec.yml

• Uses riot to manage multiple Python/package combinations as venvs

• Each venv is a self-contained environment

• Docker services are managed per suite lifecycle

• Tests can pass optional pytest arguments with --

Supported Suite Types

Primary suites for validation:

• tracer : Core tracing functionality tests

• internal : Internal component tests

• contrib::* : Integration with specific libraries (flask, django, etc.)

• integration_* : Cross-library integration scenarios

• Specialized: telemetry , profiling , appsec , llmobs , etc.

Environment Variables

Some suites require environment setup:

• DD_TRACE_AGENT_URL : For snapshot-based tests

• Service-specific variables for Docker containers

• These are handled automatically by the script