fp-audit

Analyze the gap between E2E specifications and existing test implementations

Safety Notice

This listing is imported from skills.sh public index metadata. Review upstream SKILL.md and repository scripts before running.

Copy this and send it to your AI assistant to learn

Install skill "fp-audit" with this command: npx skills add endorhq/flightplanner/endorhq-flightplanner-fp-audit

Audit E2E Test Coverage

Analyze the gap between E2E specifications and existing test implementations. This is a read-only command — it reports findings but does not modify any files.

Additional instructions from the user: "$ARGUMENTS". Ignore if empty.

Phase 1: Discover

  1. Find all E2E_TESTS.md files by searching recursively from the project root.
  2. Read the root-level spec first (docs/E2E_TESTS.md or E2E_TESTS.md) to understand project-wide testing constraints.
  3. Read each package-level E2E_TESTS.md and extract:
    • All suites (H2 headings)
    • All features within each suite (H4 headings)
    • Category metadata (<!-- category: ... -->)
    • Skip metadata (<!-- skip: ... -->)
  4. Find all existing E2E test files:
    • Files matching **/*.e2e.test.*
    • Files inside **/e2e/**/*.test.*
  5. Read existing test files and extract:
    • Suite/group blocks — the framework's grouping construct (e.g., describe() in vitest/jest, mod tests in Rust, class in pytest) — map to suites
    • Individual test cases — the framework's test declaration (e.g., it() / test() in vitest/jest, #[test] in Rust, def test_ in pytest) — map to features
    • Skip/conditional-skip markers — the framework's mechanism for skipping tests (e.g., skip / skipIf in vitest/jest, #[ignore] in Rust, @pytest.mark.skip in pytest)

Phase 2: Map

Build a coverage matrix:

Spec SuiteSpec FeatureCategoryTest FileTest CaseStatus
Task CreationCreate basic taskcoretask.e2e.test.ts"Create basic task"Covered
Task CreationCreate with long nameedgeMissing
PushPush to remotecorepush.e2e.test.ts"push to remote"Covered

For each spec feature, determine:

  • Covered: A corresponding test case exists and its assertions match the spec
  • Missing: No corresponding test case
  • Stale: Test exists but assertions don't match current spec
  • Skipped: Test exists with a skip marker (expected for <!-- skip: ... --> features)

Phase 3: Report

Present findings in this format:

E2E Test Coverage Audit
========================

Specs found: N files
Test files found: N files

Overall coverage: X/Y features (Z%)

By category:
  core:        X/Y (Z%)
  edge:        X/Y (Z%)
  error:       X/Y (Z%)
  side-effect: X/Y (Z%)
  idempotency: X/Y (Z%)

Gaps by suite:
  ## Task Creation (3/5 features covered)
    MISSING  [edge]        Create with very long name
    MISSING  [idempotency] Re-create after deletion

  ## Push (4/4 features covered)
    (fully covered)

Stale tests:
  push.e2e.test.ts: "push with pending changes"
    Spec says: Exits with non-zero code
    Test asserts: Exits with code 0

Structural issues:
  - task.e2e.test.ts: Tests share mutable state at suite level
  - init.e2e.test.ts: Missing afterEach cleanup for env.HOME

Do NOT suggest fixes or modifications. This command is purely diagnostic.

Source Transparency

This detail page is rendered from real SKILL.md content. Trust labels are metadata-based hints, not a safety guarantee.

Related Skills

Related by shared tags or category signals.

General

fp-update-spec

No summary provided by upstream source.

Repository SourceNeeds Review
General

fp-add

No summary provided by upstream source.

Repository SourceNeeds Review
General

fp-generate

No summary provided by upstream source.

Repository SourceNeeds Review
General

fp-fix

No summary provided by upstream source.

Repository SourceNeeds Review