Skill Optimizer (Autoresearch Loop + Anthropic Structure Audit)

Two-phase improvement system: (1) structural audit against Anthropic best practices, (2) iterative output quality loop.

Phase 1: Structure Audit (run first, always)

Before optimizing output quality, audit the skill's architecture. Score against these 5 structural checks:

Structural Checklist:

1. Gotchas section — Does SKILL.md have a ## Gotchas section with at least one real failure case? (Highest-signal content per Anthropic)
2. Trigger-phrase description — Does the YAML description field say when to use the skill, not just what it does? Must include "Use when..." or equivalent trigger condition.
3. Progressive disclosure — Does the skill use the file system (references/, scripts/, assets/, config.json) instead of inline-dumping everything into SKILL.md?
4. Single focus — Does the skill fit cleanly into one type (Library Reference, Verification, Automation, Scaffolding, Runbook, etc.) without straddling multiple?
5. No railroading — Does the skill give Claude information + flexibility, rather than over-specifying how it must execute?

Score each: ✅ pass | ❌ fail | ⚠️ partial

For each failure: propose a concrete fix and apply if approved.

Quick wins to apply immediately:

• If no Gotchas section → add ## Gotchas\n- [Placeholder: add real failures here as they're discovered]
• If description is a summary → rewrite as trigger condition
• If all content is inline → propose a references/ folder structure

Phase 2: Output Quality Loop (autoresearch)

After structure audit, run the iterative improvement loop on the skill's actual outputs.

Setup

1. Which skill? — User specifies, or infer from context.
2. Test inputs — Get 2-3 representative inputs. If the user doesn't provide them:
   • Check the skill's own docs for example usage
   • Use recent real invocations from memory/session history
   • For extraction skills: use known-good URLs/files. For generation skills: use the skill's own example prompts.
3. Scoring checklist — Build 3-6 scoring items. Start from the examples below, then customize:
   • What's the #1 thing that makes this skill's output bad? (That's checklist item 1)
   • What would make a user say "that's exactly what I wanted"? (That's the positive framing)
   • Add 1-2 items from the "Universal structural quality" list below

Scoring Checklist Examples

See references/checklist-examples.md for starter checklists by skill type (cold outreach, content, research, extraction, process/meta-skills).

Scoring Modes

Binary mode (default for simple skills): Yes/no per checklist item. Pass rate = total yes / (items × runs).

Dimensional mode (use for complex skills or when binary plateaus): Score each dimension 0-10. Identify the weakest dimension (lowest average across runs). Target that dimension for revision — do NOT rewrite everything.

Use dimensional mode when:

• Binary scoring hits 100% but output still feels mediocre
• The skill has qualitative dimensions (tone, depth, relevance) that binary can't capture
• You want to improve from "good" to "excellent" rather than from "broken" to "working"

The Loop

Round N:
1. Run skill against each test input
2. Score each output (binary: 1 per yes | dimensional: 0-10 per dimension)
3. Calculate score:
   - Binary: pass rate = (total yes) / (items × runs)
   - Dimensional: avg score per dimension across runs
4. Identify the weakest item/dimension (most failures or lowest avg score)
5. Make ONE targeted change to SKILL.md addressing ONLY that weakness
6. Re-run and re-score
7. If new score > old score: KEEP. Else: REVERT.
8. Log: score before/after, change made, dimension targeted, kept/reverted

Stop when: binary ≥ 95% (3 consecutive rounds) OR dimensional weakest ≥ 8/10 (3 consecutive) OR 20 rounds reached.

Output Files

• skills/{skill-name}/SKILL-optimized.md — improved version (original untouched)
• skills/{skill-name}/optimization-changelog.md — full round log

Changelog Format

## Structural Audit
- Gotchas section: ❌ → Added placeholder
- Description: ❌ → Rewritten as trigger condition
- Progressive disclosure: ⚠️ → Noted, deferred

## Round 1 (binary mode)
- Score: 4/10 (40%)
- Weakest item: "Does it mention business name?"
- Change: Added rule "Always open with [Business Name],"
- New score: 7/10 (70%)
- Decision: KEPT

## Round 2 (dimensional mode)
- Scores: Accuracy 8/10 | Tone 5/10 | Brevity 9/10 | Relevance 7/10
- Weakest dimension: Tone (5/10)
- Change: Added "Match prospect's industry language, not generic sales speak"
- New scores: Accuracy 8/10 | Tone 7/10 | Brevity 9/10 | Relevance 7/10
- Decision: KEPT (Tone +2)

Optimizing Meta-Skills (Process Skills)

Some skills don't produce text — they drive a process (e.g., this skill itself, planning workflows, research pipelines). For these:

What to score: Score the experience of following the process, not a text artifact.

• Did the process produce a clear result?
• Were there moments of confusion where the instructions were ambiguous?
• Did any step feel unnecessary or redundant?
• Could someone follow this without prior context?

How to test: Run the skill on 2-3 real tasks (not hypothetical). Score after each real use. The test inputs ARE the tasks you're applying the skill to.

Dimensional scoring for process skills:

• Clarity — Can I follow each step without re-reading?
• Completeness — Does the process cover the full workflow?
• Actionability — Do I know exactly what to do at each step, or do I have to infer?
• Efficiency — Are there wasted/redundant steps?
• Self-applicability — Can this process improve itself? (Meta-test)

Checklist Sweet Spot

• 3-6 questions = optimal
• Too few: not granular enough to guide changes
• Too many: skill starts gaming the checklist (like a student memorizing answers without understanding)

When to Use

• Before running any skill at scale (cold outreach, content generation, scraping)
• After a new model upgrade — re-validate existing skills
• When a skill has inconsistent output quality
• Monthly maintenance pass on high-use skills
• Immediately after creating a new skill (structural audit only takes 5 min)

When to Run Which Phase

• Any new skill → Structure audit (5 min, catches issues early)
• Before scale use → Output loop (validate quality before mass runs)
• After model upgrade → Output loop (re-validate existing skills)
• Inconsistent output → Output loop (find the failing item/dimension)
• High-revenue skills → Both phases (cold outreach, content gen — quality variance = revenue impact)

Gotchas

• Output loop requires skills that produce scoreable text outputs — scripts/tools that produce side effects need a different verification approach (use a Product Verification skill type instead)
• Don't run output loop on skills that call expensive APIs without rate limit awareness — each round runs the skill multiple times
• Phase 1 (structure audit) should always run before Phase 2 — fixing structure first makes the output loop more effective
• 3-6 checklist questions is the sweet spot — more than 6 and the skill starts gaming individual checks rather than improving overall quality