Agent Evaluation (AI Agent Evals)

Based on Anthropic's "Demystifying evals for AI agents"

When to use this skill

• Designing evaluation systems for AI agents

• Building benchmarks for coding, conversational, or research agents

• Creating graders (code-based, model-based, human)

• Implementing production monitoring for AI systems

• Setting up CI/CD pipelines with automated evals

• Debugging agent performance issues

• Measuring agent improvement over time

Core Concepts

Eval Evolution: Single-turn → Multi-turn → Agentic

Type Turns State Grading Complexity

Single-turn 1 None Simple Low

Multi-turn N Conversation Per-turn Medium

Agentic N World + History Outcome High

7 Key Terms

Term Definition

Task Single test case (prompt + expected outcome)

Trial One agent run on a task

Grader Scoring function (code/model/human)

Transcript Full record of agent actions

Outcome Final state for grading

Harness Infrastructure running evals

Suite Collection of related tasks

Instructions

Step 1: Understand Grader Types

Code-based Graders (Recommended for Coding Agents)

• Pros: Fast, objective, reproducible

• Cons: Requires clear success criteria

• Best for: Coding agents, structured outputs

Example: Code-based grader

def grade_task(outcome: dict) -> float: """Grade coding task by test passage.""" tests_passed = outcome.get("tests_passed", 0) total_tests = outcome.get("total_tests", 1) return tests_passed / total_tests

SWE-bench style grader

def grade_swe_bench(repo_path: str, test_spec: dict) -> bool: """Run tests and check if patch resolves issue.""" result = subprocess.run( ["pytest", test_spec["test_file"]], cwd=repo_path, capture_output=True ) return result.returncode == 0

Model-based Graders (LLM-as-Judge)

• Pros: Flexible, handles nuance

• Cons: Requires calibration, can be inconsistent

• Best for: Conversational agents, open-ended tasks

Example: LLM Rubric for Customer Support Agent

rubric: dimensions: - name: empathy weight: 0.3 scale: 1-5 criteria: | 5: Acknowledges emotions, uses warm language 3: Polite but impersonal 1: Cold or dismissive

- name: resolution
  weight: 0.5
  scale: 1-5
  criteria: |
    5: Fully resolves issue
    3: Partial resolution
    1: No resolution

- name: efficiency
  weight: 0.2
  scale: 1-5
  criteria: |
    5: Resolved in minimal turns
    3: Reasonable turns
    1: Excessive back-and-forth

Human Graders

• Pros: Highest accuracy, catches edge cases

• Cons: Expensive, slow, not scalable

• Best for: Final validation, ambiguous cases

Step 2: Choose Strategy by Agent Type

2.1 Coding Agents

Benchmarks:

• SWE-bench Verified: Real GitHub issues (40% → 80%+ achievable)

• Terminal-Bench: Complex terminal tasks

• Custom test suites with your codebase

Grading Strategy:

def grade_coding_agent(task: dict, outcome: dict) -> dict: return { "tests_passed": run_test_suite(outcome["code"]), "lint_score": run_linter(outcome["code"]), "builds": check_build(outcome["code"]), "matches_spec": compare_to_reference(task["spec"], outcome["code"]) }

Key Metrics:

• Test passage rate

• Build success

• Lint/style compliance

• Diff size (smaller is better)

2.2 Conversational Agents

Benchmarks:

• τ2-Bench: Multi-domain conversation

• Custom domain-specific suites

Grading Strategy (Multi-dimensional):

success_criteria:

• empathy_score: >= 4.0
• resolution_rate: >= 0.9
• avg_turns: <= 5
• escalation_rate: <= 0.1

Key Metrics:

• Task resolution rate

• Customer satisfaction proxy

• Turn efficiency

• Escalation rate

2.3 Research Agents

Grading Dimensions:

• Grounding: Claims backed by sources

• Coverage: All aspects addressed

• Source Quality: Authoritative sources used

def grade_research_agent(task: dict, outcome: dict) -> dict: return { "grounding": check_citations(outcome["report"]), "coverage": check_topic_coverage(task["topics"], outcome["report"]), "source_quality": score_sources(outcome["sources"]), "factual_accuracy": verify_claims(outcome["claims"]) }

2.4 Computer Use Agents

Benchmarks:

• WebArena: Web navigation tasks

• OSWorld: Desktop environment tasks

Grading Strategy:

def grade_computer_use(task: dict, outcome: dict) -> dict: return { "ui_state": verify_ui_state(outcome["screenshot"]), "db_state": verify_database(task["expected_db_state"]), "file_state": verify_files(task["expected_files"]), "success": all_conditions_met(task, outcome) }

Step 3: Follow the 8-Step Roadmap

Step 0: Start Early (20-50 Tasks)

Create initial eval suite structure

mkdir -p evals/{tasks,results,graders}

Start with representative tasks

- Common use cases (60%)

- Edge cases (20%)

- Failure modes (20%)

Step 1: Convert Manual Tests

Transform existing QA tests into eval tasks

def convert_qa_to_eval(qa_case: dict) -> dict: return { "id": qa_case["id"], "prompt": qa_case["input"], "expected_outcome": qa_case["expected"], "grader": "code" if qa_case["has_tests"] else "model", "tags": qa_case.get("tags", []) }

Step 2: Ensure Clarity + Reference Solutions

Good task definition

task: id: "api-design-001" prompt: | Design a REST API for user management with: - CRUD operations - Authentication via JWT - Rate limiting reference_solution: "./solutions/api-design-001/" success_criteria: - "All endpoints documented" - "Auth middleware present" - "Rate limit config exists"

Step 3: Balance Positive/Negative Cases

Ensure eval suite balance

suite_composition = { "positive_cases": 0.5, # Should succeed "negative_cases": 0.3, # Should fail gracefully "edge_cases": 0.2 # Boundary conditions }

Step 4: Isolate Environments

Docker-based isolation for coding evals

eval_environment: type: docker image: "eval-sandbox:latest" timeout: 300s resources: memory: "4g" cpu: "2" network: isolated cleanup: always

Step 5: Focus on Outcomes, Not Paths

GOOD: Outcome-focused grader

def grade_outcome(expected: dict, actual: dict) -> float: return compare_final_states(expected, actual)

BAD: Path-focused grader (too brittle)

def grade_path(expected_steps: list, actual_steps: list) -> float: return step_by_step_match(expected_steps, actual_steps)

Step 6: Always Read Transcripts

Transcript analysis for debugging

def analyze_transcript(transcript: list) -> dict: return { "total_steps": len(transcript), "tool_usage": count_tool_calls(transcript), "errors": extract_errors(transcript), "decision_points": find_decision_points(transcript), "recovery_attempts": find_recovery_patterns(transcript) }

Step 7: Monitor Eval Saturation

Detect when evals are no longer useful

def check_saturation(results: list, window: int = 10) -> dict: recent = results[-window:] return { "pass_rate": sum(r["passed"] for r in recent) / len(recent), "variance": calculate_variance(recent), "is_saturated": all(r["passed"] for r in recent), "recommendation": "Add harder tasks" if saturated else "Continue" }

Step 8: Long-term Maintenance

Eval suite maintenance checklist

maintenance: weekly: - Review failed evals for false negatives - Check for flaky tests monthly: - Add new edge cases from production issues - Retire saturated evals - Update reference solutions quarterly: - Full benchmark recalibration - Team contribution review

Step 4: Integrate with Production

CI/CD Integration

GitHub Actions example

name: Agent Evals on: [push, pull_request]

jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Evals run: | python run_evals.py --suite=core --mode=compact - name: Upload Results uses: actions/upload-artifact@v4 with: name: eval-results path: results/

Production Monitoring

Real-time eval sampling

class ProductionMonitor: def init(self, sample_rate: float = 0.1): self.sample_rate = sample_rate

async def monitor(self, request, response):
    if random.random() < self.sample_rate:
        eval_result = await self.run_eval(request, response)
        self.log_result(eval_result)
        if eval_result["score"] < self.threshold:
            self.alert("Low quality response detected")

A/B Testing

Compare agent versions

def run_ab_test(suite: str, versions: list) -> dict: results = {} for version in versions: results[version] = run_eval_suite(suite, agent_version=version) return { "comparison": compare_results(results), "winner": determine_winner(results), "confidence": calculate_confidence(results) }

Best Practices

Do's ✅

• Start with 20-50 representative tasks

• Use code-based graders when possible

• Focus on outcomes, not paths

• Read transcripts for debugging

• Monitor for eval saturation

• Balance positive/negative cases

• Isolate eval environments

• Version your eval suites

Don'ts ❌

• Don't over-rely on model-based graders without calibration

• Don't ignore failed evals (false negatives exist)

• Don't grade on intermediate steps

• Don't skip transcript analysis

• Don't use production data without sanitization

• Don't let eval suites become stale

Success Patterns

Pattern 1: Graduated Eval Complexity

Level 1: Unit evals (single capability) Level 2: Integration evals (combined capabilities) Level 3: End-to-end evals (full workflows) Level 4: Adversarial evals (edge cases)

Pattern 2: Eval-Driven Development

1. Write eval task for new feature
2. Run eval (expect failure)
3. Implement feature
4. Run eval (expect pass)
5. Add to regression suite

Pattern 3: Continuous Calibration

Weekly: Review grader accuracy Monthly: Update rubrics based on feedback Quarterly: Full grader audit with human baseline

Troubleshooting

Problem: Eval scores at 100%

Solution: Add harder tasks, check for eval saturation (Step 7)

Problem: Inconsistent model-based grader scores

Solution: Add more examples to rubric, use structured output, ensemble graders

Problem: Evals too slow for CI

Solution: Use toon mode, parallelize, sample subset for PR checks

Problem: Agent passes evals but fails in production

Solution: Add production failure cases to eval suite, increase diversity

References

• Anthropic: Demystifying evals for AI agents

• SWE-bench

• WebArena

• τ2-Bench

Examples

Example 1: Simple Coding Agent Eval

Task definition

task = { "id": "fizzbuzz-001", "prompt": "Write a fizzbuzz function in Python", "test_cases": [ {"input": 3, "expected": "Fizz"}, {"input": 5, "expected": "Buzz"}, {"input": 15, "expected": "FizzBuzz"}, {"input": 7, "expected": "7"} ] }

Grader

def grade(task, outcome): code = outcome["code"] exec(code) # In sandbox for tc in task["test_cases"]: if fizzbuzz(tc["input"]) != tc["expected"]: return 0.0 return 1.0

Example 2: Conversational Agent Eval with LLM Rubric

task: id: "support-refund-001" scenario: | Customer wants refund for damaged product. Product: Laptop, Order: #12345, Damage: Screen crack expected_actions: - Acknowledge issue - Verify order - Offer resolution options max_turns: 5

grader: type: model model: claude-3-5-sonnet-20241022 rubric: | Score 1-5 on each dimension: - Empathy: Did agent acknowledge customer frustration? - Resolution: Was a clear solution offered? - Efficiency: Was issue resolved in reasonable turns?