Judge

Overview

Evaluate agent task outputs against a three-dimension rubric and produce structured verdict records. The judge operates as a quality gate at the task completion boundary, scoring outputs on Semantic accuracy, Pragmatic usefulness, and Syntactic consistency.

Rubric: Three KLS Dimensions

The rubric reuses three dimensions from the KLS (Krogstie-Lindland-Sindre) quality framework defined in disciplined-quality-evaluation:

Scoring

Verdict Thresholds

Verdict Format

Every judge evaluation produces a JSON verdict record. See automation/judge/verdict-schema.json for the full schema. Minimal structure:

{
  "task_id": "issue-18",
  "model": "opencode/gpt-5-nano",
  "mode": "quick",
  "verdict": "accept",
  "scores": {
    "semantic": 4,
    "pragmatic": 4,
    "syntactic": 5
  },
  "average": 4.33,
  "reasoning": "Brief justification for scores",
  "improvements": [],
  "timestamp": "2026-02-17T14:30:00Z"
}

Output MUST be valid JSON only -- no markdown fencing, no preamble, no trailing text.

Judge Modes

Quick Judge

• Model: opencode/gpt-5-nano (fast, low latency)
• Purpose: Rapid pass/fail screening at task completion
• Timeout: 30 seconds
• Prompt template: references/prompt-quick.md
• Output: Verdict JSON with scores and one-line reasoning

Deep Judge

• Model: opencode/kimi-k2.5-free (thorough, reasoning-capable)
• Purpose: Detailed evaluation with improvement suggestions
• Timeout: 60 seconds
• Prompt template: references/prompt-deep.md
• Output: Verdict JSON with scores, reasoning chain, and improvement list

Tiebreaker

• Model: opencode/gpt-5.1-codex-mini (independent perspective, different provider)
• Purpose: Resolve accept/reject disagreement between quick and deep judges
• Timeout: 45 seconds
• Prompt template: Uses deep prompt with prior verdicts appended
• Trigger: Quick and deep verdicts disagree on accept vs reject/improve

Multi-Iteration Protocol

1. Run quick judge
   |
   +-- accept --> DONE (accept)
   +-- reject --> DONE (reject, log improvements)
   +-- improve --> Run deep judge
                   |
                   +-- accept --> DONE (accept)
                   +-- reject --> DONE (reject, log improvements)
                   +-- improve --> Human fallback

When quick and deep disagree on accept vs reject:

Quick: accept + Deep: reject --> Run tiebreaker
Quick: reject + Deep: accept --> Run tiebreaker
Tiebreaker verdict is final (no further iteration)

Maximum 3 model calls before human fallback.

Human Fallback Criteria

Escalate to human review when:

1. Three model calls completed without consensus
2. Any model returns invalid JSON after retry
3. Any model times out after retry
4. Verdict is "improve" from both quick and deep judges (ambiguous quality)
5. Task involves security-sensitive or compliance-critical content

Integration Points

Runner Script

The judge is invoked by automation/judge/run-judge.sh (v2), which:

• Writes prompts to temp files (eliminates shell escaping issues with file content)
• Calls opencode with --file <tempfile> for reliable prompt delivery
• Parses verdict JSON from opencode's JSON event stream
• Implements the multi-iteration protocol (quick -> deep -> tiebreaker)
• Optionally enriches verdict logging with terraphim-cli term matching

Verdict Logging

All verdicts are appended to automation/judge/verdicts.jsonl as one JSON object per line. This enables:

• Audit trail of all quality evaluations
• Trend analysis across tasks
• Model performance comparison

Pre-Push Hook

The judge can be invoked as a pre-push hook via automation/judge/pre-push-judge.sh, blocking pushes that receive a "reject" verdict.

Terraphim Integration (Optional)

When terraphim-cli is available, the judge runner uses knowledge graph-based term normalization to identify rubric dimensions in model reasoning.

Knowledge Graph Setup

# Install judge KG files and configure role
bash automation/judge/setup-judge-kg.sh

# Verify installation
terraphim-cli thesaurus --limit 50
terraphim-cli find "factual correctness and actionability"

KG Files

Located in automation/judge/kg/:

Fail-Open Design

If terraphim-cli is not installed, the judge falls back to direct JSON extraction without term normalization. All core functionality works without terraphim -- the KG integration is an enrichment layer.

Resources

references/

• prompt-quick.md -- Quick judge prompt template for fast screening
• prompt-deep.md -- Deep judge prompt template for thorough evaluation