Jury System Skill

Specialized knowledge for running synthetic user validation using the Condorcet Jury Theorem.

When to Use

• Validating research findings resonate broadly

• Testing PRD user stories match mental models

• Evaluating prototype usability

• Checking graduation criteria between phases

Core Principle

If each synthetic persona has >50% accuracy in judging whether a feature matches their needs, aggregating 100-500+ votes produces near-certain collective judgment.

Stratified Sampling

Dimension Distribution

Role Sales Rep: 40%, Sales Leader: 25%, CSM: 20%, RevOps: 15%

Tech Proficiency Novice: 25%, Intermediate: 50%, Advanced: 25%

AI Adoption Skeptic: 15% (min), Curious: 40%, Early Adopter: 35%, Power User: 10%

Critical: Always include 15% skeptics. They catch issues optimists miss.

Validation by Phase

Research Validation

Sample: 100-200 personas Pass: >60% rate resonance 4+

PRD Validation

Sample: 200-300 personas Pass: >70% rate relevance 4+

Prototype Evaluation

Sample: 300-500 personas Pass: >70% combined pass rate

Aggregation Rules

Verdict Threshold

Validated

60% rate 4+

Contested 40-60% rate 4+

Rejected <40% rate 4+

Evaluation Prompt Templates

Research Validation Prompt

You ARE {persona.name}, a {persona.role} at a {persona.company_size} company.

Your context:

• Tech comfort: {persona.tech_literacy}
• AI attitude: {persona.ai_adoption_stage}
• Your primary pain: {persona.primary_pain}

A product team has identified this pain point from customer research:

PAIN POINT: {extracted_pain_point} SUPPORTING QUOTE: "{supporting_quote}"

As yourself, respond in JSON:

{ "resonance_score": [1-5, where 1="not my problem", 5="exactly my frustration"], "perspective": "[2-3 sentences explaining why this does/doesn't resonate, in first person]", "missing_aspect": "[Optional: related pain they might have overlooked]" }

PRD User Story Validation Prompt

You ARE {persona.name}. A product team proposes this user story:

USER STORY: As a {story.persona}, I want to {story.action} so that {story.benefit}.

ACCEPTANCE CRITERIA: {story.criteria}

Evaluate as yourself, respond in JSON:

{ "relevance_score": [1-5], "clarity": "clear" | "somewhat_clear" | "confusing", "missing_from_your_perspective": "[what's missing]", "usage_frequency": "daily" | "weekly" | "monthly" | "rarely" | "never" }

Prototype Evaluation Prompt

You ARE {persona.name}, a {persona.role} at a {persona.company_size} company.

YOUR CONTEXT:

• Tech comfort: {persona.tech_literacy}
• AI trust: {persona.trust_in_ai}
• Patience for learning: {persona.patience_for_learning}

SCENARIO: {scenario.description} YOUR TASK: {scenario.task.primary_goal}

PROTOTYPE: {prototype_description}

THE COMPLETE EXPERIENCE JOURNEY:

1. DISCOVERY: {discovery_mechanism} How you would first learn this feature exists.
2. ACTIVATION: {activation_flow} How you would set it up / enable it for the first time.
3. USAGE: {usage_description} What your first interaction looks like.
4. ONGOING VALUE: {ongoing_value_description} What happens when you come back the next day / next week.
5. FEEDBACK: {feedback_mechanism} How the product team would hear from you about whether this is working.

Evaluate this prototype AND the full experience in JSON:

{ "first_impression": "[What you notice first, what's unclear]", "task_walkthrough": { "steps_you_would_try": ["step 1", "step 2"], "hesitation_points": ["where you'd pause"], "would_give_up": true | false, "give_up_reason": "[if true, why]" }, "experience_journey_scores": { "discovery": { "score": [1-5], "reason": "Would I actually find this?" }, "activation": { "score": [1-5], "reason": "Could I set this up alone?" }, "usage": { "score": [1-5], "reason": "Does the first interaction make sense?" }, "ongoing_value": { "score": [1-5], "reason": "Would I come back to this?" }, "feedback_loop": { "score": [1-5], "reason": "Would I bother giving feedback?" }, "experience_coherence": [1-5, "Does the full journey feel connected?"] }, "weakest_experience_step": "[which of the 5 steps is weakest and why]", "heuristic_scores": { "visibility_of_status": { "score": [1-5], "reason": "..." }, "match_with_expectations": { "score": [1-5], "reason": "..." }, "user_control": { "score": [1-5], "reason": "..." }, "consistency": { "score": [1-5], "reason": "..." }, "error_prevention": { "score": [1-5], "reason": "..." } }, "issues": [ { "what": "[description]", "where": "[UI element or experience step]", "severity": "cosmetic" | "minor" | "major" | "catastrophic", "why_matters_to_you": "[persona-specific impact]" } ], "emotional_response": { "frustration": [1-5], "confidence": [1-5], "would_recommend": [1-5] }, "verdict": { "would_use": true | false, "reasoning": "[why/why not]" } }

Self-Consistency Filter

Run each evaluation 3x with temperature 0.7. Only count vote if 2/3 or 3/3 agree. Discard inconsistent responses.

Model Selection

Operation Model Rationale

Persona Generation Claude Haiku Cost-effective for volume

Research Validation Claude Haiku Simple resonance scoring

PRD Validation Claude Haiku Structured output

Prototype Evaluation Claude Haiku Volume of evaluations

Synthesis/Aggregation Claude Sonnet Quality of final insights

Temperature Settings:

• Persona generation: 0.9 (maximize diversity)

• Evaluation: 0.7 (balanced for self-consistency)

• Synthesis: 0.3 (consistent, coherent output)

Cost Estimation

Phase Sample Size Estimated Cost

Research Validation 200 personas × 5 pains ~$0.50

PRD Validation 300 personas × 10 stories ~$1.00

Prototype Evaluation 500 personas ~$2.00

Synthesis 1 aggregation ~$0.50

Total per initiative

~$4.00

Output File Locations

Save to pm-workspace-docs/initiatives/active/[name]/jury-evaluations/ :

• research-v1.json

• Pain point resonance

• prd-v1.json

• User story validation

• proto-v1.json

• Usability evaluation (raw)

• jury-report.md

• Human-readable synthesis

• iteration-log.md

• Change tracking

Quality Checks

Before trusting results:

• Sample size adequate (≥100 research, ≥200 PRD, ≥300 proto)

• Skeptic representation ≥15%

• All relevant archetypes represented

• Self-consistency applied

• Variance check (std > 0.5 on 5-point scales)

Scripts Reference

Existing scripts in pm-workspace-docs/scripts/jury-system/ :

• simulate_jury.py

• Run jury simulation

• iterate_from_feedback.py

• Generate iteration docs

This System Supplements, Not Replaces

Use for: ✅ Rapid validation between real interviews ✅ Catching obvious mismatches before investing ✅ Covering personas you haven't talked to yet

Do NOT use to: ❌ Replace actual customer conversations ❌ Make final launch decisions without real validation