Eval Session Scorecard

Use this skill to evaluate a full conversation (multiple user/assistant turns) for continuous monitoring at the session level.

Inputs

Require:

• A conversation transcript containing multiple user/assistant turns.
• The transcript must clearly label turns as "User:" and "Assistant:".

Workflow

1. Parse the transcript into assistant turns and their immediate context (the preceding user turn and any relevant prior context).
2. For each assistant turn, run eval-core-scorecard on that assistant turn using:
   • User request/context: the preceding user message (plus brief prior context if needed).
   • Assistant response: the assistant message for that turn.
3. Collect per-turn outputs (each is the JSON object returned by eval-core-scorecard).
4. Compute session-level aggregates:
   • session_average_score: mean of each turn’s average_score (float allowed).
   • dimension_averages: for each of the 7 dimensions, mean of that dimension’s score across turns.
   • lowest_scoring_turns: list the 3 assistant turns with lowest average_score (include turn index + average_score).
5. Return a single strict JSON object.

Output Contract

Return JSON only. Do not include markdown, backticks, prose, or extra keys.

Use exactly this schema:

{ "dimension": "session_scorecard", "assistant_turn_count": 0, "turn_count": 0, "session_average_score": 0, "dimension_averages": { "clarity": 0, "relevance": 0, "accuracy": 0, "tone_empathy": 0, "guidance_actionability": 0, "conversation_flow": 0, "boundary_adherence": 0 }, "lowest_scoring_turns": [ { "assistant_turn_index": 1, "average_score": 0 }, { "assistant_turn_index": 1, "average_score": 0 }, { "assistant_turn_index": 1, "average_score": 0 } ], "turn_scorecards": [ { "assistant_turn_index": 1, "user_message": "...", "assistant_message": "...", "scorecard": { "dimension": "core_scorecard", "average_score": 0, "results": [ { "dimension": "clarity", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "relevance", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "accuracy", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "tone_empathy", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "guidance_actionability", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "conversation_flow", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] }, { "dimension": "boundary_adherence", "score": 1, "rationale": "...", "improvement_suggestions": ["..."] } ] } } ] }

Hard Rules

• dimension must always equal "session_scorecard".
• Output must be valid JSON and include all keys exactly as shown.
• turn_scorecards must include one entry per assistant turn found in the transcript.
• assistant_turn_index starts at 1 and increments by 1 for each assistant turn in the transcript.
• Do not include step-by-step reasoning.
• Never output text outside the JSON object.