Quality Scoring

Multi-dimensional assessment for training data quality.

When Activates

Quality assessment, data scoring, multi-dimensional evaluation, IFD scoring, factuality checks, reasoning validation, training data prep

Core Concepts

Quality Scorers (6 Types)

Fast to comprehensive scoring approaches:

FastIFD - Instruction-following difficulty (10-20x faster)
Quality - LLM-based quality (Qwen3-30B, 0.85 ex/s)
MultiDimensional - 5-dimension composite
LLMQuality - Multi-backend (MLX/OpenRouter)
Ensemble - Cross-model ensemble
Tulu3 - Multi-dimensional reference (training_metrics.py)

Quality Dimensions (6 Metrics)

IFD Score (0.0-1.0) - Instruction-following difficulty
Factuality (0.0-1.0) - Hallucination detection
Reasoning (0.0-1.0) - Step-by-step logic quality
Diversity (0.0-1.0) - Dataset-level diversity
Domain (0.0-1.0) - Domain-specific relevance
LLM Quality (1-10) - Tulu3 comprehensive score

Training Thresholds

Type Quality IFD Use Case

SFT ≥8.0 ≥0.3 Base training

DPO chosen ≥9.0 ≥0.5 High quality only

DPO rejected ≤6.0 any Low quality

RLVR ≥9.0 ≥0.5 Verified solutions

Calibration ≥8.0 ≥0.4 Uncertainty examples

Quick Reference

Concept Details Reference

Scorers 6 types (FastIFD to Ensemble) quality-scorers.md

Dimensions 6 metrics (IFD to LLM Quality) quality-dimensions.md

Thresholds By training type (SFT, DPO, RLVR) training-thresholds.md

Library training_metrics.py

Integration functions

IFD Score Calculation

from training_metrics import calculate_ifd_score

IFD = PPL(response) / PPL(response|instruction)

ifd_score = calculate_ifd_score( instruction="Explain quantum computing", response="Quantum computing uses qubits..." )

Higher score = more challenging

DPO Pair Validation

from training_metrics import validate_dpo_pairs

Validate chosen/rejected quality gap

is_valid = validate_dpo_pairs( chosen_score=9.2, # High quality rejected_score=5.8 # Low quality )

Ensures quality gap ≥0.15

REQUIRED: DPO Multi-Dimensional Scoring

Every DPO pair MUST have multi-dimensional quality scores before training.

This is a hard requirement — DPO data without quality scores will learn shortcuts (e.g., "longer = better") instead of genuine preference signal.

Required output fields per pair:

chosen_score (float): Composite quality score for chosen response
rejected_score (float): Composite quality score for rejected response
margin (float): chosen_score - rejected_score (must be ≥3.0)

Length bias audit (MUST run before DPO training):

from pathlib import Path from training_metrics import validate_dpo_pairs

metrics = validate_dpo_pairs(dpo_path=Path("dpo_pairs.jsonl"))

Check length bias

longer_chosen = sum(1 for p in metrics.pairs if len(p.chosen) > len(p.rejected)) length_bias = longer_chosen / metrics.total_pairs

if length_bias > 0.70: raise ValueError( f"DPO length bias {length_bias:.0%} > 70% threshold.\n" f"Model will learn 'longer = better' shortcut.\n" f"Fix: Score by quality dimensions, not length." )

Check quality scores present

missing = sum(1 for p in metrics.pairs if p.chosen_score is None) if missing > 0: raise ValueError(f"{missing} pairs missing quality scores — run scoring first")

Scoring workflow:

Generate DPO pairs (dpo-rlvr-generation skill)
Score all pairs with multi-dimensional scorer (this skill)
Filter by quality margin ≥3.0
Audit length bias ≤70%
Only then proceed to training

RLVR Verifiability

from training_metrics import assess_rlvr_verifiability

Assess reasoning trace verifiability

verifiable = assess_rlvr_verifiability( reasoning_trace="Step 1: ...\nStep 2: ...", domain="math" )

Math/coding: 90%+ verifiable required

Progressive Disclosure

Detailed guides: See docs/*.md

docs/quality-scorers.md
6 scorer implementations
docs/quality-dimensions.md
6 dimension definitions
docs/training-thresholds.md
Thresholds, CLI, distributed performance

Security Considerations

Input Validation (CWE-20)

Validate score ranges (0.0-1.0 or 1-10)
Sanitize data inputs before scoring
Check threshold values before application

Path Traversal (CWE-22)

Sanitize file paths for data loading
Whitelist directories for training data
Validate output paths for scored datasets

Security Patterns (training_metrics.py)

from pathlib import Path

def safe_load_data(data_path: str) -> dict: """Load data with path validation.""" # Validate path within allowed directory path = Path(data_path).resolve() if not str(path).startswith('/allowed/data/'): raise ValueError(f"Path outside allowed directory: {path}")

# Load safely
return json.loads(path.read_text())

Distributed Performance

Single Machine Performance

M4 Max: ~0.85 ex/s (Qwen3-30B)
M3 Ultra: ~0.85 ex/s (Qwen3-30B)

Parallel Processing

Combined throughput: ~1.7 ex/s (50/50 split)
Scaling: Linear with machine count
Bottleneck: Model inference, not I/O

CLI Commands

Score dataset with FastIFD

python -m training_metrics score
--input data/train.jsonl
--output data/scored.jsonl
--scorer fastifd
--threshold 0.3

Multi-dimensional scoring

python -m training_metrics score
--input data/train.jsonl
--output data/scored.jsonl
--scorer multidim
--quality-threshold 8.0
--ifd-threshold 0.5

DPO pair filtering

python -m training_metrics filter_dpo
--input data/dpo_pairs.jsonl
--output data/filtered_pairs.jsonl
--chosen-threshold 9.0
--rejected-threshold 6.0

RLVR verifiability check

python -m training_metrics assess_rlvr
--input data/rlvr_traces.jsonl
--output data/verified.jsonl
--domain math
--threshold 0.9

Related Skills

data-distillation - IFD methodology and KenLM filtering
preference-data-quality - DPO and RLVR metrics
python-standards - Code quality standards

Library Integration

Primary library: training_metrics.py

Key functions:

calculate_ifd_score()
IFD calculation
validate_dpo_pairs()
DPO pair validation
assess_rlvr_verifiability()
RLVR assessment
score_quality()
Multi-dimensional scoring
ensemble_score()
Cross-model ensemble

Key Takeaways

6 scorers - FastIFD (fast) to Ensemble (comprehensive)
6 dimensions - IFD, Factuality, Reasoning, Diversity, Domain, LLM Quality
Training thresholds - SFT ≥8.0, DPO chosen ≥9.0, RLVR ≥9.0
IFD score - PPL(response) / PPL(response|instruction), higher = harder
Security - CWE-20 (input validation), CWE-22 (path traversal)
Distributed - ~1.7 ex/s with 2 machines (linear scaling)
CLI commands - training_metrics module for all operations
Integration - Use training_metrics library functions
DPO pairs - Chosen ≥9.0, Rejected ≤6.0, gap ≥0.15
RLVR - Math/coding 90%+ verifiable, general 80%+
DPO scoring REQUIRED - Every pair must have chosen_score, rejected_score, margin before training
Length bias audit - ≤70% of pairs where chosen is longer (prevents "longer = better" shortcut)

quality scoring

Safety Notice

Copy this and send it to your AI assistant to learn