JSON Data File Validation Test Design

Extracted: 2026-02-11 Context: Validating a large JSON data file (exam questions) generated by a build script against its schema, source data, and business rules

Problem

JSON data files generated by scripts (from text, CSV, API, etc.) can contain subtle issues:

• Stray characters from OCR/copy-paste (e.g., ß mixed into Japanese text)

• Schema violations that the app silently swallows

• Cross-reference mismatches (source data vs generated output)

• Missing or duplicate entries

• Business rule violations (e.g., correct answer not in choices)

Manual review of large files (60+ entries, 3000+ lines) is unreliable.

Solution: Layered Pytest Validation

Structure tests in layers from structural to semantic:

Layer 1: Top-level structure

class TestTopLevelStructure: def test_required_fields(self, data): ... def test_count_matches(self, data): assert data["totalItems"] == len(data["items"])

Layer 2: Per-entry schema validation

class TestEntryFields: def test_required_fields(self, entries): for e in entries: missing = REQUIRED - e.keys() assert not missing, f"Entry {e['id']}: missing {missing}"

def test_enum_values(self, entries):
    for e in entries:
        assert e["type"] in VALID_TYPES

Layer 3: Cross-entry consistency

class TestConsistency: def test_no_duplicates(self, entries): ids = [e["id"] for e in entries] assert len(ids) == len(set(ids))

def test_references_resolve(self, entries, categories):
    # Every entry's category exists in categories list

Layer 4: Source cross-reference

class TestSourceCrossReference: @pytest.fixture def source_data(self): # Parse original source files ...

def test_values_match_source(self, entries, source_data):
    mismatches = []
    for e in entries:
        if e["answer"] != source_data[e["id"]]:
            mismatches.append(...)
    assert not mismatches, f"{len(mismatches)} mismatches"

Layer 5: Content quality heuristics

class TestContentQuality: def test_min_text_length(self, entries): for e in entries: assert len(e["text"]) >= THRESHOLD

def test_no_stray_characters(self, entries):
    stray = {"ß", "€", "£"}  # Characters unlikely in this domain
    issues = []
    for e in entries:
        for ch in stray:
            if ch in e["text"]:
                issues.append(f"{e['id']}: '{ch}'")
    assert not issues

Key Design Decisions

• Module-scoped fixtures for the parsed JSON (scope="module" ) to avoid re-reading per test

• Collect-all-errors pattern: accumulate issues in a list, assert at end, so one test run shows all problems

• Graceful degradation: source cross-reference tests skip with pytest.skip() if source files are absent

• Domain-aware thresholds: min length for text depends on the domain (e.g., 2 chars for Japanese terms like "過学習")

When to Use

• After generating/rebuilding JSON data files from external sources

• As a CI gate for data files that feed into apps

• When a data file is too large for manual review

• When data is parsed from inconsistent sources (OCR, PDF export, manual entry)