Shadow Mode Migration Pattern

Shadow mode mirrors production traffic to a new system without affecting users. The shadow system's responses are discarded — only the production response reaches the user — but both responses are logged and compared to validate correctness.

When to Use This Skill

Use this skill when... Use dual-write instead when...

Validating read behavior of a replacement service Both systems need to persist writes

Testing performance under real production load You need the new store to be authoritative

Comparing response correctness before cutover Migrating data stores that must stay in sync

Evaluating a new service version safely The new system needs to receive and store mutations

Load testing a new deployment with real traffic You need strong consistency between systems

Core Concepts

Traffic Flow

Client Request │ ▼ ┌─────────────┐ │ Router / │ │ Proxy │ ├──────┬──────┤ │ │ │ ▼ │ ▼ Prod │ Shadow System │ System │ │ │ ▼ │ ▼ Prod │ Shadow Response│ Response │ │ │ ▼ │ (discard) Client │ │ │ ▼ │ Compare & │ Log ▼

Shadow Modes

Mode Description Use case

Full mirror 100% of traffic duplicated Final validation before cutover

Sampled mirror Percentage of traffic (e.g., 10%) Early validation, capacity-constrained shadow

Selective mirror Specific request types or endpoints Targeted validation of changed behavior

Replay mirror Recorded traffic replayed offline Testing without live shadow infrastructure

Implementation Architecture

Key Components

Component Responsibility

Traffic splitter Duplicates requests to shadow system

Shadow router Forwards mirrored requests, manages timeouts

Response comparator Compares prod vs shadow responses

Discrepancy logger Records differences with full context

Metrics collector Tracks match rates, latency, error rates

Kill switch Disables shadow traffic instantly if issues arise

Deployment Topology

Topology How it works Trade-offs

Proxy-based Load balancer or API gateway mirrors requests Simple setup, adds proxy hop

Application-level Application code sends async copy of request Fine-grained control, code coupling

Infrastructure-level Service mesh (Istio, Linkerd) mirrors traffic No code changes, requires mesh

Log replay Capture request logs, replay against shadow No live infrastructure needed, not real-time

Implementation Patterns

Proxy-Based Mirroring

Configure the load balancer or API gateway to:

• Forward the original request to the production backend

• Clone the request and send it to the shadow backend

• Return only the production response to the client

• Shadow response is logged but never returned

• Shadow request timeout is independent of production

Application-Level Mirroring

• Intercept the incoming request at the application layer

• Process the request normally through the production path

• Asynchronously send a copy of the request to the shadow service

• Do not block the production response on the shadow response

• Compare responses in a background worker

Response Comparison Strategy

Compare responses field by field with configurable rules:

Field type Comparison approach

IDs, timestamps Ignore (expected to differ)

Computed values Compare within tolerance (e.g., floating point)

Collections Compare as sets (ignore ordering unless significant)

Status codes Exact match required

Error responses Categorize and compare error types

Headers Compare relevant headers only (Content-Type, Cache-Control)

Handling Stateful Requests

Shadow mode works best with read-only requests. For stateful (write) requests:

Approach Description

Skip writes Only mirror read requests to shadow

Isolated state Shadow has its own database seeded from production

Dry-run writes Shadow validates the write but does not persist

Record-only Log what shadow would have written, compare intent

Gradual Rollout

Phase Traffic % Duration Goal

1. Smoke test 1% Hours Verify shadow receives and processes requests

2. Canary 5-10% Days Identify obvious discrepancies

3. Validation 25-50% Days-weeks Build confidence in match rate

4. Full mirror 100% Days-weeks Final validation before cutover

Validation Metrics

Metric Target Description

Response match rate

99.9% Percentage of identical responses

Shadow latency (P50) Within 2x of prod Shadow performance baseline

Shadow latency (P99) Monitored Tail latency under real load

Shadow error rate < prod error rate Shadow should not produce more errors

Shadow availability Monitored Shadow uptime (not a blocker)

Discrepancy categories Trending to zero Known differences resolved over time

Common Pitfalls

Pitfall Mitigation

Shadow affects production performance Async mirroring, independent timeouts, kill switch

Shadow writes to shared resources Isolate shadow databases, queues, and external services

Non-deterministic responses cause false mismatches Configure comparison rules to ignore timestamps, IDs, nonces

Shadow receives stale data Seed shadow database from recent production snapshot

Traffic amplification overwhelms shadow Use sampled mirroring, auto-scaling, or circuit breakers

Request ordering differs between prod and shadow Compare request-by-request, not sequence-dependent

Authentication tokens expire for shadow Mint shadow-specific tokens or bypass auth in shadow

Integration with Dual Write

Shadow mode and dual write are complementary migration techniques:

Migration phase Technique Purpose

Early validation Shadow mode (reads) Verify the new system returns correct responses

Data sync Dual write Keep both stores authoritative during transition

Pre-cutover Both simultaneously Shadow validates reads, dual write maintains data

Cutover Dual write reversal New system becomes primary, old becomes secondary

Post-cutover Shadow mode (reversed) Mirror to old system to verify nothing broke

Strangler Fig Context

Both patterns are tactics within the broader Strangler Fig migration strategy:

• Identify a component to migrate

• Shadow traffic to validate the replacement

• Dual write to synchronize data stores

• Cut over reads, then writes

• Decommission the old component

• Repeat for the next component

Kill Switch Requirements

Shadow mode must have an immediate disable mechanism:

• Feature flag or configuration toggle (no deployment required)

• Disables within seconds, not minutes

• Monitored — alerts if shadow causes production impact

• Tested before enabling shadow traffic

Monitoring Checklist

• Production latency impact (should be zero or negligible)

• Shadow request success rate

• Shadow response latency distribution

• Response match rate by endpoint

• Discrepancy log volume and categories

• Shadow system resource utilization

• Kill switch status and responsiveness

Agentic Optimizations

Context Approach

Architecture review Verify shadow isolation (no shared writes), kill switch exists

Code review Check async mirroring does not block production path

Implementation Start with proxy-based mirroring at 1%, increase gradually

Testing Verify kill switch works, confirm production is unaffected when shadow fails

Quick Reference

Term Definition

Shadow system The new system receiving mirrored traffic

Production system The live system serving real users

Traffic splitter Component that duplicates requests

Match rate Percentage of shadow responses matching production

Kill switch Mechanism to instantly disable shadow traffic

Dark launching Synonym for shadow mode — feature is live but invisible to users

Canary traffic Small percentage of mirrored requests for initial validation

Strangler fig Broader migration strategy of incrementally replacing components