TigerBeetle Deterministic Simulation Style Guide⁠‍⁠​‌​‌​​‌‌‍​‌​​‌​‌‌‍​​‌‌​​​‌‍​‌​​‌‌​​‍​​​​​​​‌‍‌​​‌‌​‌​‍‌​​​​​​​‍‌‌​​‌‌‌‌‍‌‌​​​‌​​‍‌‌‌‌‌‌​‌‍‌‌​‌​​​​‍​‌​‌‌‌‌‌‍​‌​​‌​‌‌‍​‌‌​‌​​‌‍‌​‌​‌‌‌​‍​​‌​‌​​​‍‌‌‌​‌​‌‌‍‌‌​‌‌​‌‌‍‌‌‌​​‌​​‍‌​‌​​‌​​‍‌‌​‌‌‌​​‍​​​​‌​‌‌‍‌​‌​‌​​‌⁠‍⁠

Overview

TigerBeetle, created by Joran Dirk Greef, uses deterministic simulation testing to achieve correctness guarantees that traditional testing cannot provide. By controlling all sources of non-determinism (time, network, disk I/O, random numbers), the system can simulate years of operation in minutes, replay any failure scenario exactly, and find bugs that would otherwise take decades to manifest in production. This approach was pioneered by FoundationDB and refined by TigerBeetle for financial-grade reliability.

Core Philosophy

"If you can't reproduce it, you can't fix it. If you can't simulate it, you can't test it."

"Real-world testing finds bugs in hours. Deterministic simulation finds the same bugs in milliseconds."

"Time compression lets you experience a decade of production failures before your first deployment."

The fundamental insight is that distributed systems are impossible to test exhaustively with real hardware and real time. By simulating the environment deterministically, you can explore the state space millions of times faster than real time, with perfect reproducibility.

Design Principles

Control All Non-Determinism: Time, network, disk, random—everything must be injectable.

Simulate, Don't Emulate: The simulation is the source of truth, not an approximation.

Time Compression: Simulate years of operation in minutes.

Perfect Reproducibility: Any bug can be replayed with a single seed.

Failure Injection at Scale: Test every possible failure mode, not just the ones you imagine.

VOPR: Viewstamped Operation Replication Tester

TigerBeetle's VOPR runs deterministic simulations:

┌─────────────────────────────────────────────────────────────┐ │ VOPR Simulation Loop │ ├─────────────────────────────────────────────────────────────┤ │ │ │ Seed (u64) ─────► Deterministic PRNG │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Simulated Environment │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ Node 1 │ │ Node 2 │ │ Node 3 │ ... │ │ │ │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ │ │ │ │ │ │ │ │ │ └────────────┼────────────┘ │ │ │ │ │ │ │ │ │ ┌───────▼───────┐ │ │ │ │ │ Simulated │ │ │ │ │ │ Network/Disk │ │ │ │ │ │ Time/Faults │ │ │ │ │ └───────────────┘ │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Invariant Checking │ │ (after every operation) │ │ │ │ │ ▼ │ │ Pass/Fail + Reproducible Seed │ └─────────────────────────────────────────────────────────────┘

When Building Deterministic Simulation

Always

• Inject all sources of non-determinism through interfaces

• Use a seeded PRNG for all random decisions

• Make time a controllable parameter, not clock_gettime()

• Log enough state to reproduce any execution

• Check invariants after every state transition

• Run millions of simulations, not dozens

• Make the simulation faster than real-time

Never

• Call real system time in simulation mode

• Use real network or disk I/O in simulation

• Rely on thread scheduling for ordering

• Assume "it worked 1000 times, it's correct"

• Skip failure injection (partitions, crashes, slowness)

• Trust that rare events won't happen

Prefer

• Deterministic single-threaded simulation over multi-threaded tests

• Injected time over real time

• Simulated I/O over mocked I/O

• Invariant assertions over output checking

• Seed-based reproduction over log analysis

• Exhaustive failure injection over happy path

Code Patterns

Time Abstraction

const std = @import("std");

/// Abstract time source - real or simulated pub const Time = struct { /// Real time implementation pub const Real = struct { pub fn now() i64 { return std.time.milliTimestamp(); }

    pub fn sleep(ns: u64) void {
        std.time.sleep(ns);
    }
};

/// Simulated time - fully controlled
pub const Simulated = struct {
    current_tick: u64 = 0,
    pending_events: std.PriorityQueue(Event, void, Event.compare),
    
    pub fn now(self: *Simulated) i64 {
        return @intCast(self.current_tick);
    }
    
    pub fn sleep(self: *Simulated, ns: u64) void {
        // In simulation, sleep is instant - just advance the clock
        _ = ns;
    }
    
    pub fn advanceTo(self: *Simulated, tick: u64) void {
        std.debug.assert(tick >= self.current_tick);
        self.current_tick = tick;
    }
    
    pub fn scheduleAt(self: *Simulated, tick: u64, callback: Event) void {
        self.pending_events.add(.{
            .tick = tick,
            .callback = callback,
        }) catch unreachable;
    }
    
    pub fn processNextEvent(self: *Simulated) ?Event {
        if (self.pending_events.peek()) |event| {
            if (event.tick <= self.current_tick) {
                return self.pending_events.remove();
            }
        }
        return null;
    }
};

};

Deterministic PRNG

/// Deterministic random number generator /// Same seed = same sequence = reproducible bugs pub const PRNG = struct { state: u64,

pub fn init(seed: u64) PRNG {
    return .{ .state = seed };
}

/// Xorshift64* - fast, deterministic, good distribution
pub fn next(self: *PRNG) u64 {
    var x = self.state;
    x ^= x >> 12;
    x ^= x << 25;
    x ^= x >> 27;
    self.state = x;
    return x *% 0x2545F4914F6CDD1D;
}

pub fn boolean(self: *PRNG) bool {
    return self.next() & 1 == 0;
}

pub fn range(self: *PRNG, min: u64, max: u64) u64 {
    return min + (self.next() % (max - min + 1));
}

/// Weighted boolean - returns true with given probability (0.0-1.0)
pub fn chance(self: *PRNG, probability: f64) bool {
    const threshold = @as(u64, @intFromFloat(probability * @as(f64, std.math.maxInt(u64))));
    return self.next() < threshold;
}

};

Simulated Network

/// Simulated network with controllable failures pub const SimulatedNetwork = struct { prng: *PRNG, time: *Time.Simulated, partitions: std.AutoHashMap(NodePair, void), in_flight: std.ArrayList(Message), config: NetworkConfig,

pub const NetworkConfig = struct {
    /// Probability of message loss (0.0-1.0)
    message_loss_rate: f64 = 0.01,
    /// Probability of message duplication
    duplication_rate: f64 = 0.001,
    /// Probability of message reordering
    reorder_rate: f64 = 0.05,
    /// Min/max latency in ticks
    min_latency: u64 = 1,
    max_latency: u64 = 100,
    /// Probability of partition occurring
    partition_rate: f64 = 0.001,
    /// Average partition duration in ticks
    partition_duration: u64 = 1000,
};

pub fn send(self: *SimulatedNetwork, from: NodeId, to: NodeId, msg: Message) void {
    // Check for partition
    if (self.isPartitioned(from, to)) {
        // Message is silently dropped
        return;
    }
    
    // Random message loss
    if (self.prng.chance(self.config.message_loss_rate)) {
        return;
    }
    
    // Calculate delivery time
    const latency = self.prng.range(self.config.min_latency, self.config.max_latency);
    const delivery_tick = self.time.current_tick + latency;
    
    // Schedule delivery
    self.time.scheduleAt(delivery_tick, .{
        .kind = .message_delivery,
        .message = msg,
        .to = to,
    });
    
    // Possible duplication
    if (self.prng.chance(self.config.duplication_rate)) {
        const dup_latency = self.prng.range(self.config.min_latency, self.config.max_latency);
        self.time.scheduleAt(self.time.current_tick + latency + dup_latency, .{
            .kind = .message_delivery,
            .message = msg,
            .to = to,
        });
    }
}

pub fn injectPartition(self: *SimulatedNetwork, nodes: []const NodeId) void {
    for (nodes) |a| {
        for (nodes) |b| {
            if (a != b) {
                self.partitions.put(.{ .a = a, .b = b }, {}) catch unreachable;
            }
        }
    }
    
    // Schedule partition heal
    const duration = self.prng.range(
        self.config.partition_duration / 2,
        self.config.partition_duration * 2
    );
    self.time.scheduleAt(self.time.current_tick + duration, .{
        .kind = .heal_partition,
        .nodes = nodes,
    });
}

fn isPartitioned(self: *SimulatedNetwork, a: NodeId, b: NodeId) bool {
    return self.partitions.contains(.{ .a = a, .b = b }) or
           self.partitions.contains(.{ .a = b, .b = a });
}

};

Simulated Storage

/// Simulated storage with crash and corruption injection pub const SimulatedStorage = struct { prng: *PRNG, data: std.AutoHashMap(u64, Block), pending_writes: std.ArrayList(PendingWrite), config: StorageConfig,

pub const StorageConfig = struct {
    /// Probability of read error
    read_error_rate: f64 = 0.0001,
    /// Probability of write error  
    write_error_rate: f64 = 0.0001,
    /// Probability of bit flip (corruption)
    corruption_rate: f64 = 0.00001,
    /// Probability of losing pending writes on crash
    crash_lose_pending: f64 = 0.5,
    /// Simulate torn writes
    torn_write_enabled: bool = true,
};

pub fn read(self: *SimulatedStorage, block_id: u64) !Block {
    // Simulate read error
    if (self.prng.chance(self.config.read_error_rate)) {
        return error.ReadError;
    }
    
    const block = self.data.get(block_id) orelse return error.NotFound;
    
    // Simulate corruption
    if (self.prng.chance(self.config.corruption_rate)) {
        var corrupted = block;
        const bit_position = self.prng.range(0, block.data.len * 8 - 1);
        const byte_idx = bit_position / 8;
        const bit_idx: u3 = @intCast(bit_position % 8);
        corrupted.data[byte_idx] ^= (@as(u8, 1) << bit_idx);
        return corrupted;
    }
    
    return block;
}

pub fn write(self: *SimulatedStorage, block_id: u64, block: Block) !void {
    // Simulate write error
    if (self.prng.chance(self.config.write_error_rate)) {
        return error.WriteError;
    }
    
    // Add to pending (not yet durable)
    self.pending_writes.append(.{
        .block_id = block_id,
        .block = block,
    }) catch unreachable;
}

pub fn fsync(self: *SimulatedStorage) !void {
    // Move pending writes to durable storage
    for (self.pending_writes.items) |write| {
        self.data.put(write.block_id, write.block) catch unreachable;
    }
    self.pending_writes.clearRetainingCapacity();
}

pub fn simulateCrash(self: *SimulatedStorage) void {
    // Decide which pending writes survive
    var survivors = std.ArrayList(PendingWrite).init(self.pending_writes.allocator);
    
    for (self.pending_writes.items) |write| {
        if (!self.prng.chance(self.config.crash_lose_pending)) {
            // This write survived the crash
            if (self.config.torn_write_enabled and self.prng.chance(0.1)) {
                // Torn write - partial data written
                var torn = write;
                const tear_point = self.prng.range(0, torn.block.data.len);
                @memset(torn.block.data[tear_point..], 0);
                survivors.append(torn) catch unreachable;
            } else {
                survivors.append(write) catch unreachable;
            }
        }
    }
    
    // Apply survivors
    for (survivors.items) |write| {
        self.data.put(write.block_id, write.block) catch unreachable;
    }
    
    self.pending_writes.clearRetainingCapacity();
}

};

The Simulation Harness

/// Main simulation harness - runs millions of deterministic simulations pub const Simulator = struct { seed: u64, prng: PRNG, time: Time.Simulated, network: SimulatedNetwork, storage: []SimulatedStorage, nodes: []Node, invariant_checker: InvariantChecker, stats: SimulationStats,

pub fn init(seed: u64, node_count: usize) Simulator {
    var prng = PRNG.init(seed);
    var time = Time.Simulated{};
    
    return .{
        .seed = seed,
        .prng = prng,
        .time = time,
        .network = SimulatedNetwork.init(&prng, &time),
        .storage = allocateStorages(node_count, &prng),
        .nodes = allocateNodes(node_count),
        .invariant_checker = InvariantChecker.init(),
        .stats = .{},
    };
}

/// Run simulation for given number of ticks
pub fn run(self: *Simulator, max_ticks: u64) SimulationResult {
    while (self.time.current_tick < max_ticks) {
        // Process all events at current tick
        while (self.time.processNextEvent()) |event| {
            self.processEvent(event);
            
            // Check invariants after every state change
            if (self.invariant_checker.check(self.nodes)) |violation| {
                return .{
                    .outcome = .invariant_violation,
                    .seed = self.seed,
                    .tick = self.time.current_tick,
                    .violation = violation,
                };
            }
        }
        
        // Random fault injection
        self.maybeInjectFault();
        
        // Advance time
        self.time.advanceTo(self.time.current_tick + 1);
        self.stats.ticks_simulated += 1;
    }
    
    return .{
        .outcome = .success,
        .seed = self.seed,
        .tick = self.time.current_tick,
        .stats = self.stats,
    };
}

fn maybeInjectFault(self: *Simulator) void {
    // Network partition
    if (self.prng.chance(0.001)) {
        const partition_size = self.prng.range(1, self.nodes.len / 2);
        var partition = std.ArrayList(NodeId).init(allocator);
        // Randomly select nodes for partition
        for (0..partition_size) |_| {
            partition.append(@intCast(self.prng.range(0, self.nodes.len - 1))) catch {};
        }
        self.network.injectPartition(partition.items);
        self.stats.partitions_injected += 1;
    }
    
    // Node crash
    if (self.prng.chance(0.0001)) {
        const node_id = self.prng.range(0, self.nodes.len - 1);
        self.crashNode(@intCast(node_id));
        self.stats.crashes_injected += 1;
    }
    
    // Disk corruption
    if (self.prng.chance(0.00001)) {
        const node_id = self.prng.range(0, self.nodes.len - 1);
        // Corruption happens in storage simulation
        self.stats.corruptions_injected += 1;
    }
}

fn crashNode(self: *Simulator, node_id: NodeId) void {
    // Simulate crash: lose in-memory state, maybe lose pending writes
    self.storage[node_id].simulateCrash();
    self.nodes[node_id].restart();
}

fn processEvent(self: *Simulator, event: Event) void {
    switch (event.kind) {
        .message_delivery => {
            self.nodes[event.to].receiveMessage(event.message);
        },
        .timeout => {
            self.nodes[event.node].handleTimeout(event.timeout_id);
        },
        .heal_partition => {
            for (event.nodes) |a| {
                for (event.nodes) |b| {
                    self.network.partitions.remove(.{ .a = a, .b = b });
                }
            }
        },
    }
}

};

/// Run millions of simulations with different seeds pub fn runSimulationCampaign(config: CampaignConfig) CampaignResult { var results = CampaignResult{};

var seed: u64 = config.starting_seed;
while (seed < config.starting_seed + config.num_simulations) : (seed += 1) {
    var sim = Simulator.init(seed, config.node_count);
    const result = sim.run(config.ticks_per_simulation);
    
    if (result.outcome != .success) {
        results.failures.append(.{
            .seed = seed,
            .tick = result.tick,
            .violation = result.violation,
        }) catch {};
    }
    
    results.total_ticks += result.tick;
    results.simulations_run += 1;
    
    if (results.simulations_run % 10000 == 0) {
        std.debug.print("Completed {} simulations, {} failures\n", .{
            results.simulations_run,
            results.failures.items.len,
        });
    }
}

return results;

}

Invariant Checking

/// Invariant checker - validates system correctness after every operation pub const InvariantChecker = struct { /// All invariants that must hold pub fn check(self: *InvariantChecker, nodes: []Node) ?Violation { // Invariant 1: Consensus safety - no two nodes disagree on committed values if (self.checkConsensusSafety(nodes)) |v| return v;

    // Invariant 2: Replication - committed data exists on quorum
    if (self.checkReplication(nodes)) |v| return v;
    
    // Invariant 3: Monotonicity - committed log only grows
    if (self.checkMonotonicity(nodes)) |v| return v;
    
    // Invariant 4: Linearizability - operations appear atomic
    if (self.checkLinearizability(nodes)) |v| return v;
    
    return null;
}

fn checkConsensusSafety(self: *InvariantChecker, nodes: []Node) ?Violation {
    // For each log index, all nodes must agree on committed value
    var max_commit_index: u64 = 0;
    for (nodes) |node| {
        max_commit_index = @max(max_commit_index, node.commit_index);
    }
    
    for (0..max_commit_index) |index| {
        var reference_value: ?Entry = null;
        
        for (nodes, 0..) |node, node_id| {
            if (node.commit_index > index) {
                const entry = node.log.get(index);
                
                if (reference_value) |ref| {
                    if (!std.mem.eql(u8, &entry.data, &ref.data)) {
                        return Violation{
                            .kind = .consensus_disagreement,
                            .message = "Nodes disagree on committed value",
                            .index = index,
                            .nodes = .{ node_id, 0 },  // First disagreement
                        };
                    }
                } else {
                    reference_value = entry;
                }
            }
        }
    }
    
    return null;
}

fn checkLinearizability(self: *InvariantChecker, nodes: []Node) ?Violation {
    // Track operation history and verify linearizability
    // This requires a history of all operations and their results
    // Using a linearizability checker like Jepsen's Knossos
    
    const history = self.buildOperationHistory(nodes);
    
    if (!isLinearizable(history)) {
        return Violation{
            .kind = .linearizability,
            .message = "Operation history is not linearizable",
            .history = history,
        };
    }
    
    return null;
}

};

Time Compression Demonstration

Conceptual demonstration of time compression value

def time_compression_analysis(): """ Why deterministic simulation is so powerful. """

# Real-world testing
real_time = {
    'ticks_per_second': 1,
    'test_hours_per_day': 8,
    'test_days_per_year': 250,
    'ticks_per_year': 8 * 3600 * 250,  # 7.2M ticks/year
}

# Deterministic simulation (TigerBeetle achieves ~1M ticks/second)
simulation = {
    'ticks_per_second': 1_000_000,
    'seconds_to_simulate_year': real_time['ticks_per_year'] / 1_000_000,
    # 7.2 seconds to simulate a year of real testing
}

# Bug that occurs once per million operations
rare_bug = {
    'occurrence_rate': 1 / 1_000_000,
    'real_time_to_find': '~139 days of continuous testing',
    'simulation_time_to_find': '~1 second',
}

# Run 1000 different seeds
coverage = {
    'seeds': 1000,
    'ticks_per_seed': 10_000_000,
    'total_ticks': 10_000_000_000,
    'real_time_equivalent': '1,389 years of testing',
    'simulation_time': '~3 hours',
}

return {
    'compression_ratio': simulation['ticks_per_second'] / real_time['ticks_per_second'],
    'years_simulated_per_hour': 500,  # Approximate
    'reproducibility': 'Perfect - same seed = same execution',
}

Mental Model

TigerBeetle approaches testing by asking:

• Can I reproduce this? If not, make it deterministic

• Am I testing time? Compress it—simulate years in seconds

• Am I testing failure? Inject it—don't wait for it

• Am I checking invariants? Every operation, not just at the end

• How many scenarios? Millions, not hundreds

The Simulation Checklist

□ All non-determinism abstracted (time, network, disk, random) □ PRNG seeded and reproducible □ Time is injectable and compressible □ Network failures simulated (partition, loss, reorder, duplicate) □ Disk failures simulated (crash, corruption, torn writes) □ Invariants checked after every state transition □ Simulation faster than real-time (1M+ ticks/sec target) □ Millions of seeds tested, not dozens □ Any failure reproducible with just the seed □ Bug report includes seed for exact reproduction

Signature TigerBeetle Moves

• VOPR (Viewstamped Operation Replication tester)

• Time compression (years in minutes)

• Deterministic single-threaded simulation

• Seed-based perfect reproducibility

• Continuous invariant checking

• Simulated network partitions and message loss

• Crash recovery testing with torn writes

• Million-seed simulation campaigns