Distributed Systems Basics
Overview
Use this skill to reason about correctness and reliability in systems where network faults and partial failures are normal.
Scope Boundaries
- Multi-service workflows require explicit consistency and ordering guarantees.
- Retry/timeout/duplicate-message behavior can change business correctness.
- Teams need to define reliability primitives before implementation or rollout.
Shared References
- Failure mode and consistency rules:
references/failure-mode-consistency-rules.md
Templates And Assets
- Distributed flow risk template:
assets/distributed-flow-risk-template.md
Inputs To Gather
- Component boundaries and communication patterns.
- Consistency and ordering requirements per workflow.
- Failure scenarios (partition, timeout, duplicate, out-of-order, stale read).
- Recovery and observability capabilities.
Deliverables
- Failure-mode map and risk ranking.
- Consistency decision record per critical flow.
- Reliability mechanism selection (retry, idempotency, backoff, timeout).
- Validation plan (fault injection and invariant checks).
Workflow
- Capture critical flows with
assets/distributed-flow-risk-template.md. - Map failure assumptions and consistency requirements per flow.
- Select reliability primitives using
references/failure-mode-consistency-rules.md. - Define observability and recovery behavior.
- Validate assumptions with targeted failure tests and invariant checks.
Quality Standard
- Critical flows have explicit consistency and ordering rules.
- Retry/timeout semantics are bounded and intentional.
- Idempotency strategy exists where at-least-once delivery is possible.
- Failure handling is observable and testable.
Failure Conditions
- Stop when consistency assumptions are implicit or contradictory.
- Stop when retries/timeouts can amplify failure unboundedly.
- Escalate when critical failure modes have no mitigation path.