Unit 08 · Chapter 3 · 9 min read

Operational resilience and failure design

Keep critical services correct when components fail.

The fraud vendor is slow, the queue is growing, and every client retries. More traffic arrives because the system is struggling with the traffic it already has. Resilience means designing the failure path before this loop begins.

Define the critical service

Start from the customer service that must continue or recover: receive a report, protect a balance, make an authorized payout, or preserve evidence. Map the systems, people, partners, and data needed to deliver it.

A service-level objective measures a chosen reliability target. It should include correctness and coverage where they matter, not just uptime. A payment API that returns success quickly while dropping ledger events is not delivering a reliable payment service. Define the observable good event and the acceptable failure budget with the responsible owners.

Define the critical service Define the critical service — the flow Follow the sequence. Measure successful delivery of that outcome. Service Name the customer or control outcome Dependencies Map what it requires Objective Measure successful delivery of that outcome — Follow the sequence. Measure successful delivery of that outcome. Chapter sources · Open image

Define the critical service Define the critical service — the distinction These concepts answer different questions. Read each definition in the context of the section. Availability Endpoint responds Correct service Required action and records are completed accurately — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Define the critical service Service objective Fictional teaching record. Response speed is insufficient. Service objective Illustrative data; not a real customer record or a prescribed policy. Endpoint responding Availability evidence Ledger event missing Correctness failure Customer outcome unproven Response speed is insufficient Correctness and coverage are part of reliability — Fictional teaching record. Response speed is insufficient. Chapter sources · Open image

Define the critical service Define the critical service — control and failure modes Correctness and coverage are part of reliability. The branches show why alternative designs fail. Control design Measure the critical outcome beyond uptime. Correctness and coverage are part of reliability. Failure mode 1 Use only HTTP success counts. The financial action may still fail. avoid Failure mode 2 Ignore people and partners. They are dependencies too. avoid Failure mode 3 Set a target without an owner. Breaches may receive no decision. avoid — Correctness and coverage are part of reliability. The branches show why alternative designs fail. Chapter sources · Open image

Control retries and overload

Retries can help recover transient failures but can also amplify load. Use bounded attempts, backoff, jitter, and idempotent operation keys where appropriate. Coordinate retry behavior across layers so a client, gateway, worker, and vendor do not multiply attempts unexpectedly.

Apply backpressure and admission control to protect critical work. Rejecting or deferring new optional work can be safer than allowing every queue to grow without bound. Preserve accepted financial obligations and give clients an accurate status. Overload handling must not turn unknown outcomes into automatic new payments.

Retries are useful when failure is temporary, but they can amplify an outage. If every client immediately repeats a timed-out request, the recovering service receives extra work before it can clear the original demand. Use bounded retries, appropriate backoff, and a defined response when the outcome is unknown. Financial requests also need the idempotency and reconciliation controls described earlier in the book.

Protect the work that must continue. Admission controls, queue limits, and priority handling can prevent low-value traffic from consuming capacity required for critical operations. A queue needs a size and age policy; otherwise it can preserve requests until they are too old to be safe or useful. Recovery includes deciding which delayed work remains valid.

Control retries and overload Control retries and overload — the flow Follow the sequence. Prioritize accepted critical obligations. Bound Limit attempts and waiting work Spread Use backoff and jitter Protect Prioritize accepted critical obligations — Follow the sequence. Prioritize accepted critical obligations. Chapter sources · Open image

Control retries and overload Control retries and overload — the distinction These concepts answer different questions. Read each definition in the context of the section. Retry recovery Another attempt after a transient failure Retry storm Repeated attempts amplify the outage — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Control retries and overload Retry multiplication Fictional teaching record. Before other retrying layers. Retry multiplication Illustrative data; not a real customer record or a prescribed policy. Client attempts 3 Outer retry layer Worker attempts 3 Inner retry layer Potential calls 9 Before other retrying layers Independent policies can multiply load — Fictional teaching record. Before other retrying layers. Chapter sources · Open image

Control retries and overload Control retries and overload — control and failure modes Independent policies can multiply load. The branches show why alternative designs fail. Control design Coordinate bounded retries across layers. Independent policies can multiply load. Failure mode 1 Retry immediately forever. That can sustain the outage. avoid Failure mode 2 Generate a new payment key each time. Retries can create duplicate effects. avoid Failure mode 3 Accept unlimited work without capacity. Backlogs and deadlines can become unmanageable. avoid — Independent policies can multiply load. The branches show why alternative designs fail. Chapter sources · Open image

Use a durable handoff pattern

A database update and a message publish can fail between the two operations. A transactional outbox records the business change and the event to publish in the same database transaction. A worker later delivers the event and tracks progress.

The pattern still requires idempotent consumers because delivery can repeat. Monitor unpublished events and reconcile them with downstream effects. Do not claim universal exactly-once processing from one outbox table. The useful contract is explicit: durable recording, retryable delivery, and a consumer that prevents duplicate business effects at its boundary.

The transactional outbox pattern records a business state change and a pending message in the same local database transaction. A separate publisher delivers the message and records progress. This reduces the gap in which a state change commits but its notification is lost. Delivery can still repeat, so consumers need their own duplicate handling. The pattern provides a durable handoff, not a universal promise of exactly-once effects across every external system. Reconciliation remains necessary at the financial boundary.

Use a durable handoff pattern Use a durable handoff pattern — the flow Follow the sequence. Apply each business effect idempotently. Transact Write business state and outbox together Deliver Publish pending events with retries Consume Apply each business effect idempotently — Follow the sequence. Apply each business effect idempotently. Chapter sources · Open image

Use a durable handoff pattern Use a durable handoff pattern — the distinction These concepts answer different questions. Read each definition in the context of the section. Durable event Recorded for later delivery Single business effect Consumer prevents duplicate application — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Use a durable handoff pattern Outbox failure case Fictional teaching record. Consumer handles duplicates. Outbox failure case Illustrative data; not a real customer record or a prescribed policy. Database committed Business state and event saved Publisher crashed Delivery incomplete Recovery retry pending event Consumer handles duplicates At-least-once delivery can repeat — Fictional teaching record. Consumer handles duplicates. Chapter sources · Open image

Use a durable handoff pattern Use a durable handoff pattern — control and failure modes At-least-once delivery can repeat. The branches show why alternative designs fail. Control design Combine durable handoff with idempotent consumption. At-least-once delivery can repeat. Failure mode 1 Publish then forget the database update. The two states can diverge. avoid Failure mode 2 Claim exactly once without boundary definitions. The guarantee may not span all systems. avoid Failure mode 3 Ignore outbox age. Undelivered obligations can accumulate. avoid — At-least-once delivery can repeat. The branches show why alternative designs fail. Chapter sources · Open image

Test restoration and data integrity

A backup is useful only if it can restore the required service and records. Test restoration with the actual dependencies, keys, schemas, and access controls. Define recovery time and recovery point objectives for the critical service.

After restoration, reconcile ledger state, event streams, decisions, and pending work. Restoring a database snapshot can lose later events or replay earlier ones unless the recovery design accounts for them. Use a controlled exercise with explicit expected results. A successful file restore does not prove the customer balances are correct.

Test restoration and data integrity Test restoration and data integrity — the flow Follow the sequence. Confirm the critical service outcome. Restore Recover data and required dependencies Reconcile Check events balances and pending work Verify Confirm the critical service outcome — Follow the sequence. Confirm the critical service outcome. Chapter sources · Open image

Test restoration and data integrity Test restoration and data integrity — the distinction These concepts answer different questions. Read each definition in the context of the section. Recovery time objective Target time to restore service Recovery point objective Target tolerance for lost data history — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Test restoration and data integrity Restore exercise Fictional teaching record. File restore alone is incomplete. Restore exercise Illustrative data; not a real customer record or a prescribed policy. Snapshot 02:00 Recovered base state Last valid event 02:17 Additional history needed Reconciliation required File restore alone is incomplete Readable files do not prove correct balances — Fictional teaching record. File restore alone is incomplete. Chapter sources · Open image

Test restoration and data integrity Test restoration and data integrity — control and failure modes Readable files do not prove correct balances. The branches show why alternative designs fail. Control design Verify restored business invariants. Readable files do not prove correct balances. Failure mode 1 Test backups without keys. Encrypted data may be unusable. avoid Failure mode 2 Ignore events after the snapshot. Accepted obligations can disappear. avoid Failure mode 3 Declare recovery at server startup. The service and records still need checks. avoid — Readable files do not prove correct balances. The branches show why alternative designs fail. Chapter sources · Open image

Exercise realistic failure combinations

Real incidents often combine failures: a vendor outage, rising retries, an unavailable approver, and a backlog. Test combinations that threaten the critical service. Use safe synthetic traffic and controlled fault injection in an appropriate environment.

Define stop conditions, expected containment, and evidence to collect. Include operations and support in the exercise. The result should produce concrete repairs and a retest, not only a meeting note. A resilience exercise is successful when it reveals and closes weaknesses without creating uncontrolled customer impact.

Exercise realistic failure combinations Exercise realistic failure combinations — the flow Follow the sequence. Retest the specific weakness found. Scenario Combine plausible dependent failures Exercise Observe containment and recovery safely Repair Retest the specific weakness found — Follow the sequence. Retest the specific weakness found. Chapter sources · Open image

Exercise realistic failure combinations Exercise realistic failure combinations — the distinction These concepts answer different questions. Read each definition in the context of the section. Component test One dependency fails alone Service exercise Several failures affect the customer outcome — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Exercise realistic failure combinations Exercise scenario Fictional teaching record. Human dependency. Exercise scenario Illustrative data; not a real customer record or a prescribed policy. Vendor timeout Primary failure Queue near capacity Compounding condition Approver unavailable Human dependency Technical and human dependencies interact — Fictional teaching record. Human dependency. Chapter sources · Open image

Exercise realistic failure combinations — control and failure modes

Control design

Test the full operating response. Technical and human dependencies interact.

Failure mode 1avoid: Simulate only easy isolated failures. Combined weaknesses remain hidden.
Failure mode 2avoid: Run uncontrolled faults on live customer funds. The exercise needs a bounded safe environment.
Failure mode 3avoid: Close with a meeting summary only. Repairs require verification.

Technical and human dependencies interact. The branches show why alternative designs fail. Chapter sources · Open image

Chapter connections

This chapter builds on Treasury, liquidity, and settlement operations. Continue with Risk incidents, containment, and learning to follow the next part of the system. Use the glossary for terminology and risk mathematics for formulas and worked calculations.

Sources

Reviewed 2026-09-17