Decision engines, rules, and reliable execution
Turn evidence into one controlled action under time pressure.
Two services both approve the same withdrawal. Each saw enough available balance. Together, they sent too much. A risk decision must be correct in the presence of concurrency, retries, and partial failure, not only in a clean unit test.
Separate constraints scores and actions
A decision engine should distinguish mandatory constraints, risk estimates, and product actions. A confirmed prohibition cannot be outweighed by a favorable model score. Missing required evidence is also a state that needs an explicit response.
Use a clear precedence order and record every relevant rule result. The following pseudocode illustrates structure, not a deployable legal policy. The actual checks and responses need approved definitions.
if required_control_unavailable(context):
return approved_contingency(context)
if prohibited_under_applicable_policy(context):
return required_disposition(context)
return choose_product_action(score(context), context)
- ConstraintsApply required boundaries
- EstimateCompute permitted risk signals
- ActionChoose the approved product response
- Mandatory constraint
- Cannot be offset by expected profit
- Risk estimate
- Informs a choice within permitted activity
Decision precedence
Illustrative data; not a real customer record or a prescribed policy.
- Sanctions dispositionrequired hold
Binding policy result
- Fraud scorelow
Separate estimate
- Final actionrequired hold
No score override
Different decision types carry different authority
Make precedence explicit and auditable. Different decision types carry different authority.
- Failure mode 1avoid
- Average all rules into one score. Mandatory constraints can disappear.
- Failure mode 2avoid
- Treat missing screening as low risk. Unavailable evidence is not clearance.
- Failure mode 3avoid
- Return only a number. The product still needs an action.
Reserve capacity atomically
A limit or balance check must remain valid when multiple requests arrive together. Use database transactions, unique constraints, or another reviewed concurrency design to reserve the resource once. A separate read followed by an unprotected write can overspend capacity.
The teaching SQL below performs a conditional decrement and returns a row only when capacity was reserved. A real system also needs a unique operation key, an immutable journal, expiry handling, and reconciliation.
UPDATE risk_capacity
SET available_minor = available_minor - :amount_minor
WHERE account_id = :account_id
AND available_minor >= :amount_minor
RETURNING available_minor;
Suppose an account has $100 of remaining capacity and two $80 requests arrive together. Each request can read the same available amount and independently pass a correct-looking limit check. The combined result exceeds the limit. The safety property belongs to the shared state transition, not to the arithmetic in either request.
Use a concurrency strategy appropriate to the datastore and service boundary, such as a transactional conditional update with defined conflict handling. The reservation needs an identifier, amount, currency, status, and expiry or release rule. A failed downstream action must not leave capacity reserved forever. A late success must not consume capacity that was released and used elsewhere without a defined resolution path. These are lifecycle cases that deserve explicit engineering treatment.
- Check and reserveUse one atomic operation
- CommitLink the reservation to the action
- ReleaseReturn capacity if the action does not proceed
- Unprotected read
- Can become stale before the write
- Atomic reservation
- Enforces the condition at the update boundary
Concurrency example
Illustrative data; not a real customer record or a prescribed policy.
- Available100000 minor units
Shared capacity
- Two requests80000 each
Compete for the same funds
- Allowedone request
Second lacks remaining capacity
Concurrent reads can both look safe
Enforce shared limits at the transactional boundary. Concurrent reads can both look safe.
- Failure mode 1avoid
- Trust a cached balance indefinitely. It may not reflect other reservations.
- Failure mode 2avoid
- Retry with a new operation key. That can reserve twice.
- Failure mode 3avoid
- Forget failed reservations. Capacity can remain unavailable without cause.
Budget latency and failure modes
The decision path has a time budget. Identify required dependencies, optional enrichment, timeout behavior, and the point at which an action can no longer wait. A slow optional feature should not silently become a mandatory outage for every payment.
Use bounded timeouts and explicit fallback reasons. Mandatory legal controls require an approved contingency rather than an automatic fail-open. Optional fraud enrichment may have a different risk-based fallback. Test the actual combination of failures. Several dependencies that each meet a latency target can still exceed the total budget when called sequentially.
Failure behavior should follow the action’s consequence and the applicable constraint. A missing optional signal may justify a reduced limit or another evidence request. An unavailable mandatory control may require the affected action to wait. One global fail-open switch cannot express these differences. Document dependency-specific behavior, latency budgets, and the owner who can authorize a change. Test the customer-visible outcome as well as the service response: an HTTP success is not evidence that the intended protection occurred.
- BudgetAllocate time across the decision path
- BoundUse dependency-specific timeouts
- FallbackRecord the approved degraded action
- Required dependency
- Its absence changes permitted processing
- Optional enrichment
- May have a bounded alternative treatment
Latency budget
Illustrative data; not a real customer record or a prescribed policy.
- Total200 milliseconds
Illustrative service target
- Required check80 milliseconds
Allocated portion
- Optional enrichment40 milliseconds
Bounded timeout
Not every failure permits the same response
Define dependency-specific contingencies. Not every failure permits the same response.
- Failure mode 1avoid
- Fail open for all timeouts. Required controls may be bypassed.
- Failure mode 2avoid
- Wait without a bound. The product can stall indefinitely.
- Failure mode 3avoid
- Add sequential targets without measuring total. Combined latency may exceed the promise.
Make decisions replayable
A replay reconstructs the decision from the evidence and versions available at the time. Record policy version, model version, feature definition, input references, action, reasons, and relevant dependency states. A current model score does not explain a historical action.
Replay should not execute financial side effects. Separate pure evaluation from release, posting, and notification. Use dry-run outputs in a restricted analysis environment. Compare historical and candidate policies to understand changes, while recognizing that counterfactual outcomes for previously rejected transactions may remain unknown.
- CapturePreserve evidence and versions
- EvaluateRecompute without side effects
- CompareExplain differences and unknown outcomes
- Decision replay
- Reconstructs a historical evaluation
- Payment replay
- Repeats an external action and can move value
Replay contract
Illustrative data; not a real customer record or a prescribed policy.
- Modeevaluation only
No payout calls
- Policyhistorical v9
Original version
- Outputaction comparison
Analysis artifact
Replay must not repeat money movement
Separate evaluation from side effects. Replay must not repeat money movement.
- Failure mode 1avoid
- Use the current policy for historical explanation. That can change the result.
- Failure mode 2avoid
- Call payment APIs during analysis. It can create real duplicate actions.
- Failure mode 3avoid
- Treat counterfactual approval as observed repayment. The outcome may never have occurred.
Release rules with rollback evidence
A policy release can affect many customers immediately. Use versioned configuration, review, controlled rollout, monitoring, and an executable rollback. Shadow evaluation can compare decisions without applying the candidate action, but it does not reveal all behavioral outcomes.
Define stop conditions for error, customer harm, coverage, and latency. Reconcile which decisions used each version. If a rollback occurs, identify any actions already taken under the candidate policy and determine whether remediation is needed. Restoring the old configuration does not undo past declines or transfers.
- PreviewCompare candidate decisions safely
- Roll outLimit exposure and observe guardrails
- RollbackRestore and assess affected actions
- Configuration rollback
- Future evaluations use the old version
- Customer remediation
- Addresses actions already taken
Rule release
Illustrative data; not a real customer record or a prescribed policy.
- Candidatev14
New policy
- Affected decisions600
Observed rollout population
- Rollbackcomplete
Past actions still need impact review
Changing configuration does not undo prior effects
Pair rollback with impact assessment. Changing configuration does not undo prior effects.
- Failure mode 1avoid
- Release globally without version tracking. The affected population becomes hard to find.
- Failure mode 2avoid
- Treat shadow results as full outcome proof. Customers did not experience candidate actions.
- Failure mode 3avoid
- Stop monitoring after deployment. Behavior and data can change.
Chapter connections
This chapter builds on Risk data contracts and event time. Continue with Risk models, calibration, and delayed outcomes to follow the next part of the system. Use the glossary for terminology and risk mathematics for formulas and worked calculations.