Unit 07 · Chapter 4 · 10 min read

Experiments, causal effects, and risk tradeoffs

Measure whether a control improves outcomes rather than only changing them.

The new rule launches on Monday. Loss falls on Tuesday. On the same day, the largest risky merchant leaves the platform. The chart shows a change. It does not yet show what caused it.

Define the intervention and estimand

An experiment needs a specific intervention and a quantity it aims to estimate. For example, compare two permitted authentication flows on completion, confirmed loss, and customer support cost. The estimand states whose outcomes, over what period, under which assignment.

Keep mandatory legal controls outside an experiment that would bypass them. Use an approved test population and guardrails. The goal is to learn within permitted activity. A vague objective such as improve risk cannot specify a sample, a stopping rule, or a useful result.

An intervention is the change applied to the system. An estimand is the particular effect the analysis intends to measure. “Does the new rule help” is incomplete until help has a unit, population, time horizon, and outcome. The effect on all eligible accounts can differ from the effect on accounts that actually received a challenge, because challenge receipt itself depends on the policy and customer behavior.

Write the analysis contract before reading the result. Define assignment, exclusions, outcome maturity, primary measures, and material customer or operational limits. Otherwise, a team can unintentionally select the time window or subgroup that makes a weak result look convincing. A clear contract also makes an inconclusive result useful by showing which uncertainty remains.

Define the intervention and estimand Define the intervention and estimand — the flow Follow the sequence. Specify the effect and observation window. Intervention Name the changed control Population Define eligible participants Outcome Specify the effect and observation window — Follow the sequence. Specify the effect and observation window. Chapter sources · Open image

Define the intervention and estimand Define the intervention and estimand — the distinction These concepts answer different questions. Read each definition in the context of the section. Observed difference Groups have different outcomes Causal effect Difference attributable to the intervention under the design — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Define the intervention and estimand Experiment contract Fictional teaching record. Customer and financial protection. Experiment contract Illustrative data; not a real customer record or a prescribed policy. Change clearer step-up prompt Permitted intervention Primary outcome completed legitimate purchases Defined measure Guardrail confirmed loss and complaints Customer and financial protection The experiment needs a clear success definition — Fictional teaching record. Customer and financial protection. Chapter sources · Open image

Define the intervention and estimand Define the intervention and estimand — control and failure modes The experiment needs a clear success definition. The branches show why alternative designs fail. Control design Specify the effect and guardrails before launch. The experiment needs a clear success definition. Failure mode 1 Disable mandatory controls to learn faster. That exceeds permitted experimentation. avoid Failure mode 2 Change many unrelated things at once. Attribution becomes difficult. avoid Failure mode 3 Choose the metric after seeing results. That increases selection bias. avoid — The experiment needs a clear success definition. The branches show why alternative designs fail. Chapter sources · Open image

Randomize at the right level

Randomization helps balance confounders on average, but the assignment unit matters. If one customer sees both treatments across retries, behavior can spill across groups. Merchant-level interventions may require merchant-level assignment.

Use a stable assignment key and preserve it through the relevant lifecycle. Check sample-ratio mismatch and implementation errors. Randomization does not guarantee balance in every small sample, nor does it solve missing outcomes. Document exclusions and whether they were decided before or after treatment, because post-treatment exclusions can bias the estimate.

Randomization must account for interference. If one customer can make many payments, assigning each payment independently can expose that customer to both policies and change later behavior. Shared merchants, devices, or counterparties can also connect observations. Choose an assignment unit that fits the intervention and use an analysis that respects the resulting dependence. Larger assignment groups may reduce the number of independent observations, which changes uncertainty even when the raw transaction count is large.

Randomize at the right level Randomize at the right level — the flow Follow the sequence. Verify balance and actual exposure. Unit Choose account merchant or transaction assignment Assign Use a stable randomized rule Check Verify balance and actual exposure — Follow the sequence. Verify balance and actual exposure. Chapter sources · Open image

Randomize at the right level Randomize at the right level — the distinction These concepts answer different questions. Read each definition in the context of the section. Assignment Intended treatment group Exposure Treatment the participant actually received — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Randomize at the right level Assignment defect Fictional teaching record. Avoid cross-treatment contamination. Assignment defect Illustrative data; not a real customer record or a prescribed policy. Customer three retries Same decision journey Groups A then B then A Inconsistent exposure Fix stable journey assignment Avoid cross-treatment contamination Retries and spillovers can contaminate the comparison — Fictional teaching record. Avoid cross-treatment contamination. Chapter sources · Open image

Randomize at the right level Randomize at the right level — control and failure modes Retries and spillovers can contaminate the comparison. The branches show why alternative designs fail. Control design Keep assignment stable at the relevant unit. Retries and spillovers can contaminate the comparison. Failure mode 1 Randomize every page render. One user can receive mixed treatments. avoid Failure mode 2 Ignore sample-ratio mismatch. It can reveal implementation defects. avoid Failure mode 3 Exclude inconvenient outcomes after treatment. That can bias the result. avoid — Retries and spillovers can contaminate the comparison. The branches show why alternative designs fail. Chapter sources · Open image

Wait for mature outcomes

Risk outcomes often arrive after the customer interaction. Conversion can be measured quickly; disputes, defaults, and recoveries take longer. A test can show a short-term benefit before its loss cost becomes visible.

Define leading and final outcomes with separate reporting. Compare groups at equal maturity and keep uncertainty visible. Avoid stopping as soon as one metric looks favorable. Sequential monitoring requires an appropriate analysis plan if repeated looks influence the decision. The operational team also needs stop conditions for clear harm, independent of the final statistical conclusion.

Wait for mature outcomes Wait for mature outcomes — the flow Follow the sequence. Use the planned analysis and harm guardrails. Immediate Observe completion and latency Delayed Wait for comparable loss maturity Decide Use the planned analysis and harm guardrails — Follow the sequence. Use the planned analysis and harm guardrails. Chapter sources · Open image

Wait for mature outcomes Wait for mature outcomes — the distinction These concepts answer different questions. Read each definition in the context of the section. Leading metric Early signal of behavior Mature outcome Result after the relevant observation window — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Wait for mature outcomes Experiment timeline Fictional teaching record. Not a final loss conclusion. Experiment timeline Illustrative data; not a real customer record or a prescribed policy. Day 1 conversion improves Early result Day 30 disputes still developing Incomplete cost Decision provisional evidence Not a final loss conclusion Fast conversion data cannot establish final net benefit — Fictional teaching record. Not a final loss conclusion. Chapter sources · Open image

Wait for mature outcomes Wait for mature outcomes — control and failure modes Fast conversion data cannot establish final net benefit. The branches show why alternative designs fail. Control design Separate early signals from mature outcomes. Fast conversion data cannot establish final net benefit. Failure mode 1 Stop at the first favorable chart. Repeated peeking can distort inference. avoid Failure mode 2 Compare unequal follow-up periods. Groups have different opportunity for loss. avoid Failure mode 3 Ignore clear harm while waiting. Operational guardrails still apply. avoid — Fast conversion data cannot establish final net benefit. The branches show why alternative designs fail. Chapter sources · Open image

Account for selection and counterfactuals

A declined payment does not reveal whether it would have become fraud if approved. A reviewed customer may behave differently because of the review. Observed labels are shaped by the policy that produced them.

Use randomized evidence where appropriate, carefully designed observational analysis where necessary, and explicit uncertainty where the counterfactual is unavailable. Do not label every decline a prevented loss. In a teaching example, 1,000 declines with a model estimate of 5 percent fraud are not 1,000 confirmed fraud events. Even the 50-event estimate depends on model validity and population assumptions.

Account for selection and counterfactuals Account for selection and counterfactuals — the flow Follow the sequence. Use a justified method and state uncertainty. Policy Determines which outcomes become observable Counterfactual Outcome under an alternative action Estimate Use a justified method and state uncertainty — Follow the sequence. Use a justified method and state uncertainty. Chapter sources · Open image

Account for selection and counterfactuals Account for selection and counterfactuals — the distinction These concepts answer different questions. Read each definition in the context of the section. Declined amount Value of blocked attempts Prevented loss Estimated or evidenced loss avoided by the control — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Account for selection and counterfactuals Counterfactual example Fictional teaching record. Not equal to all declines. Counterfactual example Illustrative data; not a real customer record or a prescribed policy. Declines 1000 Observed actions Estimated fraud 5 percent Model assumption Confirmed prevented events unknown Not equal to all declines The rejected outcome is often unobserved — Fictional teaching record. Not equal to all declines. Chapter sources · Open image

Account for selection and counterfactuals Account for selection and counterfactuals — control and failure modes The rejected outcome is often unobserved. The branches show why alternative designs fail. Control design Distinguish observed actions from estimated avoided loss. The rejected outcome is often unobserved. Failure mode 1 Count every decline as fraud stopped. That overstates effectiveness. avoid Failure mode 2 Treat model estimates as confirmed labels. They remain estimates. avoid Failure mode 3 Ignore policy-driven selection. It shapes the available training data. avoid — The rejected outcome is often unobserved. The branches show why alternative designs fail. Chapter sources · Open image

Report net effect and uncertainty

A useful experiment report includes the intervention, population, assignment, sample, duration, outcomes, uncertainty, guardrails, and limitations. Show both the main effect and important segments without turning every noisy subgroup into a firm conclusion.

Translate the result into the product decision. A small approval gain can be valuable if loss and support costs remain acceptable; a large gain may be unacceptable if it creates harm or violates constraints. Keep the decision owner and review date explicit. The report should allow a later reader to understand why the change was adopted or rejected.

Report net effect and uncertainty Report net effect and uncertainty — the flow Follow the sequence. Record the product action and rationale. Report Show design outcomes and uncertainty Interpret Connect effects to costs and constraints Decide Record the product action and rationale — Follow the sequence. Record the product action and rationale. Chapter sources · Open image

Report net effect and uncertainty Report net effect and uncertainty — the distinction These concepts answer different questions. Read each definition in the context of the section. Statistical signal Evidence of a measured difference Business decision Choice considering magnitude cost harm and constraints — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Report net effect and uncertainty Net-effect example Fictional teaching record. Before uncertainty and omitted costs. Net-effect example Illustrative data; not a real customer record or a prescribed policy. Contribution gain 12000 USD Illustrative benefit Added loss and support 9000 USD Defined costs Net 3000 USD Before uncertainty and omitted costs A significant result is not automatically a good decision — Fictional teaching record. Before uncertainty and omitted costs. Chapter sources · Open image

Report net effect and uncertainty Report net effect and uncertainty — control and failure modes A significant result is not automatically a good decision. The branches show why alternative designs fail. Control design Report magnitude costs and uncertainty together. A significant result is not automatically a good decision. Failure mode 1 Publish only the winning metric. Tradeoffs disappear. avoid Failure mode 2 Treat every subgroup fluctuation as real. Small samples can be noisy. avoid Failure mode 3 Omit the tested population. Readers may generalize beyond the evidence. avoid — A significant result is not automatically a good decision. The branches show why alternative designs fail. Chapter sources · Open image

Chapter connections

This chapter builds on Risk models, calibration, and delayed outcomes. Continue with Model governance and AI-assisted risk work to follow the next part of the system. Use the glossary for terminology and risk mathematics for formulas and worked calculations.

Sources

Reviewed 2026-09-17