Risk data contracts and event time

An event contract gives fields stable meaning. Include event_id, entity references, event_type, occurred_at, received_at, schema_version, and explicit money units. A payment amount without a currency is incomplete. A timestamp without a documented time basis is difficult to compare.

Validate at ingestion and preserve rejected records in a controlled repair path. Do not silently coerce invalid amounts or unknown event types into safe defaults. Additive schema changes still need compatibility tests when downstream consumers assume a complete field set. The contract is shared by producers and consumers, not owned only by the data warehouse.

Define an event contract — the flow

1. 01ProduceEmit a versioned event
2. 02ValidateCheck types units and required meaning
3. 03ConsumeUse the documented contract

Define an event contract — the distinction

Occurred time

• When the underlying event happened

Received time

• When this system learned about it

Event specimen

Fictional educational excerpt / Not for execution

Event specimen

Illustrative data; not a real customer record or a prescribed policy.

1. 1

   amount_minor12500

   125.00 USD in this example

2. 2

   currencyUSD

   Explicit unit

3. 3

   schema_version3

   Consumer contract version

Safe-looking defaults can distort risk

Define an event contract — control and failure modes

Control design

Reject or repair invalid semantics explicitly. Safe-looking defaults can distort risk.

Failure mode 1avoid

Store money without currency. Amounts cannot be interpreted reliably.

Failure mode 2avoid

Use one timestamp for every purpose. Observation and arrival differ.

Failure mode 3avoid

Assume schema-valid means meaning-valid. Units and definitions still need checking.

A point-in-time feature uses only information available at the decision. Filter both event time and knowledge time when delayed data matters. An event that happened yesterday but arrived tomorrow was not available to today’s model.

For a one-hour count, exclude the current event and define whether the lower boundary is inclusive. The following teaching query counts earlier attempts known by the decision cutoff. Production code also needs the actual schema, indexes, deduplication, and transaction policy.

SELECT count(*)
FROM payment_events
WHERE account_id = :account_id
  AND occurred_at >= :decision_time - interval '1 hour'
  AND occurred_at < :decision_time
  AND received_at <= :decision_time;

Make time-aware features reproducible — the flow

1. 01WindowBound event time
2. 02KnowledgeExclude records learned later
3. 03ReplayRebuild the feature at the original cutoff

Make time-aware features reproducible — the distinction

Historical event

• Happened before the decision

Available evidence

• Was also known before the decision

Late-arrival example

Fictional educational excerpt / Not for execution

Late-arrival example

Illustrative data; not a real customer record or a prescribed policy.

1. 1

   Occurred09:00

   Earlier real event

2. 2

   Received11:00

   Later system knowledge

3. 3

   Decision10:00

   Cannot use the event then

Late data can create hidden leakage

Make time-aware features reproducible — control and failure modes

Control design

Filter by availability as well as event time. Late data can create hidden leakage.

Failure mode 1avoid

Use the final warehouse snapshot. It includes facts learned later.

Failure mode 2avoid

Include the current attempt in prior history. That changes the feature definition.

Failure mode 3avoid

Ignore window boundaries. Off-by-one events can change decisions.

Missing data can come from a new customer, an unsupported source, a timeout, or a broken pipeline. These causes have different meanings. Use explicit validity and freshness fields rather than treating every missing value as zero.

Track completeness by source, product, and relevant population. A global 99 percent completeness rate can hide a fully broken small segment. Define which defects prevent a decision, which permit a bounded fallback, and which require later repair. The data-quality decision should be visible in the risk result so operations can distinguish customer risk from system uncertainty.

Missing is a state with possible causes. A device signal may be absent because the customer uses an unsupported environment, a provider is down, consent is unavailable, or the event arrived through a different product path. Replacing every absence with zero makes these situations look like the same measured value. Preserve the missing state and, where reliable and appropriate, its reason.

The decision policy must define what to do with that state. A model can be trained to handle missing inputs, but a new production outage may create a missingness pattern it never encountered during training. Monitor availability by feature and traffic segment. An aggregate health check can look normal while one high-impact population receives incomplete evidence.

Preserve missingness and quality — the flow

1. 01DetectIdentify missing stale or invalid evidence
2. 02ClassifyRecord the cause where known
3. 03FallbackUse the approved response for that defect

Preserve missingness and quality — the distinction

Known zero

• A valid measured absence

Unknown value

• Measurement is unavailable or invalid

Quality record

Fictional educational excerpt / Not for execution

Quality record

Illustrative data; not a real customer record or a prescribed policy.

1. 1

   history_countnull

   Unavailable value

2. 2

   qualityprovider_timeout

   Known cause

3. 3

   actionbounded fallback

   Policy-defined treatment

System uncertainty must remain visible

Preserve missingness and quality — control and failure modes

Control design

Keep quality status in the decision record. System uncertainty must remain visible.

Failure mode 1avoid

Replace all nulls with zero. Unknown becomes a false measured fact.

Failure mode 2avoid

Monitor only global averages. Small segments can fail completely.

Failure mode 3avoid

Let every consumer invent a fallback. Behavior becomes inconsistent.

Lineage connects a feature to its source events, transformations, and versions. It supports debugging, model review, customer corrections, and incident analysis. A column name is not enough if its definition changed over time.

Store the feature definition version and the source snapshot or reproducible reference needed for the use. Apply privacy controls to the retained data. When a source defect is found, use lineage to identify affected decisions and models. Without it, teams often rerun everything or miss part of the impact because they cannot trace which records consumed the bad field.

Use lineage as an engineering tool — the flow

1. 01SourceIdentify original evidence
2. 02TransformVersion the feature computation
3. 03DecisionLink the resulting value to its use

Use lineage as an engineering tool — the distinction

Column label

• Human-readable field name

Lineage

• Traceable path from source to decision

Feature lineage

Fictional educational excerpt / Not for execution

Feature lineage

Illustrative data; not a real customer record or a prescribed policy.

1. 1

   Featurerefund_ratio

   Output field

2. 2

   Definitionv8

   Window and denominator rules

3. 3

   Source batchbatch-117

   Impact tracing reference

The same name can hide changed meaning

Use lineage as an engineering tool — control and failure modes

Control design

Version definitions and preserve traceable sources. The same name can hide changed meaning.

Failure mode 1avoid

Rename columns as the only history. Past calculations remain unclear.

Failure mode 2avoid

Keep raw data everywhere for convenience. Lineage still needs controlled access.

Failure mode 3avoid

Ignore downstream models during a data incident. They may inherit the defect.

Risk systems need population checks: eligible source events, accepted ingestion, feature computation, decisions, and downstream cases. Compare counts and identifiers across these stages. A model can be accurate on the records it sees while missing a large part of the business.

Use control totals and exception reports with explicit exclusions. Investigate unexpected differences by event type and partner. Preserve duplicates separately from missing events. A total count match can still hide one extra and one missing record, so perform identifier-level checks for consequential paths.

Population reconciliation tests whether the pipeline sees the activity it claims to cover. Compare source transactions with accepted events, feature rows, decisions, and final outcomes using stable identifiers and documented exclusions. An event can be valid in isolation while an entire partition is absent. Count checks, amount checks within currency, and age checks help locate these gaps. Keep duplicates, late arrivals, rejected records, and genuinely out-of-scope activity distinct so the reconciliation can explain a difference rather than merely detect one.

Reconcile the population — the flow

1. 01EligibleDefine the source population
2. 02ProcessedTrace each required stage
3. 03ReconcileExplain every material difference

Reconcile the population — the distinction

Model accuracy

• Quality on scored records

System coverage

• Whether required records were scored at all

Coverage reconciliation

Fictional educational excerpt / Not for execution

Coverage reconciliation

Illustrative data; not a real customer record or a prescribed policy.

1. 1

   Eligible10000

   Source events

2. 2

   Decided9800

   Risk results

3. 3

   Unexplained200

   Coverage gap despite good model accuracy

Coverage is a separate property from accuracy

Reconcile the population — control and failure modes

Control design

Reconcile identifiers as well as totals. Coverage is a separate property from accuracy.

Failure mode 1avoid

Measure only scored records. Missing events disappear from the metric.

Failure mode 2avoid

Assume equal counts mean equal populations. Offsetting errors can remain.

Failure mode 3avoid

Exclude failures without reporting them. The denominator becomes misleading.

Continue with Decision engines, rules, and reliable execution to follow the next part of the system. Use the glossary for terminology and risk mathematics for formulas and worked calculations.