Unit 07 · Chapter 5 · 10 min read

Model governance and AI-assisted risk work

Keep model use bounded, reviewable, and accountable.

A model can be wrong in a sophisticated way. An AI assistant can write a fluent case note with an invented fact. Governance makes those failures visible before the output becomes an unchecked decision.

Use the current model-risk framework

The federal banking agencies issued revised model-risk guidance in April 2026 through SR 26-2, replacing SR 11-7 and SR 21-8. The letter states its expected relevance to Federal Reserve-regulated banking organizations over $30 billion in assets and emphasizes a risk-based approach. Applicability must be assessed for the actual institution.

For this textbook’s engineering design, maintain an inventory of models and their uses, owners, limitations, dependencies, and review evidence. A small rule-based model can still be important if it controls a large exposure. Governance effort should reflect the consequence and complexity of use rather than the prestige of the algorithm.

Use the current model-risk framework Use the current model-risk framework — the flow Follow the sequence. Assign proportionate review and ownership. Inventory Identify models and actual uses Assess Evaluate consequence complexity and limitations Govern Assign proportionate review and ownership — Follow the sequence. Assign proportionate review and ownership. Chapter sources · Open image

Use the current model-risk framework Use the current model-risk framework — the distinction These concepts answer different questions. Read each definition in the context of the section. Algorithm complexity Technical sophistication of the method Use risk Consequence of relying on its output — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Use the current model-risk framework Model inventory Fictional teaching record. Complexity alone is insufficient. Model inventory Illustrative data; not a real customer record or a prescribed policy. Model simple reserve estimator Low algorithm complexity Exposure large merchant portfolio High consequence Review proportionate to use Complexity alone is insufficient A simple model can support a consequential decision — Fictional teaching record. Complexity alone is insufficient. Chapter sources · Open image

Use the current model-risk framework Use the current model-risk framework — control and failure modes A simple model can support a consequential decision. The branches show why alternative designs fail. Control design Assess model risk in its actual use. A simple model can support a consequential decision. Failure mode 1 Treat SR 11-7 as the current sole guidance. The 2026 letter replaced it. avoid Failure mode 2 Apply the banking letter identically to every startup. Institutional scope matters. avoid Failure mode 3 Govern only machine-learning models. Other quantitative tools can be important. avoid — A simple model can support a consequential decision. The branches show why alternative designs fail. Chapter sources · Open image

Separate development from effective challenge

Developers understand the model deeply, but they also know the assumptions they intended. Independent challenge tests whether those assumptions hold and whether the use is appropriate. The structure should fit the organization and applicable expectations.

Review conceptual soundness, data, implementation, outcomes, limitations, and controls. Track findings through remediation and retest. A reviewer’s signature is not evidence that every concern was resolved. The model owner should know which limitations remain and what use is permitted while they remain.

Effective challenge examines whether the model is suitable for its stated use, including limitations that the development team may not have emphasized. It can inspect the target, data, assumptions, validation evidence, implementation, and downstream policy. Independence is useful because the people responsible for delivery may face pressure to interpret ambiguous results favorably. The form and depth of review should fit the model’s materiality and the applicable framework.

A limitation becomes operational when it changes the permitted use. If a model has weak evidence for a new product segment, the response may be restricted deployment, more review, additional validation, or another approved control. Recording the limitation in a document while using the model without restriction does not manage it.

Separate development from effective challenge Separate development from effective challenge — the flow Follow the sequence. Track findings and permitted limitations. Develop Document assumptions and evidence Challenge Test the design and actual use Resolve Track findings and permitted limitations — Follow the sequence. Track findings and permitted limitations. Chapter sources · Open image

Separate development from effective challenge Separate development from effective challenge — the distinction These concepts answer different questions. Read each definition in the context of the section. Developer explanation Why the model was built this way Independent challenge Evidence that tests the explanation — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Separate development from effective challenge Review finding Fictional teaching record. Finding remains tracked. Review finding Illustrative data; not a real customer record or a prescribed policy. Issue weak new-merchant performance Identified limitation Use restriction known merchants only Bounded permitted use Retest scheduled with new evidence Finding remains tracked Approval alone does not remove limitations — Fictional teaching record. Finding remains tracked. Chapter sources · Open image

Separate development from effective challenge Separate development from effective challenge — control and failure modes Approval alone does not remove limitations. The branches show why alternative designs fail. Control design Connect review findings to use restrictions and retesting. Approval alone does not remove limitations. Failure mode 1 Treat a signature as full proof. The underlying findings matter. avoid Failure mode 2 Let unresolved issues disappear at launch. They remain part of the risk. avoid Failure mode 3 Review only code style. Conceptual and data weaknesses can dominate. avoid — Approval alone does not remove limitations. The branches show why alternative designs fail. Chapter sources · Open image

Bound generative AI to evidence-supported tasks

Generative AI can help summarize records, draft narratives, or retrieve relevant policy passages. It can also invent facts, omit context, or follow instructions embedded in untrusted documents. Treat retrieved customer material as evidence to analyze, not instructions to the system.

Use constrained inputs, source references, and a reviewable output. Require factual claims to point to supporting records. Keep sensitive reporting information within approved access boundaries. A model-generated narrative should not automatically file a report, release funds, or change a customer restriction without the authorized decision process.

A generative system can help summarize retained evidence, but fluent wording is not evidence. Each material factual statement should be traceable to the permitted source material, and the workflow needs a response when support is absent or contradictory. Treat retrieved documents and customer submissions as data rather than trusted instructions. Separate the assistant’s proposal from the authorized decision, especially when the action changes funds, account access, or a regulated process. Evaluation should include fabricated facts, missing evidence, private-data exposure, and misleading instructions embedded in source material.

Bound generative AI to evidence-supported tasks Bound generative AI to evidence-supported tasks — the flow Follow the sequence. Verify claims before consequential action. Retrieve Provide approved relevant evidence Draft Generate a bounded supported output Review Verify claims before consequential action — Follow the sequence. Verify claims before consequential action. Chapter sources · Open image

Bound generative AI to evidence-supported tasks Bound generative AI to evidence-supported tasks — the distinction These concepts answer different questions. Read each definition in the context of the section. Fluent narrative Text reads plausibly Grounded narrative Claims are supported by the supplied records — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Bound generative AI to evidence-supported tasks AI drafting record Fictional teaching record. Do not accept fluent invention. AI drafting record Illustrative data; not a real customer record or a prescribed policy. Claim customer admitted intent Generated statement Source none Unsupported claim Action remove and investigate Do not accept fluent invention Fluency does not establish truth — Fictional teaching record. Do not accept fluent invention. Chapter sources · Open image

Bound generative AI to evidence-supported tasks Bound generative AI to evidence-supported tasks — control and failure modes Fluency does not establish truth. The branches show why alternative designs fail. Control design Require claim-level evidence and authorized review. Fluency does not establish truth. Failure mode 1 Let customer documents instruct the assistant. They are untrusted task data. avoid Failure mode 2 Auto-release funds from a generated summary. Consequential authority needs controls. avoid Failure mode 3 Copy restricted reports into broad AI tools. Access and data rules still apply. avoid — Fluency does not establish truth. The branches show why alternative designs fail. Chapter sources · Open image

Evaluate AI failure modes before use

An AI evaluation should reflect the actual task: factual accuracy, omissions, unsupported claims, confidentiality, instruction handling, and consistency. Include difficult cases and deliberate misleading content in controlled fixtures.

Measure the harm of errors, not only an average quality score. A rare invented admission can be more consequential than several awkward sentences. Compare against a useful baseline and define when a human must intervene. Version prompts, retrieval rules, models, and evaluation sets so a vendor update does not silently invalidate the evidence.

Evaluate AI failure modes before use Evaluate AI failure modes before use — the flow Follow the sequence. Use thresholds and human review appropriate to consequence. Task Define the permitted output and harm model Evaluate Test representative and adversarial fixtures Gate Use thresholds and human review appropriate to consequence — Follow the sequence. Use thresholds and human review appropriate to consequence. Chapter sources · Open image

Evaluate AI failure modes before use Evaluate AI failure modes before use — the distinction These concepts answer different questions. Read each definition in the context of the section. Average fluency General writing quality Critical error rate Frequency of consequential unsupported or unsafe output — These concepts answer different questions. Read each definition in the context of the section. Chapter sources · Open image

Evaluate AI failure modes before use AI evaluation Fictional teaching record. Average style score is insufficient. AI evaluation Illustrative data; not a real customer record or a prescribed policy. Cases 200 synthetic records Controlled test set Invented facts 3 Critical errors Decision not ready for autonomous use Average style score is insufficient A good average can hide unacceptable rare failures — Fictional teaching record. Average style score is insufficient. Chapter sources · Open image

Evaluate AI failure modes before use Evaluate AI failure modes before use — control and failure modes A good average can hide unacceptable rare failures. The branches show why alternative designs fail. Control design Evaluate consequential errors separately. A good average can hide unacceptable rare failures. Failure mode 1 Score only writing style. Truth and confidentiality remain untested. avoid Failure mode 2 Reuse old evaluations after major changes. The system behavior may differ. avoid Failure mode 3 Use real confidential cases without approval. Test data also has access constraints. avoid — A good average can hide unacceptable rare failures. The branches show why alternative designs fail. Chapter sources · Open image

Maintain a complete change record

Model behavior can change through data, features, code, thresholds, prompts, retrieval sources, or vendor versions. Keep a release record that connects the change to evaluation, approval, rollout, monitoring, and rollback.

Define what counts as a material change and who decides. A prompt edit that adds a new tool can be more consequential than a model patch. Preserve the affected population and outputs for review under the retention policy. Governance is complete when the organization can explain what changed, why it was allowed, and how the result was checked.