Model Risk Management for LLMs: Applying SR 11-7 to Generative AI in Financial Services

Financial institutions have had a model risk management framework since 2011. The Federal Reserve and the OCC's SR 11-7 / OCC 2011-12 guidance defines how banks must validate, govern, and monitor the quantitative models that drive credit decisions, trading, and risk measurement. Large language models are now making decisions that affect those same domains. SR 11-7 applies to them, and most banks have not yet worked out how.

This is not a speculative problem. Regulators have begun asking financial institutions directly how their AI systems, including generative AI tools, fit within their existing model risk management programs. The OCC's 2023 bank supervision operating plan identified AI and machine learning as a supervisory priority. Banks that cannot demonstrate a coherent MRM approach to their LLM deployments face examination findings that are expensive to remediate and operationally disruptive to fix after the fact.

This post maps SR 11-7's three core components (model development and implementation, model validation, and ongoing monitoring) to the specific challenges that LLMs present, identifies where the existing guidance is silent and requires supervisory interpretation, and gives CROs and technology risk leaders a practical starting framework for their LLM model inventory and validation programs.

What SR 11-7 Actually Says

SR 11-7 defines a model as "a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates." The guidance breaks model risk into two sources: the model producing incorrect outputs when used correctly, and the model being used incorrectly or in inappropriate circumstances.

The three-pillar structure of SR 11-7 requires: first, that model development follows a documented methodology with clear assumptions, limitations, and intended use; second, that model validation is performed by a function independent of model development, covering conceptual soundness, ongoing monitoring, and outcomes analysis; and third, that governance ensures models are used within their intended scope, that changes are controlled, and that senior management and the board receive appropriate reporting on model risk.

The guidance was written before transformer architectures existed, and it shows. SR 11-7 assumes models have explicit, interpretable mathematical formulations, defined input-output mappings, and quantifiable uncertainty. LLMs have none of these properties in the traditional sense. A 70-billion-parameter language model does not have a "conceptual soundness" that can be evaluated by reading a model documentation package. Its outputs emerge from patterns in pretraining data that cannot be exhaustively characterized. This creates genuine gaps between the framework's assumptions and the realities of LLM deployment, gaps that banks must bridge through supervisory judgment and documented interpretation.

"SR 11-7 was written for models you can open and inspect. LLMs are models you can only probe. That changes the validation methodology, not the obligation."

Does SR 11-7 Apply to LLMs?

The short answer is: it depends on what the LLM is doing. SR 11-7's definition of a model is broad enough to capture most substantive LLM applications in financial services, but regulators have been careful not to assert that all generative AI tools are models in the SR 11-7 sense.

An LLM that generates marketing copy for customer communications is probably not a model under SR 11-7, because it is not producing quantitative estimates that drive financial decisions. An LLM that summarizes credit analyst research to support underwriting decisions, assesses the credit quality of loan documentation, generates risk narratives for regulatory capital models, or flags suspicious transaction patterns for BSA/AML review is almost certainly a model under SR 11-7, because its outputs directly influence decisions within the scope of the guidance.

The OCC's 2021 guidance on bank use of artificial intelligence (OCC 2021-78, the interagency statement on AI risk management) explicitly states that AI systems used in credit underwriting, fraud detection, risk management, and customer compliance are subject to existing model risk management supervisory expectations. This includes both traditional ML models and, by clear implication, LLM-based systems performing similar functions.

The practical implication is that every financial institution needs to apply an SR 11-7 materiality assessment to each LLM application in its portfolio. The assessment should document the LLM's role in decision-making, the financial decisions influenced by its outputs, the population of affected customers or transactions, and the potential for consumer harm if the model produces incorrect outputs. High-materiality LLM applications require full SR 11-7 validation. Low-materiality applications require at minimum a documented rationale for why they fall outside the guidance's scope.

The Three Pillars: Where LLMs Break the Framework

Pillar 1: Model Development and Implementation

SR 11-7 requires that model documentation cover the theoretical and conceptual basis of the model, the data used in development, key assumptions and limitations, and the intended use. For a traditional credit scoring model, this documentation is straightforward: the logistic regression coefficients are interpretable, the training data is proprietary and auditable, and the intended use is precisely defined.

For a foundation model like Llama or a GPT-series model, the "theoretical and conceptual basis" is a transformer architecture trained on trillions of tokens of internet text, clinical records, code, and other data that the model developer has documented at varying levels of specificity. Most financial institutions deploying a commercial or open-weight LLM did not build it and cannot fully characterize its pretraining data. This creates a documentation gap: SR 11-7 requires the bank to document what it often cannot know.

The practical approach is to distinguish between the foundation model and the bank's application layer. The foundation model is treated as a third-party model component, subject to the vendor management requirements of SR 11-7 Section III, which requires that the bank conduct appropriate due diligence on external model components and obtain sufficient information to validate the component's fitness for purpose. The bank's application layer, including system prompts, retrieval pipelines, fine-tuning, and output processing, is within the bank's development scope and must be fully documented. This bifurcated documentation approach is not explicitly described in SR 11-7, but it is consistent with the guidance's third-party risk principles and is the approach that validation teams are converging on in practice.

Pillar 2: Model Validation

SR 11-7 requires independent validation covering three elements: evaluation of conceptual soundness, ongoing monitoring, and outcomes analysis. Each element requires significant adaptation for LLMs.

Conceptual soundness evaluation for a traditional model means reviewing the mathematical derivation, checking that the model's assumptions are appropriate for the intended use, and assessing whether the model structure is theoretically justified. For an LLM, conceptual soundness evaluation means something different: assessing whether the model architecture and pretraining data are appropriate for the specific financial task, whether the prompting and retrieval design is sound, and whether the model's known failure modes (hallucination, sycophancy, context window limitations) are adequately mitigated by the application design. This is a task for validation teams that combine financial domain expertise with ML engineering knowledge. Most bank validation teams built for traditional model risk do not yet have this combination.

Ongoing monitoring of LLMs requires establishing quantitative performance metrics for tasks that are not inherently quantitative. A credit narrative generation model does not have a single scalar performance metric analogous to a credit model's Gini coefficient. Banks are developing domain-specific evaluation harnesses that score LLM outputs on accuracy, completeness, consistency, and hallucination rate using a combination of automated scoring (LLM-as-judge approaches, factual consistency checkers) and periodic human review by subject matter experts. The monitoring cadence, thresholds for escalation, and remediation protocols must be defined in the model's validation documentation before the model goes into use.

Outcomes analysis for LLMs in decision-support roles requires comparing the decisions made with LLM assistance against decisions made without it, and tracking whether LLM-influenced decisions produce better or worse outcomes over time. This is methodologically challenging because the counterfactual is hard to observe, and because LLM-influenced decisions are often confounded with analyst quality, deal characteristics, and market conditions. Banks are addressing this through controlled pilot designs that randomize LLM assistance across comparable decision units, enabling pre-post and treatment-control comparisons that produce actionable outcomes data.

Pillar 3: Governance

SR 11-7's governance requirements include a model inventory, tiering of models by materiality, board and senior management reporting, and a policy framework for model change management. Adapting these to LLMs requires several specific decisions.

The model inventory must include every LLM application that meets the materiality threshold, with its intended use, the population it affects, the validation status, and the assigned model owner. The inventory should capture the specific model version in deployment, because LLMs from commercial providers update frequently and version changes may constitute model changes that require re-validation under SR 11-7's change management provisions.

Tiering is the most consequential governance decision. SR 11-7 calls for higher validation rigor for higher-risk models. For LLMs, the relevant risk dimensions are the directness of the model's influence on financial decisions (decision-support vs. autonomous decision-making), the size and vulnerability of the affected population (retail customers vs. internal analysts), the regulatory sensitivity of the domain (credit vs. marketing), and the reversibility of potential errors. An LLM that generates first-draft credit memos for analyst review carries materially lower risk than an LLM whose output directly populates a credit decision system without analyst review. These two use cases should be in different model tiers with correspondingly different validation requirements.

The Gaps SR 11-7 Does Not Address

SR 11-7 was built for models with stable, documentable behavior. LLMs have three properties that the guidance does not address and that require supervisory interpretation or supplementary bank policy to manage.

Non-Determinism

Traditional models produce the same output for the same input, every time. LLMs do not. The temperature parameter in an LLM inference call introduces stochasticity: the same prompt can produce materially different outputs on successive calls. This breaks SR 11-7's assumption that model outputs are reproducible for validation purposes. Banks are addressing this by setting temperature to zero for all SR 11-7-covered LLM applications, producing deterministic outputs that can be replicated in validation testing. For applications where temperature is set above zero, validation must include testing across a distribution of outputs for a fixed prompt, not just a single output.

Prompt Sensitivity

LLM outputs are highly sensitive to the exact wording, structure, and content of the input prompt. A one-word change in a system prompt can shift model behavior in ways that a traditional model's sensitivity analysis would never reveal. SR 11-7's sensitivity testing requirements, designed for models where inputs are numerical variables with defined ranges, do not map cleanly to prompt sensitivity. Banks must develop prompt sensitivity testing methodologies specific to their LLM applications: systematic variation of prompt phrasing, ordering, and context to characterize how sensitive the model's outputs are to prompt design choices, and to ensure that the validated prompt is the prompt in use in the application.

Model Drift via Provider Updates

When a bank uses a foundation model from a commercial provider, the underlying model can change without the bank's direct action. A model provider may update their model's weights, safety filters, or inference behavior as part of routine operations. Under SR 11-7's change management provisions, a material change to a model requires documented review and, for material changes, re-validation. Banks using cloud-hosted LLMs must negotiate contractual provisions that give them advance notice of model updates and the ability to pin to a specific model version during validation cycles. Without these provisions, the bank cannot maintain the change management audit trail that SR 11-7 requires.

The EU AI Act Intersection

Financial institutions operating in the European Union face an additional layer of AI governance requirements from Regulation (EU) 2024/1689, the EU AI Act. Several LLM applications in financial services qualify as high-risk AI systems under Annex III of the Act, specifically systems used in creditworthiness assessment, credit scoring, and access to financial services. High-risk AI systems under the EU AI Act require conformity assessments, technical documentation, logging of system operation, and registration in the EU database before deployment.

The interaction between SR 11-7 and the EU AI Act creates a complex compliance matrix for internationally active banks. The two frameworks share intent but differ in specifics: SR 11-7's model inventory maps partially to the EU AI Act's technical documentation requirement; SR 11-7's ongoing monitoring maps to the Act's post-market monitoring obligations; but the Act's conformity assessment process has no direct SR 11-7 equivalent and requires separate procedures. Banks operating under both frameworks should build a unified AI governance framework that satisfies both sets of requirements simultaneously rather than running parallel compliance programs.

A Practical Gap Analysis

The table below maps common LLM applications in financial services to SR 11-7 classification and the primary validation gaps to address.

What to Ask Your Chief Risk Officer

• Have we completed an SR 11-7 materiality assessment for every LLM application in the firm? If your model inventory does not yet include LLM applications, you are behind the current supervisory expectation. The OCC's 2023 supervision operating plan signals that examiners will ask this question.
• Does our validation team have the capability to evaluate LLM conceptual soundness? SR 11-7 validation of LLMs requires a combination of ML engineering knowledge and financial domain expertise that most traditional model validation teams do not have. Identifying this capability gap and addressing it is a prerequisite for compliant validation.
• What contractual protections do we have against provider model updates triggering uncontrolled model changes? If you are using a commercial LLM API and you cannot pin to a specific model version, you cannot maintain the change management audit trail SR 11-7 requires. This is a procurement issue that legal and technology risk need to resolve together.
• How are we addressing prompt sensitivity as a validation requirement? If your current LLM validation methodology does not include systematic prompt variation testing, your validation is incomplete under the conceptual soundness standard.
• What is our EU AI Act conformity assessment status for high-risk AI applications? For banks operating in the EU, the Act's Annex III classification of credit-related AI as high-risk creates conformity assessment obligations that are separate from and additional to SR 11-7 requirements.