An insurance company's AI model was denying claims at a higher rate for applicants in one zip code. Nobody had flagged it. Nobody had a process to catch it. The model had been running for two years. This is not a hypothetical story: versions of this incident have appeared in insurance, lending, and hiring in multiple jurisdictions. AI governance is how you prevent your organization from becoming the next case study.
By the end of this module you will be able to
Explain the four pillars of an AI governance program
Design a minimal viable model inventory for your organization
Map any AI use case to its EU AI Act risk tier and explain the compliance implications
Describe what ISO/IEC 42001 certification requires and who should pursue it
What AI Governance Actually Is
AI governance is not a compliance checkbox. It is the system of policies, processes, and accountability structures that determines who makes decisions about AI, how those decisions are made, who is responsible when things go wrong, and how you detect problems before they become crises.
Most organizations have governance for other high-risk technology: change management processes for software updates, access controls for sensitive data, audit trails for financial systems. AI governance applies the same discipline to decisions made or influenced by AI models. The stakes are higher in most AI cases because the decisions are harder to inspect: a software bug is traceable, but a biased model output may look like a correct result until someone examines patterns across thousands of decisions.
Governance vs compliance
Compliance is meeting the minimum standard required by law or regulation. Governance is having the processes to know whether your AI systems are behaving as intended. Good governance produces compliance as a side effect. Compliance without governance means you are passing audits while problems accumulate that no audit has been designed to catch yet.
The Four Governance Pillars
Pillar 1: Model Inventory. You cannot govern what you cannot see. An AI model inventory is a living register of every AI system your organization uses or relies upon, including third-party AI embedded in vendor products. Each entry records: what the model does, who owns it, what data it processes, what decisions it influences, the risk tier it falls into, and when it was last evaluated. Most organizations discover during their first inventory that they have two to four times more AI systems in use than leadership was aware of.
Pillar 2: Impact Assessment. Before deploying any AI system in a consequential context, conduct a structured assessment of what could go wrong. What are the error modes? Who is affected when the model is wrong? Are any affected groups protected classes? What is the escalation path when the model produces an output that requires human review? Document the answers and require sign-off from the model owner and a designated AI risk function before deployment.
Pillar 3: Monitoring. A model that passes its pre-deployment evaluation will drift over time as the real-world data it sees diverges from the data it was trained on. Ongoing monitoring tracks model performance against agreed metrics, flags statistical anomalies in outputs across demographic groups, and triggers a review when thresholds are breached. Monitoring is not optional for high-risk AI systems. It is the mechanism that converts a point-in-time evaluation into a continuous assurance.
Pillar 4: Incident Response. When an AI system produces a harmful output, you need a documented process: who is notified, who has the authority to suspend the system, how affected parties are identified, and how the organization communicates about the incident. Without an incident response procedure, your first harmful AI output will be handled by whoever happens to pick up the phone, with whatever authority they happen to have.
AI Model Inventory: What to Track
A minimal viable model inventory has eight fields per entry. Every additional field is optional until you have these eight working consistently across your portfolio.
The eight fields: model name and version, business purpose (one sentence), model owner (named individual, not a team), data processed (PII, financial, health, or general), deployment context (where it runs and who sees the output), risk tier (described below), last evaluation date, and monitoring status (active, pending setup, or not required).
The model owner field is the most important and the most often left vague. A team name is not a model owner. A named individual who is accountable for the model's behavior and responsible for initiating a review when problems are flagged is a model owner. That person should know they are the owner.
The EU AI Act Risk Tiers
Regulation (EU) 2024/1689, the EU AI Act, classifies AI systems into four risk tiers. If your organization operates in the EU or serves EU customers, understanding these tiers is now a legal requirement, not an option.
Fig 3 · EU AI Act Risk Pyramid
The unacceptable risk tier includes AI systems that are banned entirely under Article 5 of the EU AI Act: real-time biometric surveillance in public spaces by law enforcement (with narrow exceptions), social scoring systems by public authorities, and AI that exploits psychological vulnerabilities to manipulate behavior.
The high-risk tier includes AI systems used in employment decisions (CV screening, performance assessment, promotion), access to education, credit scoring and insurance pricing, critical infrastructure management, law enforcement, migration and asylum management, and AI used in medical device contexts. High-risk systems must maintain detailed technical documentation, undergo conformity assessment, implement human oversight, and maintain audit logs. Penalties for non-compliance can reach 3% of global annual revenue for providers and deployers who fail to meet obligations.
The limited-risk tier applies to AI systems where the primary governance requirement is transparency: users must be informed they are interacting with an AI, and AI-generated content must be labeled as such. This covers most consumer-facing chatbots and content generation tools.
Minimal-risk systems, which include the vast majority of AI applications such as spam filters, recommendation engines, and search ranking, face no specific obligations under the Act.
ISO/IEC 42001: The AI Management System Standard
ISO/IEC 42001:2023 is the international standard for AI management systems. It provides a framework for organizations to establish, implement, maintain, and improve an AI governance program. Certification under ISO 42001 demonstrates to customers, regulators, and partners that your AI governance practices meet a recognized international standard.
Who should pursue ISO 42001 certification: organizations that develop or deploy AI for external customers in regulated industries, organizations bidding on government AI contracts, and organizations for whom AI governance is a customer trust differentiator. The certification process typically takes 6-18 months depending on the organization's existing governance maturity, and requires an independent third-party audit.
Interactive 3: Governance Maturity ChecklistTry it
Check each governance capability your organization has today. Your maturity score updates as you go.
Governance maturity: 0 of 7 Level: Ad Hoc
JSON · Model Inventory Schema
{
"model_inventory": [
{
"id": "mdl-2024-001",
"name": "Loan Application Scorer v2.3",
"version": "2.3.1",
"business_purpose": "Scores personal loan applications 0-100 for underwriting triage",
"model_owner": "[email protected]",
"vendor": "Internal (built by Data Science team)",
"data_processed": ["income", "credit_history", "employment", "PII"],
"deployment_context": "Loan origination system, outputs visible to underwriters",
"eu_ai_act_tier": "high_risk",
"risk_tier_rationale": "Credit scoring for individuals falls under Annex III, point 5(b)",
"last_evaluation_date": "2025-11-15",
"evaluation_frequency": "quarterly",
"monitoring_status": "active",
"monitoring_thresholds": {
"accuracy_floor": 0.88,
"disparate_impact_ceiling": 0.20,
"drift_alert_threshold": 0.05
},
"human_override_required": true,
"last_incident_date": null,
"conformity_assessment_required": true,
"notes": "Pending external audit Q1 2026 for EU AI Act compliance"
}
]
}
Try this
List every AI system your organization currently uses or plans to deploy in the next 12 months. For each one, write down: the business purpose in one sentence, the name of the person who would be held accountable if the system produced a harmful output, and the EU AI Act risk tier it falls into. If you cannot name an accountable individual for any system on the list, that gap is your first governance priority.
Why does a model that passes pre-deployment evaluation still need ongoing monitoring?▼
The data a model sees in production drifts away from the data it was trained and evaluated on. Seasonality, economic shifts, changes in user behavior, changes in the underlying population, and even changes in how data is collected can all shift the distribution. A credit scoring model trained before a recession may perform very differently during one, even though the model itself has not changed. Monitoring catches this drift before it accumulates into a systematic problem. Pre-deployment evaluation tells you the model works today. Monitoring tells you whether it still works next quarter.
Knowledge check
An organization is using AI to screen job applications and rank candidates for hiring manager review. Under the EU AI Act, what risk tier does this AI system fall into?
Correct. Under the EU AI Act (Regulation (EU) 2024/1689), AI systems used for employment decisions including recruitment, selection, and evaluation of persons are listed as high-risk in Annex III. The fact that a human makes the final decision does not reduce the risk tier.
AI used in hiring falls into the high-risk tier under Annex III of the EU AI Act, regardless of whether a human makes the final decision. The AI's influence on the outcome is what determines the risk tier, not the formal decision authority.
Which of the four governance pillars is most commonly missing when an AI bias incident is discovered months or years after deployment?
Exactly. Most AI bias incidents that surface months or years after deployment involve a model that was evaluated before deployment but had no ongoing monitoring in place. The model drifted, or a previously undetected pattern became statistically significant at scale, and there was no process to detect it until someone noticed manually.
The most commonly missing pillar in delayed bias discoveries is ongoing monitoring. Pre-deployment impact assessments catch some risks, but they cannot detect drift, emerging patterns, or disparate impact that only becomes visible at production scale over time.
Before you go
AI governance has four pillars: model inventory (know what you have), impact assessment (evaluate before deployment), monitoring (track behavior in production), and incident response (know what to do when something goes wrong).
The EU AI Act (Regulation EU 2024/1689) classifies AI systems into four tiers: unacceptable (banned), high-risk (regulated, requires conformity assessment), limited risk (transparency required), and minimal risk (no specific obligations).
ISO/IEC 42001:2023 is the AI management system standard. Certification is worth pursuing for organizations building or deploying AI for regulated-industry customers or government clients.
Reflection: Does your organization currently maintain a model inventory? If you were asked to produce a complete list of every AI system in use today, how long would it take, and how confident would you be that the list was complete?
Share this insight: "You cannot govern what you cannot see. Most organizations discover during their first AI model inventory that they have two to four times more AI systems in use than leadership was aware of." (Enterprise AI Course, arjunjaggi.com)