AI Risk and Security

Why AI Risk Differs from Traditional Software Risk

Traditional software fails in predictable ways. A database query either returns data or throws an error. A payment API either processes the transaction or declines it. The failure modes are documented, testable, and largely binary. AI systems introduce a different class of risk: they can fail silently, degrade gradually, produce outputs that are plausible but wrong, and be manipulated through their inputs in ways that no standard penetration test would catch.

This difference matters for how you govern AI. You cannot simply add AI to your existing IT risk framework and call it managed. You need a new vocabulary for the failure modes, and a new monitoring posture that watches for drift rather than just downtime.

The Four AI Risk Categories

Prompt Injection: The Attack Your Security Team Has Not Seen Before

Prompt injection is the most common attack on deployed language model systems. An attacker embeds instructions into content the model processes, causing it to ignore its original instructions and follow the attacker's instead. In the bank example, the attacker did not need to breach any firewall. They simply typed a message that overrode the system prompt.

The mechanism works because language models do not distinguish between "instructions I was given by the developer" and "content I am processing for the user." Both arrive as text. A well-crafted user message can instruct the model to: ignore all previous instructions, switch to a different persona, reveal information from its context window, or take actions it was not authorized to take.

Data Poisoning and Training Data Risk

Data poisoning occurs when malicious or corrupted data enters a model's training pipeline. The model learns incorrect patterns that an attacker can later trigger with specific inputs. This is most relevant for organizations fine-tuning foundation models on their own data, or running retrieval-augmented systems where external content can influence the model's context.

Training data risk extends beyond active poisoning. Data that was accurate when collected becomes stale. Customer service transcripts from 2020 may reflect products, policies, and pricing that no longer exist. A model trained on that data will confidently give wrong answers. The risk is not malicious; it is simply the passage of time.

Model Drift: When Your AI Slowly Stops Working

Model drift is the degradation of a model's performance over time as the real world changes but the model does not. There are two kinds. Data drift occurs when the inputs the model receives at inference time differ from the inputs it was trained on. A fraud detection model trained on 2022 transaction patterns may struggle with 2025 transaction patterns. Concept drift occurs when the relationship between inputs and correct outputs changes. Customer sentiment toward a feature may reverse after a product change, but a model trained before that change does not know.

Detecting drift requires monitoring the statistical properties of your inputs and outputs over time, comparing them to a baseline established at launch. The KL divergence measure quantifies how much the current distribution has shifted from the baseline. When it exceeds your threshold, you trigger a review or retraining cycle.

import numpy as np
from scipy.special import kl_div

def detect_model_drift(
    baseline_dist: np.ndarray,
    current_dist: np.ndarray,
    threshold: float = 0.1
) -> dict:
    """
    Detect model drift using KL divergence.

    baseline_dist: probability distribution from model launch period
    current_dist:  probability distribution from recent inference window
    threshold:     KL divergence above which drift is flagged (0.1 = moderate)

    Returns drift assessment with recommended action.
    """
    # Smooth to avoid log(0) issues
    eps = 1e-10
    baseline = np.clip(baseline_dist, eps, 1)
    current  = np.clip(current_dist,  eps, 1)

    # Normalize to valid probability distributions
    baseline = baseline / baseline.sum()
    current  = current  / current.sum()

    # KL divergence: KL(current || baseline)
    kl_score = float(np.sum(kl_div(current, baseline)))

    if kl_score < threshold * 0.5:
        severity = "none"
        action   = "No action required. Continue monitoring."
    elif kl_score < threshold:
        severity = "low"
        action   = "Schedule model review within 30 days."
    elif kl_score < threshold * 2:
        severity = "medium"
        action   = "Initiate retraining pipeline. Review within 7 days."
    else:
        severity = "high"
        action   = "Pause automated decisions. Immediate human review required."

    return {
        "kl_divergence": round(kl_score, 4),
        "threshold":     threshold,
        "severity":      severity,
        "action":        action
    }

The NIST AI Risk Management Framework

The National Institute of Standards and Technology published the AI Risk Management Framework (AI RMF 1.0) in January 2023. It is the most widely adopted voluntary framework for managing AI risk in the United States, and it is referenced in procurement requirements across the federal government and many large enterprises. The framework organizes AI risk management into four core functions.

The NIST AI RMF does not prescribe specific controls or require specific actions. It is a vocabulary and a process scaffold. Its value is that it gives cross-functional teams, including legal, compliance, security, and business leaders, a shared language for AI risk conversations.