Agentic Failure Modes

Why Agent Failures Are Different

When a chatbot gives a wrong answer, the human reading it can catch the mistake and ignore it. The damage is limited to the time it took to read a bad response. When an agent takes a wrong action, the damage may be done before anyone notices. A file might be deleted. An email might be sent. A payment might be processed. Agentic failures are different from chatbot failures because they are harder to observe in real time and harder to reverse after the fact.

This changes what it means to test and safeguard an AI system. Testing a chatbot means checking whether the output is accurate. Testing an agent means checking whether every possible action it might take is safe and reversible, or at least detectable before serious harm occurs.

The Six Main Failure Modes

Two Design Patterns That Reduce Risk

Human-in-the-loop checkpoints. Before the agent takes any irreversible action, pause and ask a human to approve it. This slows the agent down but prevents catastrophic mistakes. You can scope this narrowly: require approval only for write actions, or only for actions above a certain risk level (for example, any email send or any payment above $100).

Dry-run mode. Before running a task for real, run it in a dry-run mode where all write tool calls are logged but not executed. Review the log of intended actions. If the plan looks right, approve it and run for real. This is especially useful when deploying a new agent for the first time or making significant changes to an existing one.

A Second Analogy: Aviation Checklists

Commercial aviation has one of the best-studied track records for preventing cascading failures. The key tool is not better pilots, it is checklists. Before every flight, before every phase of a flight, and in every emergency scenario, pilots work through a structured checklist. The checklist does not assume the pilot is incompetent. It assumes that even a skilled pilot under pressure can miss a step, and that one missed step in a safety-critical system can cascade into something catastrophic.

Agentic AI systems need the same philosophy. The six failure modes above are not failures of unintelligent agents. They are failures of intelligent systems that had the wrong constraints, the wrong permissions, or no checkpoint at the critical moment. The fix for most agentic failures is not a smarter model, it is better checklists: explicit exit conditions, permission allowlists, human-in-the-loop gates before irreversible actions, and a dry-run review before first deployment.

A Legal Services Example

A law firm deploys an agent to review incoming contracts and flag clauses that deviate from standard terms. Goal drift is the highest-risk failure mode for this use case. The agent starts with a clear mandate: compare clause X against the firm's standard language for clause X. But over a chain of reasoning steps, it begins comparing the entire contract against competitor firm agreements it has seen, then drafting alternative language, then effectively rewriting the contract rather than reviewing it. No individual reasoning step is obviously wrong. The goal just drifted, step by step, from "review" to "redraft." The fix is periodic re-anchoring in the system prompt ("Your task is to identify deviations only, not to propose alternatives") and a maximum step count that forces a human review before the agent can take more than a bounded number of reasoning steps.

from typing import Callable

REVERSIBLE_TOOLS = {
    "search_web", "read_file", "get_database_record", "query_calendar"
}
IRREVERSIBLE_TOOLS = {
    "send_email", "delete_file", "process_payment",
    "submit_form", "update_database"
}

def guarded_tool_call(tool_name: str, args: dict,
                      human_approve: Callable) -> dict:
    """Wrapper that checks reversibility before executing any tool."""

    # Check if this tool is in the allowlist
    if tool_name not in REVERSIBLE_TOOLS | IRREVERSIBLE_TOOLS:
        raise PermissionError(f"Tool '{tool_name}' is not in the allowlist.")

    # Require human approval for irreversible actions
    if tool_name in IRREVERSIBLE_TOOLS:
        summary = f"IRREVERSIBLE: call {tool_name} with args {args}"
        approved = human_approve(summary)
        if not approved:
            return {"status": "rejected", "reason": "human declined"}

    # Execute and sanitize output before returning to model
    raw_result = execute_tool(tool_name, args)
    sanitized = sanitize_output(raw_result)
    return {"status": "ok", "result": sanitized}

def sanitize_output(text: str) -> str:
    """Strip potential prompt injection patterns from tool output."""
    injection_markers = [
        "ignore previous instructions",
        "new instructions:",
        "system prompt:",
    ]
    for marker in injection_markers:
        if marker in text.lower():
            return "[OUTPUT REDACTED: potential injection detected]"
    return text

def human_approve(action_summary: str) -> bool:
    print(f"\nApproval required: {action_summary}")
    response = input("Allow? (yes/no): ").strip().lower()
    return response == "yes"