Memory and Context Management

Why Memory Matters for Agents

A simple chatbot has no memory between conversations. Every session starts fresh. For answering questions, that is often fine. But for agents working on multi-step tasks that take minutes or hours, memory is everything. Without memory, an agent cannot build on what it has already discovered, avoid repeating steps it already completed, or maintain a coherent plan across a long chain of actions.

Think about what a researcher does: they take notes, organize them, refer back to earlier findings, and update their understanding as they learn more. An agent needs to do the same thing, but the mechanisms for doing it are built into software rather than a notebook.

The Four Types of Agent Memory

In-context memory is the most immediate. Everything in the current prompt window counts as in-context memory: the original task, tool results, reasoning steps so far, and any information the agent has retrieved. It is the fastest to use and the first to run out.

External retrieval memory is stored outside the model, in a database or document store. The agent uses a tool to search it when it needs something. This is the basis of Retrieval-Augmented Generation (RAG): instead of stuffing all relevant documents into the prompt, the agent fetches only what it needs for the current step.

Episodic memory is a record of what the agent has done in the past. A customer support agent with episodic memory can look up what happened in a previous ticket from the same customer. This type of memory makes agents feel more like experienced professionals and less like people with no long-term recall.

Procedural memory is encoded in the model itself through training and fine-tuning. When an agent "knows" how to write a structured report or how to handle an edge case in a refund policy, that knowledge lives in the model's weights, not in the prompt. Procedural memory is the most durable but the hardest to update.

The Context Window Problem

The context window is the total amount of text a model can see at one time. Once a conversation or task exceeds the context window, older information either gets cut off or the whole thing fails. For short tasks, this is rarely a problem. For long agentic tasks, it is a constant design constraint.

Imagine an agent processing a 500-page legal contract. Even models with very large context windows have limits, and stuffing an entire document into a prompt is often inefficient. The agent needs a strategy for what to keep in context and what to retrieve on demand.

Two Strategies for Managing Long Tasks

Summarization. As the context fills up, the agent periodically summarizes what it has learned and replaces the raw history with the summary. This compresses information but loses detail. The key design question is: what can be compressed, and what must be kept verbatim?

External note-taking. The agent uses a write-to-database tool to store key findings, decisions, and intermediate results as it goes. When it needs to recall something, it queries the database rather than relying on in-context memory. This is more reliable for long tasks but requires the agent to explicitly decide what to save, which is itself a reasoning challenge.

A Second Analogy: The Detective's Casebook

A detective investigating a complex case does not try to hold every clue in working memory. They maintain a casebook: a written record of every lead, every interview, every piece of evidence. When they need to connect two facts from a week apart, they consult the book rather than relying on recall. The casebook is their external retrieval memory.

But the detective also carries procedural knowledge from years of experience: how to interview a witness, how to read a crime scene, how to spot when a alibi has a hole in it. That knowledge is not in the casebook. It is in their training, applied automatically without requiring lookup.

And the detective remembers what they did yesterday on this case (episodic memory) without needing to reread their entire casebook every morning. They pick up where they left off. Good agent memory architecture works the same way: separate stores for different types of information, each optimized for how that information is used.

A Healthcare Example: Patient History Retrieval

A clinical decision-support agent working with a patient's record faces a real memory challenge. A patient who has been in the healthcare system for ten years has a record that may span thousands of entries: diagnoses, lab results, medications, procedures, referral letters, and clinical notes. No context window can hold all of that. But the agent does not need all of it for every question. When the physician asks about the patient's cardiac history, the agent retrieves only the relevant cardiac entries from the external record store. When the question shifts to current medications and potential interactions, the agent retrieves the medication list. The retrieval is targeted, not total.

This is why RAG (Retrieval-Augmented Generation) is the standard architecture for clinical AI: it lets the agent work with effectively unlimited external memory while keeping the in-context prompt focused on the specific information needed for the current reasoning step. The agent's procedural knowledge (clinical reasoning patterns, drug interaction logic) lives in the model weights. Patient-specific facts live in the external store. The two work together, but they are stored separately.

from datetime import datetime

class AgentMemory:
    """Two-tier memory: in-context list + external key-value store."""

    def __init__(self, context_limit: int = 20):
        self.context = []          # in-context (fast, limited)
        self.store = {}            # external store (persistent)
        self.context_limit = context_limit

    def add(self, key: str, value: str, persist: bool = False):
        """Add to in-context memory; optionally persist externally."""
        entry = {"key": key, "value": value, "ts": datetime.utcnow()}
        self.context.append(entry)

        if persist:
            self.store[key] = value  # write to external store

        # Trim context if over limit
        if len(self.context) > self.context_limit:
            self.context = self.context[-self.context_limit:]

    def retrieve(self, key: str) -> str | None:
        """Check context window first, fall back to external store."""
        for entry in reversed(self.context):
            if entry["key"] == key:
                return entry["value"]   # found in context
        return self.store.get(key)      # fall back to external store

    def get_context_window(self) -> list[dict]:
        """Return current context formatted as system messages."""
        return [{"role": "system",
                 "content": f"{e['key']}: {e['value']}"}
                for e in self.context]

# Usage
memory = AgentMemory(context_limit=20)
memory.add("patient_id", "PT-88421", persist=True)
memory.add("current_issue", "chest pain onset 2 days ago")
print(memory.retrieve("patient_id"))  # "PT-88421"