AI Agent Memory Explained: Four Types That Shape What Agents Remember

Ask most people how an AI agent remembers things and they will say "the context window." That is one of four memory types, and in most real deployments it is the least powerful one on its own. Here is how all four work, when to use each, and why memory design is one of the most underestimated variables in agent reliability.

The Problem With Context Windows as the Sole Memory

A language model's context window is the stretch of text it can read at once. Everything outside that window is invisible to the model as if it never existed. For a simple question-and-answer task, this is fine. For a multi-step agent that runs for minutes or hours, interacts with dozens of tools, and needs to maintain coherent state across many observations, relying on the context window alone is a brittle design.

Context windows have grown substantially in recent years, but larger windows do not solve the underlying problem entirely. Long contexts cost more to process, they are slower, and research shows model attention degrades on material buried in the middle of very long inputs. Liu et al. (2023) demonstrated this in their "Lost in the Middle" study, finding that language models perform significantly worse on tasks where the relevant information appears in the middle of a long context compared to at the beginning or end (arXiv:2307.03172). An agent that relies purely on its context window to remember things is vulnerable to this degradation on long tasks.

The Four Memory Types

A well-designed agent uses four categories of memory, each suited to different kinds of information with different persistence, retrieval speed, and capacity characteristics.

In-Context Memory

This is the content currently loaded into the model's context window: the user's message, the conversation history, tool outputs, agent instructions, and any retrieved documents. It is the only memory the model actively reads during any given reasoning step. Everything else must be loaded here before the model can use it.

In-context memory is fast and requires no additional infrastructure. The model can access any part of the context at any time during a reasoning step. But it is temporary: once the session ends, the context is gone. It is also finite: a long task that produces many tool outputs will eventually run out of context space, and the model must discard or compress earlier content to make room for new information.

The practical consequence is that in-context memory is best for information that is immediately relevant to the current step and not needed after the session. Conversation history, current task state, and recently retrieved documents all belong in context. Long-term knowledge, past session history, and large document libraries do not.

Effective in-context memory management involves prioritizing what occupies the limited space. Recent information is usually more relevant than older information. Task-critical facts should be kept explicitly rather than relying on the model to retrieve them from the middle of a long history. A well-designed system prompt keeps the most important constraints and instructions at the top of the context, where the model's attention is strongest.

External Retrieval Memory

This is a database, document store, or vector index that the agent can query on demand. When the agent needs information that is not in its context, it calls a retrieval tool, gets the most relevant records, and loads them into the context window. The information lives outside the model permanently and is pulled in when needed.

This pattern, commonly called retrieval-augmented generation or RAG, is how agents handle large knowledge bases that would not fit in any context window. A customer support agent might have access to a database of 50,000 support articles. It cannot load all of them at once. But it can retrieve the three most relevant articles for each customer question and reason on those.

The quality of external retrieval memory depends entirely on the quality of the retrieval mechanism. A vector search that returns semantically similar documents works well for open-ended questions. A keyword search that returns documents with matching terms works well for precise lookups. Many enterprise pilots use hybrid approaches that combine both. The wrong retrieval produces wrong documents, and wrong documents lead to wrong agent reasoning: the retrieval layer is just as important as the model itself for tasks where external knowledge matters.

External retrieval memory is also where freshness matters. A vector index built from last month's documents contains last month's information. For tasks where information changes frequently, such as prices, regulations, or current events, the retrieval system needs to be updated regularly or the agent will reason on stale data.

Episodic Memory

Episodic memory is a record of what the agent has done in previous sessions. This might be stored as a log of past conversations, a summary of previous task runs, or a structured record of decisions made and their outcomes. When a new session begins, relevant episodic memory is retrieved and loaded into the context, giving the agent continuity across sessions.

The value of episodic memory for agent reliability was explored by Park et al. (2023) in their Generative Agents paper, which demonstrated that agents with access to a memory stream of past observations and reflections could maintain consistent behavior and relationships across extended interaction periods (arXiv:2304.03442). The memory stream allowed agents to remember not just facts but the context in which those facts were relevant, enabling more natural continuity.

An agent with episodic memory can pick up where it left off without requiring the user to re-establish context. A research agent that ran yesterday can read a summary of what it already found and avoid repeating the same searches. A customer service agent can recall that a customer contacted support last week about a billing issue and can skip the step of diagnosing a problem that has already been identified.

Implementing episodic memory requires explicit design decisions: what gets recorded, in what format, how long records are retained, and how they are retrieved in future sessions. A system that logs every tool call and observation verbatim will accumulate enormous data that is mostly irrelevant to future sessions. A system that records structured summaries, key decisions, and their outcomes at the end of each session provides much more useful episodic memory with far less storage. The right design depends on the use case.

Procedural Memory

Procedural memory is the agent's instructions, behavioral rules, and trained capabilities. In practice, this includes the system prompt (which defines the agent's role, available tools, and behavioral guidelines), the tool definitions (which shape how the agent interacts with the world), any fine-tuned behaviors embedded in the model weights through training, and learned preferences from human feedback processes like reinforcement learning from human feedback.

Procedural memory is what shapes how the agent acts rather than what it knows. An agent with a system prompt that says "always confirm with the user before sending external communications" has a procedural memory that constrains its behavior regardless of what other information it has. An agent fine-tuned on examples of careful, conservative decision-making has that caution embedded in its weights in a way that persists across all interactions.

Procedural memory changes more slowly than the other three types. Model weights update during training, which happens infrequently. System prompts can be updated between sessions. Tool definitions are typically stable once a system is deployed. The key insight is that good agent behavior is as much a function of well-designed procedural memory as it is of accurate retrieval or large context windows. A well-written system prompt can make a mediocre model behave reliably. A poorly written system prompt will make an excellent model behave inconsistently.

The context window is working memory. It is fast and immediate, but it has limits and it resets when the session ends. The other three memory types are what allow agents to maintain capability across tasks, sessions, and scale.

Handling Context Overflow

When a task produces more information than fits in the context window, the agent needs a deliberate strategy for what to keep and what to compress or discard. Two approaches are common in practice, and they have different tradeoffs.

Rolling Summarization

The agent, or a separate summarization step, compresses older conversation history into a compact summary. The summary replaces the full history in the context, freeing space for new information. A research agent that has completed twelve search-and-read cycles might compress the findings from the first eight into a structured summary, keeping only the most recent four cycles in full detail.

The advantage of rolling summarization is that it is automatic and requires no external infrastructure. The disadvantage is that compression loses detail. If the exact wording of an earlier instruction matters, summarization may discard it. If a fact found early in the task is crucial to a decision being made now, summarization may have compressed it into a form that is less precise. Well-designed summarization prompts explicitly instruct the model to preserve certain categories of information (exact constraints, specific quantities, named entities) while compressing narrative context.

External Note-Taking

The agent writes its own notes to an external store as it works, and retrieves them when needed. This is more precise than summarization because nothing is lost: every note written is preserved exactly. A research agent might write a structured notes file with every confirmed fact, source, and decision, then query those notes rather than trying to hold everything in context.

External note-taking adds tool calls and latency. The agent must explicitly decide what to write and when to read, which adds reasoning overhead. But for long tasks where precision matters, such as legal research, financial analysis, or multi-day planning tasks, external note-taking produces substantially more reliable results than relying on context alone or on lossy summarization.

Why Memory Design Affects Reliability

The most common agent reliability failures trace back to memory problems. An agent forgets what step it was on and repeats work it already completed. It loses track of a constraint specified early in the task and violates it later. It retrieves the wrong document and reasons on stale or irrelevant information. It starts a new session with no record of what the previous session established, forcing the user to re-explain context they already provided once.

Each of these failures has a specific memory design solution. Repeated work is addressed by episodic memory that records task progress. Lost constraints are addressed by procedural memory that keeps critical rules in the system prompt rather than the conversation history. Wrong documents are addressed by a better retrieval mechanism or fresher index. Stateless sessions are addressed by episodic memory that captures and restores session context.

Organizations deploying agents need to ask: which of the four memory types does this system use, what are the limits of each, and what happens when those limits are reached? The agents that hold up reliably in real deployments are those where memory design received the same careful attention as model selection and prompt writing. Memory is not a technical detail to add later. It is a foundational architectural decision that shapes everything the agent can reliably do.

Choosing the Right Memory Architecture for Your Use Case

Different agent use cases require different memory architectures. A customer service agent handling single-session interactions primarily needs good in-context memory management and access to a well-maintained external retrieval store. The task begins and ends in one session, so episodic memory across sessions adds complexity without benefit. The external retrieval store gives the agent access to the full knowledge base without requiring it to hold everything in context.

A research agent running multi-day investigations needs all four memory types working together. In-context memory manages the active work session. External retrieval gives access to the document corpus. Episodic memory records what has been found and what has not, so the agent can resume intelligently the next day without repeating work. Procedural memory defines the research methodology and quality standards that apply across all sessions.

A coding agent that accumulates preferences over time, learning which libraries a particular team prefers or which patterns their codebase uses, needs episodic memory designed specifically to capture and surface those preferences. Without it, every session starts cold and the agent cannot benefit from the patterns it has seen before.

The practical guidance is to start with in-context memory and external retrieval, which are the easiest to implement and the most widely supported by existing frameworks. Add episodic memory when continuity across sessions becomes a pain point in practice. Invest in procedural memory design, particularly the system prompt, from the beginning, because it shapes the agent's behavior in every session and is the highest-leverage intervention available without changing the underlying model.

The Memory and Context module in the AI Agents course covers the practical implementation of all four memory types, including how to design a retrieval system that returns the right documents, how to structure episodic memory for efficient retrieval, and how to write system prompts that serve as effective procedural memory across a range of task conditions.

Common Memory Design Mistakes and How to Avoid Them

Understanding the four memory types is necessary but not sufficient. The most instructive lessons come from the mistakes that appear consistently when teams first implement agent memory. Three patterns come up often enough to be worth naming explicitly.

The first is over-relying on in-context memory for tasks that require persistent state. A common early implementation puts everything the agent knows into the system prompt and conversation history, then discovers that the context window fills up after a few dozen interactions, at which point the agent either truncates earlier context silently or starts producing degraded output. The fix is to treat in-context memory as a working scratchpad, not a permanent store. Anything that needs to survive beyond a single session should be committed to an external retrieval system or an episodic log before the session ends.

The second mistake is building an external retrieval system with inadequate chunking. When documents are split into chunks that are too large, the retrieved text contains the target information buried inside a block of unrelated content, and the model has to work harder to extract the relevant part. Liu et al. (arXiv:2307.03172) found that models attend more reliably to information at the beginning and end of a context window than to information in the middle, which means a large retrieved chunk with the answer in the middle is more likely to be missed than a smaller chunk where the answer is near the beginning. Keeping chunks to 200 to 400 tokens, with a short overlap between adjacent chunks to preserve sentence continuity, generally outperforms larger chunks in retrieval accuracy.

The third mistake is treating procedural memory as static. System prompts are often written once, during initial development, and then not revisited until something goes wrong. But the agent's task environment evolves: new tool signatures are added, edge cases are discovered, and the range of user inputs shifts. Reviewing the system prompt periodically, the same way a software team reviews documentation, and updating it to reflect what has been learned from production use, is one of the highest-leverage maintenance tasks for an agent in active use.

Avoiding these mistakes requires treating memory as a dimension of the agent's architecture that needs its own design review, not as an implementation detail to sort out once the core loop is working. The memory layer determines what the agent can remember, what it can look up, and what it will never forget. Getting that layer right is as important as getting the reasoning prompt right.