Memory and State: Short-Term Context and Long-Term Recall

The context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète → is the block of text the model can "see" in a single request. It includes the system instructions, the conversation so far, and any data you paste in. Think of it as the agent's desk: everything on the desk is visible at once, but the desk has a fixed size.

That size is measured in tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.Voir la définition complète → (roughly ¾ of a word each). Modern models in 2026 have large windows, often 200,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.Voir la définition complète → or more, but "large" is not "infinite," and a full window is slow and expensive.

Here is the key fact that surprises people: the model has no memory between calls. Each time you send a message, the entire conversation is re-sent. The agent only "remembers" your name because your app keeps the transcript and pastes it back in every turn.

That is the entire trick behind conversational memory. There is no magic, just a growing list.

Long-term memory: a searchable store

The desk fills up. Old tickets, past chats, and account history cannot all live in the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète → forever. So we move them to long-term memory: an external store the agent can search when a topic comes up.

The common tool for this is a vector storevector storeA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète → (also called a vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète →). It stores text as embeddings: numerical fingerprints that capture meaning. When Priya writes in again, the agent turns her new message into an embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.Voir la définition complète → and asks the store, "What past tickets are most similar to this?" It pulls back only the relevant few and drops them onto the desk.

This is called retrieval: fetch the handful of memories that matter, ignore the thousands that do not. If you want the deeper mechanics, this free primer on embeddings and vector search explains it clearly without assuming you code.

A Concrete Support Agent

Let us make this real. Priya has contacted support four times over two years. She writes in today: "My blender is smoking again."

Here is what a well-designed agent does:

1. Short-term: It keeps today's conversation in the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète → so it does not re-ask her name mid-chat.

2. Long-term retrieval: It searches the vector storevector storeA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète → for Priya's past tickets. It finds ticket #4471 from last year: "blender overheating, replaced under warranty." That is highly relevant, so it loads it.

3. It ignores her unrelated ticket about a late invoice. Not similar, not loaded.

4. It responds with context: "I see we replaced this blender for overheating last March. Let's check if it's still under warranty."

That feels like a human who actually remembers you. Notice the agent did not dump all four tickets into the window. It retrieved one.

The top_k=3 is doing a lot of work. It says "give me the three best matches, not everything." That single number is often the difference between a sharp agent and one that drowns in history.

What to Keep, Summarize, or Forget

Memory is not "save everything." It is a set of decisions. Here are practical rules.

Keep (verbatim, in the window)

• The current task and the last several turns of conversation.
• Facts the agent must not get wrong: order numbers, the user's stated goal, a confirmed shipping address.
• Anything the user just said "remember this" about.

Summarize (compress, then keep the summary)

When a conversation gets long, do not carry all fifty messages. Summarize the older ones into a short note and keep only that. This is often called a "rolling summary."

Example: forty messages of back-and-forth debugging become one line: *"User troubleshooting a smoking blender; confirmed model X200; warranty status unknown."* You just freed most of the desk while keeping what matters.

Forget (drop entirely)

• Chit-chat and pleasantries. "Thanks so much!" does not need to be remembered.
• Sensitive data you should not retain (see below).
• Failed attempts once the task succeeds. The agent does not need three wrong answers cluttering its memory of a solved ticket.

A useful mental model: keep facts, summarize journeys, forget noise.

Getting the Balance Right

Two failure modes sit on either side of good memory.

Too little memory: the agent re-asks questions, loses the thread, and feels robotic. Usually caused by not persisting the transcript, or setting top_k too low.

Too much memory: the agent gets confused by stale or irrelevant history, contradicts itself, runs slowly, and costs more per message. Usually caused by dumping full history instead of retrieving, or never summarizing.

The fix for both is the same discipline: be deliberate about what enters the window. Retrieve relevant long-term memories, summarize long conversations, and forget noise. The context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète → is prime real estate. Treat every tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.Voir la définition complète → like it costs you something, because it does.

One more practical point: write memory back. After a ticket closes, save a short structured note to the vector storevector storeA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète →: *"2026-03-11: Priya, blender X200, overheating, replaced under warranty."* That is what tomorrow's agent will retrieve. An agent that never writes new memories can only ever recall what it started with.

A Note on Privacy and Frameworks

Long-term memory means you are storing user data, sometimes sensitive data. Two rules keep you out of trouble:

• Do not store what you do not need. Redact card numbers, passwords, and health details before they hit the vector storevector storeA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète →.
• Let users see and delete their memories. "Forget everything about me" should actually work. Design for it from day one.

You do not have to build all this plumbing yourself. Every major provider now ships an agent framework that handles transcripts, retrieval, and summarization for you: the OpenAI Agents SDK, the Claude Agent SDK, and Google's Agent Development Kit (ADK). They differ in the details but share the same shape you learned here: short-term context plus long-term retrieval. The provider deep-dive blocks in this course cover the specifics of each. The concepts transfer directly, so pick whichever provider you already use.

Key Takeaways

• Short-term memory is just the transcript you resend each turn. The model has no memory between calls; your app supplies it by pasting the conversation back in.
• Long-term memory is a searchable vector store. Retrieve the few relevant past items (a low top_k), not the entire history.
• Keep facts, summarize journeys, forget noise. Carry order numbers verbatim, compress long conversations into a rolling summary, and drop pleasantries.
• Write memory back after each task so future sessions can recall it, and design deletion in from the start for privacy.
• The context window is prime real estate. Both too little and too much memory break agents; the cure is being deliberate about every tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.Voir la définition complète → that enters the window.