The context window: the model's working memory

An LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → (large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →, the technology behind ChatGPT, Claude, and Gemini) has no memory between conversations. When you start a new chat, the model knows nothing about your last one. It doesn't "remember" you the way a coworker would.

Everything the model "knows" during a chat lives in one place: the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition →.

Think of the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition → as the model's working memory, like the desk space in front of you. Whatever fits on the desk, the model can see and use. Anything that falls off the edge is gone.

This includes:

• Your messages
• The model's replies
• Any files or text you pasted in
• Hidden system instructions the app adds behind the scenes

All of it sits on the same desk. And the desk has a fixed size.

TokensTokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →: the budget you're spending

The size of the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition → is measured in tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, not words.

A tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → is a chunk of text. Roughly, 1 token is about 4 characters, or three-quarters of a word. So 100 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → is around 75 words.

Quick examples:

• "cat" = 1 tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →
• "unbelievable" = often 3 or 4 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → (broken into pieces)
• A typical page of text = about 500 to 700 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →

You can see this yourself with OpenAI's free Tokenizer tool. Paste any text and watch it split into colored tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. It's the fastest way to make this concept click.

Why does this matter? Because the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition → is a budget, and tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → are the currency.

In early 2026, typical limits look like this:

• ChatGPT (GPT-5 class models): around 128,000 to 256,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → depending on the tier
• Claude (Sonnet and Opus 4.x): 200,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, with some enterprise options far higher
• Gemini 2.x: up to 1,000,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → or more in some versions

A million tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → sounds infinite. It isn't. Let's do the math.

Why the 40-page PDF works (and the novel doesn't)

A 40-page PDF is roughly 20,000 to 28,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. Drop that into a 200,000-token windowtoken windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition → and you've used maybe 12 percent of the desk. Plenty of room left for your questions and the model's answers.

A 900-page novel is roughly 400,000 to 500,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. That overflows even a 200,000-token windowtoken windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition →. The app rejects it because it physically cannot fit on the desk.

This is the practical rule:

> If your content fits in the window, the model can reason over all of it at once. If it doesn't, you have to break it up.

When something is too big, your options are:

1. Split it into chunks and process them one at a time.

2. Summarize as you go, feeding summaries forward instead of full text.

3. Use a model with a bigger window (this is exactly why Gemini's million-token windowtoken windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition → is useful for huge documents).

Why long chats start to "forget" the beginning

Here's the behavior that confuses people most.

You're 50 messages deep into a long brainstorming chat. You ask the model to recall a detail you mentioned at the very start. It gets it wrong, or invents something.

The model isn't being lazy. The beginning of your conversation fell off the desk.

Remember: every message, yours and the model's, takes up tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → budget. A long conversation keeps adding tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. When the total exceeds the window, the oldest content gets pushed out to make room for the newest.

So in a very long chat, the model may literally no longer see what you said an hour ago. It's not in the working memory anymore.

A simple mental model:

This is also why the model can contradict itself in long sessions. The instruction you gave at message 3 ("always reply in British English") may be gone by message 80.

"But ChatGPT remembers my name across chats?"

Good catch. Modern apps add features on top of the raw context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.View full definition →:

• Memory (ChatGPT, Gemini): the app saves a few facts about you and quietly re-inserts them into new chats.
• Projects (Claude, ChatGPT): a shared space where instructions and files persist across conversations.

These aren't the model remembering. They're the app re-feeding information into the context window each time. The window is still the only thing the model actually sees.

How to work with the window, not against it

Once you treat tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → as a budget, you start making better choices. Here's the practical mindset.

1. Start fresh for new topics

Don't run your whole week through one endless chat. A bloated chat wastes budget on irrelevant history and increases the chance of confusion. New task, new chat.

2. Front-load what matters

Put your most important instructions and reference material near your actual question, not buried 40 messages up. If a rule matters, restate it.

3. Paste the relevant part, not everything

You don't need to paste a full 200-page contract to ask about the termination clause. Paste the clause and a bit of surrounding context. Less noise, sharper answers, lower cost.

4. Summarize to "save" a long chat

When a long conversation gets sluggish or forgetful, ask:

> "Summarize everything we've decided so far as a bullet list I can paste into a new chat."

Then start a clean chat with that summary on top. You've compressed hours of work into a few hundred tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →.

5. Use Projects for ongoing work

If you keep returning to the same material (a book draft, a product spec, a client account), put it in a Project so it's always loaded without re-pasting.

A concrete example: counting your budget

If you build with the APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → (the developer interface to these models), tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → usage is reported to you directly. Even non-developers benefit from seeing it once.

# pip install anthropic
import anthropic

client = anthropic.Anthropic()  # uses your API key

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    messages=[
        {"role": "user", "content": "Summarize the causes of WWI in 5 bullets."}
    ],
)

print(response.content[0].text)

# The model tells you exactly how much budget you spent:
print("Input tokens:", response.usage.input_tokens)
print("Output tokens:", response.usage.output_tokens)

input_tokens is what you sent (your prompt plus any pasted files). output_tokens is what the model wrote back. Both count against limits and, on paid plans, against cost. Watching these numbers makes the whole concept tangible.

Common misconceptions, cleared up

"A bigger context window means a smarter model." No. It means a bigger desk, not a sharper mind. A model with a huge window can still misread what's on the desk.

"The model reads my whole document carefully every time." Not exactly. With very large inputs, models can overlook details buried in the middle, a known pattern sometimes called "lost in the middle." Important instructions belong near the start or end of your input.

"Once I tell it something, it knows it forever." Only within that chat, and only while it stays in the window. Across chats, only Memory or Projects carry it forward.

Key Takeaways

• The context window is the model's working memory. Everything the model can "see" (your messages, its replies, pasted files) lives there, and it has a fixed size measured in tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →.
• Tokens are a budget: roughly 75 words per 100 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →. A 40-page PDF fits easily in a 200,000-token window; a full novel does not.