Token

What a token is

A token is the smallest unit of text that a language model reads and generates. Tokens are not always whole words. Depending on the model's tokenizer, a token can be a full word, a piece of a word (a subword), a single character, or a punctuation mark. For example, the word "unbelievable" might split into tokens like `un`, `believ`, and `able`, while a common word like "the" is usually a single token.

Models do not see raw text. The tokenizer first converts text into a sequence of token IDs (integers), and the model works entirely with those numbers. After generating output token IDs, the tokenizer converts them back into readable text.

Why tokens matter

Tokens are the unit of measurement for almost everything practical about working with AI models:

• Cost: Most API providers bill per token, separately for input (prompt) and output (completion).
• Context limits: Every model has a maximum context window measured in tokens (for example 8,000, 128,000, or more). Prompt plus response must fit inside it.
• Latency: Generation time scales roughly with the number of output tokens.
• Quality control: Truncation, summarization, and chunking strategies all depend on token counts.

How tokens are used in practice

A rough rule of thumb for English is that 1 token is about 4 characters or 0.75 words, so 1,000 tokens is roughly 750 words. Different languages and scripts tokenize differently, so non English text often uses more tokens per word.

When building applications, teams:

• Count tokens before sending requests to stay within limits and estimate cost.
• Split long documents into token sized chunks for retrieval systems.
• Set a `max_tokens` parameter to cap response length.
• Monitor token usage to forecast spend.

Concrete Example

The sentence "AI changes how we work." might tokenize into seven tokens: `AI`, ` changes`, ` how`, ` we`, ` work`, `.`, plus an end marker. Note the leading spaces are part of the tokens. If a provider charges 0.50 USD per million input tokens, processing one million such sentences (about 7 million tokens) would cost roughly 3.50 USD. Understanding tokens lets you predict and control both behavior and budget.