Token
Also: Tokens, Subword unit, BPE token
A token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.
What a token is
A token is the smallest unit of text that a language model reads and generates. Tokens are not always whole words. Depending on the model's tokenizer, a token can be a full word, a piece of a word (a subword), a single character, or a punctuation mark. For example, the word "unbelievable" might split into tokens like `un`, `believ`, and `able`, while a common word like "the" is usually a single token.
Models do not see raw text. The tokenizer first converts text into a sequence of token IDs (integers), and the model works entirely with those numbers. After generating output token IDs, the tokenizer converts them back into readable text.
Why tokens matter
Tokens are the unit of measurement for almost everything practical about working with AI models:
- Cost: Most API providers bill per token, separately for input (prompt) and output (completion).
- Context limits: Every model has a maximum context window measured in tokens (for example 8,000, 128,000, or more). Prompt plus response must fit inside it.
- Latency: Generation time scales roughly with the number of output tokens.
- Quality control: Truncation, summarization, and chunking strategies all depend on token counts.
How tokens are used in practice
A rough rule of thumb for English is that 1 token is about 4 characters or 0.75 words, so 1,000 tokens is roughly 750 words. Different languages and scripts tokenize differently, so non English text often uses more tokens per word.
When building applications, teams:
- Count tokens before sending requests to stay within limits and estimate cost.
- Split long documents into token sized chunks for retrieval systems.
- Set a `max_tokens` parameter to cap response length.
- Monitor token usage to forecast spend.
Concrete Example
The sentence "AI changes how we work." might tokenize into seven tokens: `AI`, ` changes`, ` how`, ` we`, ` work`, `.`, plus an end marker. Note the leading spaces are part of the tokens. If a provider charges 0.50 USD per million input tokens, processing one million such sentences (about 7 million tokens) would cost roughly 3.50 USD. Understanding tokens lets you predict and control both behavior and budget.
See also
Frequently asked questions
What exactly is a token in a language model?
A token is the smallest unit of text a language model reads and generates. It is not always a whole word: depending on the tokenizer, a token can be a full word, a word fragment (subword), a single character or a punctuation mark. "the" is usually one token, while "unbelievable" may split into pieces like `un`, `believ` and `able`.
Why should a manager care about tokens rather than words?
Because tokens, not words, are the billing and capacity unit of AI models. API providers charge per token, input and output counted separately; every model has a context window expressed in tokens; and generation time grows roughly with the number of output tokens. Budget forecasts and prompt design both rest on token counts.
How do I convert words into tokens for a quick estimate?
For English, use the rule of thumb that 1 token is about 4 characters or 0.75 words, so 1,000 tokens is roughly 750 words. It stays an approximation: other languages and scripts tokenize differently and often consume more tokens per word. For anything that touches cost or context limits, count tokens with the model's own tokenizer instead of estimating.
What is the difference between input tokens and output tokens?
Input tokens are the ones you send (the prompt); output tokens are the ones the model generates (the completion). Providers usually bill them at different rates, and both must fit inside the model's context window together. Output tokens also drive latency, which is why `max_tokens` is used to cap response length.
How does a team manage token usage in a real application?
Four practices cover most of it: counting tokens before sending a request to stay within the context window and estimate cost, splitting long documents into token sized chunks for retrieval systems, setting `max_tokens` to cap response length, and monitoring consumption to forecast spend. Truncation, summarization and chunking strategies all come back to token counts.