Transformer
Also: Transformer architecture, Transformer model, Attention-based model
A Transformer is a neural network architecture that uses self-attention to process sequences in parallel, powering most modern language and generative AI models.
What it is
A Transformer is a deep learning architecture introduced in 2017 in the paper "Attention Is All You Need." It processes sequential data (such as text, code, or audio tokens) using a mechanism called self-attention rather than the step-by-step recurrence used by earlier models like RNNs and LSTMs. This lets it weigh the relationships between all elements of a sequence at once, regardless of how far apart they are.
Why it matters
Before Transformers, sequence models processed tokens one at a time, which was slow to train and weak at capturing long-range dependencies. Transformers changed this by:
- Parallelizing computation: all tokens are processed simultaneously, making training on large datasets and hardware (GPUs/TPUs) far more efficient.
- Scaling well: performance keeps improving as data, parameters, and compute grow, which is the foundation of large language models (LLMs).
- Generalizing across domains: the same core design now powers text, images (Vision Transformers), audio, and multimodal systems.
The Transformer is the backbone of well-known model families such as GPT, BERT, T5, and many open models.
How it works (in brief)
- Tokenization and embeddings: input is split into tokens and mapped to numeric vectors.
- Positional encoding: since attention has no built-in order, position information is added so the model knows token sequence.
- Self-attention: each token computes Query, Key, and Value vectors to decide how much to "attend" to every other token.
- Multi-head attention: several attention computations run in parallel to capture different relationships.
- Feed-forward layers and residual connections: stacked blocks refine the representation, with normalization for stable training.
Variants include encoder-only (good for understanding, e.g. classification), decoder-only (good for generation, e.g. chatbots), and encoder-decoder (good for translation or summarization).
A concrete example
Consider the sentence: "The trophy did not fit in the suitcase because it was too big." A Transformer uses self-attention to link "it" to "trophy" rather than "suitcase," using context from the whole sentence. In practice, a customer support team might fine-tune a decoder-only Transformer on past tickets so it drafts accurate replies, while a finance team might use an encoder model to classify transaction descriptions at scale.
See also
Frequently asked questions
What is a Transformer in AI?
A Transformer is a deep learning architecture, introduced in 2017 in the paper "Attention Is All You Need", that processes sequences such as text, code or audio tokens using self-attention instead of the step-by-step recurrence of RNNs and LSTMs. It weighs the relationships between all tokens of a sequence at once, whatever the distance between them. It is the backbone of model families like GPT, BERT and T5.
Why did Transformers replace RNNs and LSTMs?
Because they train far faster and handle long-range dependencies better. RNNs and LSTMs process tokens one at a time, which is slow and loses track of distant context; a Transformer processes all tokens simultaneously, so training can exploit GPUs and TPUs efficiently. Performance also keeps improving as data, parameters and compute grow, which is what made large language models possible.
What is the difference between encoder-only, decoder-only and encoder-decoder Transformers?
They differ by task. Encoder-only models (BERT-style) are suited to understanding tasks such as classification; decoder-only models (GPT-style) are suited to generation, hence chatbots; encoder-decoder models (T5-style) fit translation and summarization, where an input sequence is converted into an output sequence. The core self-attention design is the same in all three.
How does self-attention actually work?
Each token produces three vectors, Query, Key and Value, and uses them to decide how much attention to pay to every other token in the sequence. Several of these computations run in parallel (multi-head attention) to capture different types of relationships, then feed-forward layers, residual connections and normalization refine the representation across stacked blocks. Because attention carries no notion of order, positional encoding is added so the model knows the token sequence.
Can a Transformer handle anything other than text?
Yes. The same core architecture now powers images through Vision Transformers, audio, and multimodal systems that combine several types of input. That generality is one reason the design spread so quickly beyond natural language processing.