System Atlas
the atlas
AI · AI & LLMs9 minDeep dive

How the Transformer architecture works

The architecture behind GPT, Claude, Gemini and Llama — one block at a time.

the gist

A Transformer turns tokens into embeddings, adds positional encoding, then passes them through stacked blocks of multi-head self-attention and feed-forward networks — each wrapped in a residual 'add & norm'. A final linear layer plus softmax over the vocabulary predicts the next token.

ANALOGY // Reading a sentence where every word can glance at every other word to settle its meaning — 'bank' checks whether 'river' or 'money' is nearby — then you place a bet on which word comes next.
living diagram01 / 07
Data flows bottom → top; each layer lights up in turn.
01
Output probabilities
next-token distribution
02
Linear + softmax
project over vocabulary
03
Decoder blocks
masked + cross-attention · FFN
04
Encoder blocks
self-attention · FFN · add & norm
05
Positional encoding
inject token order
06
Embeddings
tokens → vectors
07
Input tokens
sub-word IDs
01

Tokenize the input

Text is first split into tokens — sub-word chunks like 'token' plus 'ization' rather than whole words or single letters. A tokenizer maps each piece to an integer ID drawn from a fixed vocabulary of tens of thousands of entries. These IDs, not the raw characters, are the only thing the model ever actually sees.

02

Embed the tokens

Each token ID indexes into a learned embedding table, pulling out a dense vector of hundreds or thousands of dimensions. The model learns this table during training so that tokens with related meanings land near each other in the space. This is where discrete symbols become continuous numbers the network can do math on.

03

Add positional encoding

Self-attention treats its inputs as an unordered set, so 'dog bites man' and 'man bites dog' would look identical without help. A positional encoding — a signal that depends on each token's slot in the sequence — is added to every embedding to inject word order. Now the same word carries a different vector depending on where it sits.

04

Encoder blocks

Each encoder block runs multi-head self-attention, where every token forms Query, Key and Value vectors and attends to all others to gather context. A position-wise feed-forward network then transforms each token independently. Residual connections plus layer normalization wrap both sub-layers, keeping gradients stable enough to stack dozens of blocks.

05

Decoder blocks

The decoder is similar but uses masked self-attention so a token can only attend to earlier positions, never peek at the future it is trying to predict. In encoder-decoder models a cross-attention sub-layer also pulls in the encoder's representation of the input. A feed-forward network and the same add-&-norm scaffolding finish each block.

06

Linear + softmax

After the final block, the top token's vector is projected by a linear layer onto the full vocabulary, producing one raw score (a logit) per possible token. A softmax converts those logits into a proper probability distribution that sums to one. The result answers a single question: how likely is each token to come next?

07

Output probabilities

The model emits that distribution over the whole vocabulary, and a decoding strategy — greedy, temperature sampling, top-p — picks the actual next token. That chosen token is appended to the input and the entire forward pass runs again for the following token. Text is generated one token at a time, autoregressively, until a stop condition is hit.

the full picture
01

The problem the Transformer solved

Every mainstream large language model — GPT, Claude, Gemini, Llama — is, underneath the branding, a Transformer. That single architecture, introduced in the 2017 paper "Attention Is All You Need", is the reason the current wave of AI exists at all. To appreciate why it mattered, you have to know what came before it.

The previous generation of language models were recurrent neural networks (RNNs and their cousins LSTMs). They read text the way you might read a ticker tape: one token at a time, left to right, carrying a hidden "memory" vector forward as they went. This had two crippling problems. First, it was inherently sequential — you could not compute token 100 until you had computed tokens 1 through 99 — so it could not exploit the massively parallel hardware (GPUs) that makes modern training feasible. Second, information from early in a long sentence had to survive a long chain of updates to influence a decision made much later, and in practice it faded. The model struggled to connect "The keys that I left on the kitchen table yesterday" with the verb that agrees with "keys" many words later.

The Transformer's insight was to throw out recurrence entirely and replace it with a mechanism called self-attention, where every token can look directly at every other token in a single step, no matter how far apart they are. There is no long chain to traverse and no left-to-right dependency during training, so the whole sequence is processed in parallel. The rest of this article walks the data end to end — from raw text to a predicted next token — explaining each stage and, crucially, why it is there.

02

From text to vectors: tokens, embeddings, and position

A neural network only does arithmetic on numbers, so the first job is to turn text into vectors. This happens in three moves.

First, tokenization. Text is chopped into tokens — sub-word units produced by an algorithm like byte-pair encoding. Sub-words are a deliberate compromise: whole words would need an impossibly large vocabulary and would choke on anything unseen, while single characters would make sequences painfully long. Sub-words let common words be one token while rare words break into reusable pieces, and every token maps to an integer ID from a fixed vocabulary.

Second, embedding. Each token ID is used to look up a row in a learned embedding matrix, yielding a dense vector — often 768, 4096, or more dimensions. Unlike the arbitrary integer IDs, these vectors are meaningful: training arranges them so tokens with similar usage sit close together, and directions in the space capture relationships. This is the moment discrete symbols become the continuous geometry the network manipulates.

Third, positional encoding. Here is a subtlety that trips up almost everyone: self-attention has no built-in notion of order. It treats its inputs as a bag of vectors, so without help it could not tell "dog bites man" from "man bites dog". To fix this, a positional signal — either a fixed pattern of sines and cosines or a learned vector, and in modern models often a rotary scheme applied inside attention — is combined with each token's embedding. Now the vector for a word encodes both *what* it is and *where* it sits. Only after these three steps does the real processing begin.

03

Self-attention: how a token gathers context

Self-attention is the heart of the Transformer, and it is worth understanding concretely. For each token, the model computes three vectors by multiplying its embedding by three learned weight matrices: a Query, a Key, and a Value. The names come from an information-retrieval analogy. A token's Query says "here is what I am looking for." Every token's Key says "here is what I offer." And every token's Value is "here is the information I will hand over if you attend to me."

To decide how much token A should draw from token B, the model takes the dot product of A's Query with B's Key. A large dot product means the two vectors point in similar directions — the Query and Key are a good match — so B is relevant to A. These scores are computed for every pair, scaled down (dividing by the square root of the key dimension to keep them stable), and passed through a softmax so that, for each token, the attention weights over all tokens sum to one. Finally, each token's new representation is the weighted sum of all the Value vectors, using those attention weights.

Why does this "capture context"? Consider the word "bank". Its raw embedding is ambiguous. But through attention, the "bank" token's Query can match strongly against the Key of a nearby "river" or "money" token, pull in that token's Value, and emerge with a context-adjusted representation that leans toward the correct meaning. Every token, in parallel, rewrites itself as a blend of the tokens it found relevant. That is how meaning that depends on surrounding words gets baked into each position — in one parallel step rather than a long recurrent chain.

04

Multi-head attention, feed-forward, and residual add & norm

A single attention pattern can only express one kind of relationship at a time. Real language has many overlapping structures at once — syntax, coreference, subject-verb agreement, tone. The Transformer handles this with multi-head attention: instead of one set of Query/Key/Value projections, it runs several (say 8, 16, or more) in parallel, each with its own learned matrices. One head might learn to track which noun a pronoun refers to, another might follow syntactic structure, another might attend to the token immediately to the left. Their outputs are concatenated and mixed back together, giving the model many relational "views" of the sequence simultaneously.

After attention, each token passes independently through a position-wise feed-forward network — typically two linear layers with a non-linearity between them. Attention moves information *between* tokens; the feed-forward network does per-token computation, transforming each enriched representation on its own. This is where a large fraction of a model's parameters and its stored "knowledge" actually live.

Both sub-layers are wrapped in two pieces of scaffolding that make deep stacks trainable. A residual connection adds each sub-layer's input back to its output, so information (and gradient) can flow straight through even a very deep network — the layer only has to learn a refinement, not reconstruct everything from scratch. Layer normalization then rescales the result to a stable distribution, preventing the values from blowing up or vanishing as they pass through dozens of blocks. Attention, feed-forward, residual, norm — that four-part block, stacked many times, is essentially the entire model.

05

Encoder, decoder, and predicting the next token

The original Transformer had two halves. The encoder reads the whole input at once with unmasked self-attention — every token sees every other — producing a rich, bidirectional representation. This is ideal for understanding tasks (classification, retrieval embeddings), and it is the lineage behind models like BERT. The decoder generates text, and it differs in two ways: it uses masked self-attention, where a token may only attend to itself and earlier positions, and (in encoder-decoder setups like translation) a cross-attention sub-layer that lets it query the encoder's output. The mask is essential: during training the model sees the whole target sentence, and without masking it could cheat by peeking at the very next word it is supposed to predict.

Most of today's chat models — GPT, Claude, Llama — are decoder-only. They drop the encoder entirely and simply stack masked-attention blocks, treating everything, prompt included, as one sequence to continue. Whichever variant, the output stage is the same. The final token's vector is projected by a linear layer onto the full vocabulary, producing one logit per candidate token, and a softmax turns those logits into a probability distribution over what comes next.

That is the whole game: a Transformer is a next-token prediction machine. To generate, you sample a token from the distribution, append it to the input, and run the entire forward pass again — this is called autoregressive decoding. Do it repeatedly and coherent paragraphs emerge, one token at a time. The reason this simple objective produces such capable systems is scale: train a big enough stack on enough text, and accurately predicting the next token turns out to require internalizing grammar, facts, reasoning patterns, and style. Understanding this pipeline is also what makes the rest of applied AI legible — RAG changes what goes *into* the prompt, agents wrap the model in a loop, and fine-tuning adjusts these very weights.

what to remember
01

Every modern LLM — GPT, Claude, Gemini, Llama — is a Transformer at its core.

02

Input tokens become embeddings; positional encoding injects word order, which attention alone lacks.

03

Multi-head self-attention lets each token weigh the relevance of every other token, in parallel across many heads.

04

Each block adds a feed-forward network, with a residual 'add & norm' around both sub-layers to keep gradients stable.

05

The decoder uses masked self-attention plus cross-attention, then a linear layer + softmax turns the final vector into next-token probabilities.

check yourself

Quick quiz

Q1

Why is positional encoding needed at all?

Q2

What does multi-head self-attention let each token do?

follow the thread