System Atlas
the atlas

How LLMs see the world (tokens & embeddings)

To a model, your prompt isn't words — it's a list of numbers with a strict budget.

the gist

Before an LLM can process text, it splits it into tokens — subword units produced by an algorithm like BPE — and maps each to an integer ID. Those IDs become embedding vectors that place meaning in a high-dimensional space. All of this happens inside a fixed context window, and because everything is counted in tokens, tokens (not words) are what drive cost, speed and length limits.

ANALOGY // Reading a book through a fixed-size window: the text is first chopped into standard tiles, each tile is a number, and only so many tiles fit in the window at any moment.
living diagram01 / 06
Raw text"unhappiness"TokenizerBPE subwordsToken IDsintegersEmbeddingsmeaning vectorsContext windowtoken budgetModelattention
01

Start with raw text

Everything starts as a raw string of characters — your prompt, a document, a line of code. The model, though, has no notion of letters or words; it needs the text converted into a form a neural network can consume. That conversion is the tokenizer's job, and it runs before the model sees anything.

02

Tokenize into subwords (BPE)

The tokenizer splits text into tokens — subword units — using an algorithm like Byte-Pair Encoding (BPE). BPE builds its vocabulary by repeatedly merging the most frequent character pairs, so common words become a single token while rare words break into pieces: 'unhappiness' might split into 'un', 'happi', 'ness'. This keeps the vocabulary manageable while still being able to represent any word, even ones never seen in training.

03

Map tokens to integer IDs

Each token in the vocabulary has a fixed integer ID, so the tokenized text becomes a list of numbers — for example [1200, 318, ...]. This is the actual input the model receives: not words, but a sequence of token IDs. The same mapping runs in reverse to turn the model's output IDs back into readable text.

04

Turn IDs into embedding vectors

An ID by itself is just an arbitrary index, so each one is looked up in an embedding table and replaced with an embedding vector — a list of hundreds or thousands of numbers. These vectors are learned during training so that tokens with related meanings end up close together in the space, which is how the model represents meaning rather than mere spelling. From here on, the model works entirely with these vectors.

05

It all lives in the context window

The model can only attend to a fixed number of tokens at once — its context window (say 8K, 128K or more). Everything must fit inside this budget: the system prompt, the conversation history, any retrieved documents, and the response being generated all share the same window. When a conversation grows past the limit, the oldest tokens must be dropped or summarised, which is why models can 'forget' earlier parts of a long chat.

06

Why tokens drive cost and limits

Because the model counts everything in tokens, tokens — not words — are the true unit of cost, speed and limits. API providers bill per token, latency scales with token count, and context limits are stated in tokens (roughly 4 characters, or about three-quarters of a word, in English). This is why concise prompts are cheaper, why code and unusual languages can cost more tokens per word, and why token budgeting is a real engineering concern.

the full picture
01

Why models need tokens at all

A neural network is, at bottom, a giant function that multiplies matrices of numbers. It has no innate concept of the letter "a", the word "cat", or the sentence you just typed. So the very first thing that happens to any text you send an LLM is a translation step: the messy, variable-length string of human characters has to be turned into a tidy sequence of numbers the network can crunch. That translation is tokenization, and understanding it explains a surprising amount about how models behave — why they sometimes miscount letters, why some languages cost more, and why there's a hard limit on how much you can feed them.

The naive approaches both fail. Giving each word its own number produces an enormous vocabulary that still can't handle typos, new words, or the endless combinations of real language. Giving each character its own number keeps the vocabulary tiny but makes sequences painfully long and forces the model to relearn spelling from scratch. Modern LLMs take the middle path: subword tokenization, which captures the best of both.

02

Tokenization: how text becomes subwords

The dominant technique is Byte-Pair Encoding (BPE) and its close relatives. BPE builds its vocabulary from data: it starts with individual characters and then repeatedly finds the most frequent adjacent pair and merges it into a new single token, over and over, until it has a vocabulary of a fixed size (often 50,000 to 100,000 tokens). Frequent sequences like "ing", "the", or " and" get merged early and become single tokens; rare sequences never do.

The result is elegantly adaptive. Common words end up as exactly one token, so they're cheap to represent. Rare or novel words gracefully fall back to being spelled out of smaller pieces — "tokenization" might become "token" + "ization", and an unseen surname simply splits into whatever subword chunks it's made of. This means the tokenizer can represent any string, including words, emoji or code it never encountered in training, without ever hitting an "unknown word" — a property called an open vocabulary.

A few practical consequences fall out of this. English text averages roughly 4 characters per token, or about three-quarters of a word. Whitespace and capitalisation matter — " cat" and "cat" can be different tokens. And because the vocabulary is optimised for the training data (largely English), text in other languages, or dense code, often fragments into more tokens per word, which directly affects cost and how much fits in the window.

03

Token IDs and embeddings: from integers to meaning

Once text is split into tokens, each token is mapped to its integer ID — its index in the vocabulary. At this point your sentence is literally a list of numbers, and that list is what the model actually receives. When the model finishes, it emits token IDs that the same tokenizer maps back into readable text.

But an integer index carries no meaning on its own — token 5,342 isn't "bigger" or "closer" to token 5,343 in any useful sense. So the first thing the model does with each ID is look it up in an embedding table and replace it with an embedding vector: a list of hundreds or thousands of floating-point numbers. These vectors are learned during training, and they arrange themselves so that tokens used in similar contexts land near each other in the high-dimensional space. The famous demonstration is that vector arithmetic captures relationships — the direction from "king" to "queen" mirrors the one from "man" to "woman".

This is the real answer to "how does an LLM see the world": as points in a geometric space of meaning. Everything the model reasons about — grammar, analogy, topic, sentiment — is expressed as movements and relationships between these vectors. The transformer's job (covered in the transformers and how-LLMs-work concepts) is to keep refining them layer by layer until it can predict the next token.

04

The context window: a finite working memory

All of this processing happens inside a fixed-size context window: the maximum number of tokens the model can consider at once. Early models had windows of a couple of thousand tokens; newer ones stretch to hundreds of thousands, but the limit is always there, and it is a hard wall, not a soft suggestion.

The window is a shared budget, and everything competes for it: the hidden system prompt, the full back-and-forth of the conversation so far, any documents you've pasted or a RAG system has retrieved, and the space reserved for the model's own answer. When a long conversation or document exceeds the window, something has to give — the earliest messages get truncated or summarised — which is precisely why a model can seem to "forget" what you said at the start of a very long chat. It isn't forgetting; those tokens have simply fallen outside what it can see.

Larger windows are enormously useful but not free. The attention mechanism's cost grows quickly with sequence length, so doubling the context can more than double the compute, and research consistently shows models use the middle of a very long context less reliably than the beginning or end. A big window is an opportunity, not a guarantee that everything in it gets equal attention.

05

Why token counts run your bill and your limits

Because the model measures and processes everything in tokens, tokens are the fundamental unit of working with LLMs — and this has direct, practical consequences that trip up newcomers who think in words.

  • Cost. Commercial APIs bill per token, for both the input you send and the output you receive. A prompt that's twice as long costs roughly twice as much; stuffing an entire document into context on every request adds up fast.
  • Speed. Generation is token by token, so a longer response literally takes more forward passes and more time. Latency scales with the number of tokens produced.
  • Limits. Every model's maximum length is quoted in tokens, and it counts input and output together against the window.

Two subtleties matter in practice. First, tokens don't map cleanly to words — one word can be several tokens, so "estimate about 1.3 tokens per English word" is a better rule of thumb than counting words. Second, that ratio worsens for non-English languages and for code, which fragment into more tokens, making them proportionally more expensive and quicker to fill the window. All of this is why prompt engineering has a quiet cousin, token budgeting: writing prompts and designing systems (compressing history, chunking retrieved text, trimming boilerplate) so you get the most capability for the fewest tokens. Once you see text the way the model does — as a costly, finite stream of tokens — a lot of the practical craft of building with LLMs starts to make sense.

what to remember
01

LLMs don't read characters or words — they read tokens, which are subword units.

02

Tokenizers like BPE build a vocabulary of frequent chunks, so common words are one token and rare ones split into several.

03

Each token maps to an integer ID, then to an embedding vector where similar meanings sit close together.

04

The context window is the fixed maximum number of tokens the model can attend to at once.

05

Cost, latency and length limits are all measured in tokens, not words — roughly 4 characters, or about three-quarters of a word, each in English.

check yourself

Quick quiz

Q1

What is a 'token' in an LLM?

Q2

Why are token counts, not word counts, what drive cost and limits?

follow the thread