How a large
language model works

Most tokenizers learn their vocabulary with byte-pair encoding: start from single characters and repeatedly merge the most frequent adjacent pair, until the vocabulary hits its target size — tens of thousands of pieces. Frequent words earn their own token; rare words get spelled out in chunks, so any text can be represented. Rule of thumb: about 4 characters ≈ 1 token in English. The tokenizer is frozen before training and shared by the model forever — which is why a model can never learn a new word's spelling after the fact.

The lookup table is the embedding matrix — vocabulary-size × model-width, thousands of numbers per token in real models — and it is learned during training: words appearing in similar contexts get pushed together. Position used to be added as fixed sine waves or learned offsets; modern models bake rotation into the attention step itself (RoPE) — and stretch it to huge contexts with rescaling tricks like YaRN, one way open models reach quarter-million-token windows. The resulting vector joins the residual stream: one vector per token that flows through the entire network, refined at every block.

Every block has the same two moves — multi-head self-attention, then the feed-forward net — each in a residual + normalization sandwich. Blocks share a shape but not weights, so early blocks tend to handle surface patterns (word shape, grammar) while later blocks handle abstract ones (logic, long-range references). Nearly all of the model's parameters — and nearly all the compute — live in this stack.

One more 2024–2026 twist: the biggest models are mixture-of-experts. Each block's feed-forward net is split into dozens or hundreds of specialist sub-networks (“experts”), and a tiny router wakes only a handful per token — plus one shared expert that always runs. Moonshot's open-weight Kimi K2 reports about a trillion total parameters with only ~32B active per token; DeepSeek's V4-Pro reports ~1.6T total with ~49B active. Parameters scale up; the bill per token barely moves.

This is the unembedding: a d_model × vocabulary matrix — sometimes literally the same weights as the input embedding, tied. One matrix multiply turns one vector into ~100k scores. During training this happens at every position at once (each position predicts the token after it); during generation, only the last position matters.

p_i = e^z_i / Σ_j e^z_j. Because of the exponential, the top few tokens usually hog nearly all the probability while the long tail shares crumbs. Softmax has no learned parameters — it's pure arithmetic, and it's the same formula used inside the attention step back in stage 3.

Temperature T rescales the logits before softmax: p_i ∝ e^z_i/T. As T→0 this approaches argmax — greedy decoding, always the top token; T=1 leaves probabilities untouched; T>1 flattens them. Top-k keeps only the k highest-scoring tokens; top-p (nucleus sampling) keeps the smallest set whose probabilities sum to p. Chatbots typically sample around T≈0.7 with top-p≈0.9: lively, but the truly weird options never get a ticket. Reasoning models add one more knob — thinking effort (low → max in current APIs): how many tokens the model may spend reasoning before it answers. More thinking, better answers, bigger bill — inference-time compute as a dial, not a fixed cost.

Recomputing everything each step would be quadratic; the KV cache avoids it by saving every token's keys and values after its first pass, so each new step only computes the newcomer. The context window is how many tokens the model can juggle at once — a few thousand in early models, a million in current frontier ones, afforded by the compressed-attention tricks from stage 3; past the limit, the oldest tokens fall off. Generation halts on a learned end-of-text token or a length cap. And the loop no longer runs alone: modern systems call tools mid-generation — running code, searching the web, checking a calculator — which multiplies what the raw next-token engine can do without changing its weights.