How Do LLMs Work? A Technical Overview

The Core Task: Next-Token Prediction

Every large language model, regardless of the specific training recipe, shares the same fundamental objective during pretraining: given a sequence of tokens, predict what comes next. This is called the autoregressive language modeling objective, and it is deceptively simple. The model receives tokens one through N, and its job is to output a probability distribution over the vocabulary for position N+1.

The power of this objective is that it is self-supervised. No human has to label anything. The training data is the supervision: the correct answer at each position is whatever token actually appears next in the text. With enough data and enough parameters, the model learns to internalize the statistical regularities of language at every scale, from character patterns to grammatical rules to factual associations to long-range narrative structure.

What makes this surprising is how much capability falls out of that single objective. A model trained only on next-token prediction learns to write grammatical sentences, answer questions, write code, reason through math problems, and carry on coherent multi-turn conversations. None of those capabilities were directly specified. They are all consequences of predicting text well.

The Four Layers of the Stack

To understand how a prompt becomes a response, it helps to break the process into four distinct layers, each of which is covered in depth in the LLMs from Scratch course.

Tokenization converts raw text into a sequence of integer IDs. Most modern models use byte-pair encoding (BPE) or a variant. The vocabulary is typically 32,000 to 100,000 subword units. A word like "transformers" might become a single token, or it might split into "transform" and "ers" depending on how frequently those sequences appear in the training corpus.

Embedding maps each token ID to a dense vector of floating-point numbers. A vocabulary of size V and an embedding dimension of D produces an embedding matrix of shape (V, D). This matrix is learned during training. The embedding also includes a positional signal, either learned, sinusoidal, or a rotation-based encoding like RoPE, so the model can distinguish token position within the sequence.

The transformer stack processes the sequence of embedded vectors through N identical layers. Each layer contains a self-attention sub-layer and a feed-forward sub-layer, each followed by a residual connection and layer normalization. The attention mechanism allows every token to gather information from all preceding tokens. The feed-forward layers apply a position-wise nonlinear transformation that stores and retrieves factual associations.

The language model head takes the final hidden state and projects it to a vector of size V (vocabulary size), producing a logit for every possible next token. Softmax over those logits gives the probability distribution. During training, the model is penalized by the cross-entropy loss between that distribution and the actual next token. During inference, a token is sampled from the distribution using greedy decoding, top-k, or nucleus sampling.

Why the Transformer Succeeded Where Earlier Architectures Did Not

Before the transformer (Vaswani et al., 2017), language models were predominantly recurrent. An RNN or LSTM processes tokens one at a time, maintaining a hidden state that is updated at each step. That sequential structure makes it difficult to parallelize training across many GPUs, and it makes it hard for the model to retain information over long sequences, because the hidden state must compress the entire preceding context into a fixed-size vector.

The transformer replaced recurrence with attention. Instead of passing information through a sequential hidden state, attention allows every token to directly query every other token in the sequence in a single operation. The computation is expressed as matrix multiplications, which GPUs are extremely well suited to perform in parallel. Training that would take weeks on an LSTM became feasible in days on transformer architectures at the same compute budget.

The transformer did not make language models smarter in any fundamental sense. It made training at scale feasible, and scale turned out to be the main variable that mattered.

The Role of Scale

The most important empirical finding in LLM research over the past five years is that model quality, measured by perplexity on held-out text, improves predictably with scale. Kaplan et al. (2020) showed that loss follows a power law with respect to the number of parameters, the number of training tokens, and the compute budget. Doubling parameters reliably reduces loss by a measurable and predictable amount.

Hoffmann et al. (2022), the Chinchilla paper, refined this by showing that prior models had allocated too much of their compute budget to parameters and not enough to data. For a given compute budget C, the optimal strategy is to use roughly equal amounts of compute for parameters and for training tokens. A model with N parameters should be trained on approximately 20 times N tokens to be compute-optimal.

This means that a 70-billion-parameter model trained on 1.4 trillion tokens (20 times 70 billion) will outperform a 140-billion-parameter model trained on 700 billion tokens at the same total compute cost. The insight shifted the field: models released after 2022 are generally smaller and more data-efficient than their predecessors.

From Pretraining to Deployment

A pretrained language model is not yet a useful assistant. It is a next-token predictor, which means that if you give it the beginning of a news article, it will try to complete the article. If you ask it a question, it may try to continue with more questions rather than an answer, because that is what questions look like in the training data.

Converting a pretrained model into an instruction-following assistant requires fine-tuning. The standard pipeline starts with supervised fine-tuning (SFT) on a dataset of high-quality prompt-response pairs, which teaches the model to respond to instructions rather than to continue text. This is followed by reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO), which further aligns the model's outputs with human preferences for helpfulness, accuracy, and safety.

The result is a model that behaves like a conversational assistant while retaining the knowledge and language capabilities acquired during pretraining. The fine-tuning data is typically orders of magnitude smaller than the pretraining corpus: a few million examples compared to trillions of tokens. Fine-tuning shapes behavior; pretraining builds capability.

What LLMs Are Not

It is worth being precise about what the architecture does not do, because misconceptions about LLM capabilities lead to predictable engineering mistakes.

An LLM does not have persistent memory across conversations unless you explicitly inject prior context into the prompt. Each inference call starts fresh with whatever tokens appear in the context window. The model has no record of previous interactions unless they are included in the input.

An LLM does not reason by following a procedure. It generates tokens whose statistical pattern resembles reasoning. Chain-of-thought prompting improves accuracy on reasoning tasks because it changes the token distribution in ways that correlate with more careful analysis, not because it triggers a separate reasoning engine.

An LLM does not have access to information after its training cutoff unless retrieval is added. The model's knowledge is a compressed statistical summary of the training data, not a live database. This is why retrieval-augmented generation (RAG) is the standard pattern for use cases that require current information.

Understanding these constraints is the starting point for designing systems that use LLMs reliably, rather than treating them as oracles that will always produce correct answers.

Inference: How a Response Is Generated

During inference, the model receives a prompt as a token sequence and generates a response one token at a time. At each step, the full context window (prompt plus tokens generated so far) is passed through the transformer stack, and the LM head produces a probability distribution over the vocabulary. A token is sampled from that distribution and appended to the context, becoming part of the input for the next step.

This process is called autoregressive generation, and its sequential nature creates a latency bottleneck. You cannot generate token N+1 until token N exists, because token N is part of the input that produces the distribution over token N+1. This means that response latency grows linearly with the number of output tokens, regardless of how many GPUs are in use. The KV cache mitigates this by storing the key and value tensors from prior tokens so they do not need to be recomputed at each step, but the sequential dependency remains.

Sampling strategy matters for output quality and consistency. Greedy decoding selects the highest-probability token at each step. It is deterministic but tends to produce repetitive outputs because it never explores lower-probability continuations. Temperature scaling adjusts the sharpness of the distribution: temperatures below 1.0 make the model more confident, while temperatures above 1.0 make it more exploratory. Nucleus sampling restricts each step to the smallest set of tokens whose cumulative probability exceeds a threshold p, combining quality with diversity.

Context Windows: What the Model Can and Cannot See

Every transformer architecture has a maximum context length, determined by the positional encoding scheme and the memory needed to store the attention matrices. Early models supported contexts of 2,048 tokens. Modern architectures support 128,000 tokens or more, with some systems claiming multi-million-token contexts through various approximations.

A common misconception is that longer context windows allow the model to use all information equally. In practice, models show a documented effect where information near the beginning and end of a long context is retrieved more reliably than information in the middle. For applications that require the model to find specific facts inside a large document, retrieval systems that identify the relevant passage before passing it to the model often outperform naive long-context approaches.

Context length also has a direct effect on cost. Self-attention scales quadratically with sequence length: a context that is twice as long requires four times the attention computation. At 128,000 tokens, the attention computation alone accounts for a substantial fraction of total inference cost. This is why sliding window attention, linear attention, and state space model architectures have attracted research interest as alternatives to full quadratic attention for very long sequences.

How LLMs Fail: Three Patterns Worth Understanding

Understanding failure modes is as important as understanding capabilities. The most common failure patterns in deployed LLMs are hallucination, prompt sensitivity, and distribution shift.

Hallucination refers to the model producing confident, fluent, and factually incorrect statements. The mechanism is not mysterious: the model is a next-token predictor, and the tokens that look like a correct answer are also the tokens that a fluent, confident response would contain, regardless of factual accuracy. Models are trained to produce plausible text, and plausible text often resembles confident claims. Retrieval-augmented generation, where the relevant source passage is included in the prompt, is the most widely deployed mitigation for knowledge-dependent tasks.

Prompt sensitivity means that small, semantically irrelevant changes to a prompt can produce large changes in the output. Changing the order of examples, rephrasing a question, or adjusting punctuation can shift the model from a correct to an incorrect answer on the same underlying task. This is a consequence of the model's sensitivity to the token distribution of the input: the composition of the prompt shifts the probability distribution over the response in ways that are not always predictable from the semantics alone.

Distribution shift occurs when the task or domain at inference time differs from what the model encountered in training. A model trained predominantly on formal English web text may perform poorly on code-switched text, regional dialects, or domain-specific vocabulary. Fine-tuning on domain-representative examples is the standard approach to this problem, but it requires careful evaluation to confirm that fine-tuning improved the target task without degrading performance on adjacent tasks.

These failure modes matter especially in enterprise contexts, where applications require consistent behavior across a wide range of inputs. A model that produces correct outputs 95% of the time on a task with thousands of invocations per day will produce hundreds of errors daily. Evaluating reliability across edge cases, not just average-case quality, is the essential discipline for responsible deployment. The MEDFIT-LLM study (Rao, Jaggi, Naidu, IEEE RMKMATE 2025, DOI:10.1109/RMKMATE64574.2025.11042816) illustrates this principle, showing how careful evaluation design in a medical domain exposes failure modes that aggregate accuracy metrics would conceal.

The Cost Dimension: FLOPs, Tokens, and Efficiency

Every inference call has a compute cost measured in floating-point operations. For a transformer with N parameters, generating one token requires approximately 2N multiply-accumulate operations, because each parameter participates in one forward pass. Generating a 500-token response with a 70-billion-parameter model requires roughly 70 billion multiplied by 2 multiplied by 500, which is approximately 70 trillion floating-point operations.

The practical implications for enterprise deployments are significant. Inference cost scales with both model size and output length. Chen et al. (FrugalGPT, arXiv:2310.11409) showed that routing different queries to different model sizes based on difficulty can reduce inference cost substantially without degrading overall quality. The core insight is that not all queries require a frontier model: simple queries answered well by a smaller model should be routed to that model, with larger models reserved for tasks where capability genuinely matters. This is the economic logic behind the cascade architectures now common in enterprise AI systems.

Understanding the full stack, from tokenization to sampling to cost structure, is what separates engineers who can deploy LLMs reliably from those who are perpetually surprised by the gaps between benchmark performance and real-world behavior.

The key mental model is to treat the LLM as a component in a system, not as the system itself. A well-designed LLM application specifies what the model is responsible for, what it is not responsible for, and how errors from the model are caught and handled by surrounding components. The pretrained base model provides the language capabilities. Fine-tuning shapes the behavior. Retrieval provides current information. Evaluation confirms reliability. Each layer has a distinct role, and understanding how they interact is the foundation of effective LLM engineering.

Practitioners who understand the full mechanism from token prediction to inference-time cost are better positioned to make architecture decisions, set realistic expectations with stakeholders, diagnose unexpected behavior, and choose among the growing range of deployment patterns available for production AI systems. That understanding begins with the core insight: a large language model is a next-token predictor trained at scale, and every capability and every limitation follows from that foundation.