How Models Think

The context window

Every interaction with a language model happens inside a context window. This is the model's entire world for a given request: every message you've sent, every response it's given, every document you've attached, every tool result it's received. There is no memory outside of it. When the window fills, older content either gets dropped or the session has to end.

Modern frontier models have large context windows. Claude Sonnet 5 supports 1 million tokens. GPT-5 supports 400K tokens. A million tokens is roughly 750,000 words — about three full novels. In practice, this means you can pass entire codebases, long research reports, or extended conversation histories without hitting the limit on most tasks.

What the window actually contains

A context window is a flat sequence of tokens. There's no persistent database, no indexed knowledge base, no file system the model is searching. Everything the model knows about your session is in the tokens that passed through the input — in order, linearly, from the beginning of the conversation to the current message.

This means the context window is both the model's short-term memory (the current conversation) and its reference material (anything you've provided). A RAG system appears to "look things up," but what's actually happening is that retrieved chunks get pasted into the context window alongside the user's question — the model isn't searching a database, it's reading text that the application assembled and placed in the prompt.

The cost dimension

Context is not free. API pricing charges per token processed — both input tokens (everything in the window when you call the API) and output tokens (the response). A 1M-token context call at Claude Sonnet 5 rates costs roughly $3 in input tokens alone. For applications that run many API calls — agents, pipelines, multi-turn conversations — context length is a primary cost driver. Prompt caching (covered in the Building with Claude guide) addresses this by reusing computed context across calls.

What tokens are

Models don't read words. They read tokens — sub-word units produced by a tokenizer that splits text into pieces based on how common certain character sequences are in the training data.

Common English words are usually a single token. "Cat" → cat. Longer or less-common words split: "tokenization" becomes two or three tokens. Code splits differently: const is one token, but a long variable name might be five. Non-English languages tend to token worse — Japanese and Arabic use more tokens per word than English, which means they cost more per word and the model has less context budget per equivalent content.

Why it matters practically

Cost estimation. "100K tokens" is roughly 75,000 English words — a short novel. When you're passing a large document or a long conversation history, use the tokenizer to count before making assumptions. The 1M-token limit sounds generous until you're injecting a 200-file codebase.

Counting numbers. Arithmetic is harder for models than it looks because multi-digit numbers often tokenize as chunks ("42" = one token; "4211" = maybe two). The model is not adding the values — it's predicting the next token in a sequence. This is why models make careless arithmetic errors on tasks that look simple to humans.

Spelling and character manipulation. Ask a model to count the letters in "strawberry" and it will likely get it wrong. The model never sees the individual characters — it sees one or two tokens that represent the whole word. Reverse a word, count vowels, check palindromes: all of these require character-level reasoning that the tokenizer abstracts away.

Token prediction is the model's output

At inference time, the model produces output one token at a time. For each position, it computes a probability distribution over the entire vocabulary — tens of thousands of possible next tokens — and samples from it. Temperature controls the sharpness of this distribution: temperature 0 always picks the highest-probability token; higher temperatures flatten the distribution and increase variety. The model doesn't "decide" to write a paragraph — it predicts token by token, and the paragraph emerges.

What attention does

The core operation in a transformer is attention. Every token in the context window can attend to every other token — it can directly incorporate information from any earlier position in the sequence, not just what's immediately adjacent. This is what enables long-range dependencies: a pronoun in sentence 20 can attend to the noun it refers to in sentence 1 without the model having to "remember" through 19 sentences of intermediate state.

Query, key, value — the plain-language version

For each token, the model computes three vectors: a query (what this token is looking for), a key (what this token offers to other positions searching for it), and a value(what information this token passes along when found). The attention score between two tokens is the dot product of one's query and the other's key. High score → the first token's representation gets strongly influenced by the second.

In practice, this means when the model processes the word "it" in "The cat sat on the mat because it was comfortable," the query for "it" will produce high attention scores against "cat" and "mat" — the model is learning that "it" refers to one of them, and the attention weights encode that relationship. The model doesn't explicitly resolve coreference with a rule; attention weights learned from training encode these patterns implicitly.

Multiple heads, stacked layers

Modern transformers use multi-head attention — running many attention operations in parallel, each with different learned query/key/value projections. Different heads attend to different structural patterns simultaneously: one head tracks syntactic relationships, another semantic similarity, another coreference. The outputs are concatenated and fed into the next layer. Stacking many such layers lets the model build increasingly abstract representations of the input, from token-level patterns in early layers to concept-level patterns in later ones.

What reasoning means

When people say a model "reasons," they typically mean it produces correct answers to problems that require multi-step work — math, logic, coding, planning. The question of whether this is genuine reasoning or sophisticated pattern matching is philosophically contested. The practical question is more useful: what are reasoning models actually doing differently, and when does it help?

Chain-of-thought

Chain-of-thought (CoT) prompting asks the model to show its work — to produce intermediate reasoning steps before the final answer. The performance improvement is real and significant: on math benchmarks, asking the model to reason step-by-step dramatically outperforms asking for the answer directly. The explanation: the model is effectively using the output tokens as scratch space. By writing out intermediate steps, it makes those steps part of the context that subsequent tokens attend to, allowing each step to build correctly on the last rather than trying to collapse a multi-step derivation into a single prediction.

Extended thinking

Claude and other frontier models now support extended thinking — a mode where the model allocates a compute budget for reasoning tokens that are generated before the visible response. These thinking tokens are not shown to the user (in standard mode), but they allow the model to work through a problem at length before committing to an answer.

The key distinction from standard CoT is interleaving: extended thinking can be combined with tool use, so the model can think, call a tool, receive the result, think more, call another tool, and so on. This is what enables the SWE-bench and agentic coding improvements — the model can formulate a plan, execute a step, observe the result, revise, and continue, all within the same reasoning session.

What it still isn't

Extended thinking doesn't grant the model access to information it wasn't trained on. It doesn't give it the ability to verify external facts without tools. It doesn't eliminate hallucination — it reduces it for tasks where more deliberate reasoning catches errors that fast generation misses, but it can't check facts it doesn't have. For tasks requiring current information, retrieval or search tools are the correct solution; more thinking tokens are not.

Where it breaks

Understanding the failure modes is as important as understanding the capabilities. Models fail in consistent, predictable ways — knowing them helps you design prompts and applications that route around them.

Lost in the middle

When critical information is placed in the middle of a very long context, model performance degrades measurably compared to placing it near the beginning or end. This was documented in a 2023 Stanford paper and has been replicated across models. The intuition: attention patterns during training reflect real-world text distributions, where important information tends to appear at the start (system prompts, key facts) or end (the current question). Content buried in the middle of a 200K-token context receives proportionally less attention weight. Practical response: put critical instructions and key facts at the top of system prompts and near the user's most recent message — not embedded in the middle of large documents.

Hallucination

Models generate plausible-sounding text. "Plausible" and "accurate" overlap heavily for common knowledge but diverge significantly for:

Specific facts at the edges of training data — academic citations, obscure API methods, recent events near the knowledge cutoff. The model fills in gaps with statistically likely text, not verified facts.

Precise numerical claims — statistics, dates, measurements. These have high surface confidence (numbers look certain) but the model is extrapolating from patterns, not retrieving from a database.

Complex multi-hop reasoning under time pressure — when the correct answer requires chaining several facts together that may not appear together in training data, the model may confabulate an answer that looks derivationally correct but isn't.

Context limit edge cases

Performance degrades as context fills — not sharply at the limit, but gradually before it. Models trained on shorter sequences may show early degradation when given contexts near their maximum length. In practice: for very long contexts, test with real content at the lengths you'll operate at, not just under ideal conditions. The advertised context window is the ceiling; reliable performance is often somewhat below it.