AI-news notes.

OpenAI releases GPT-4: the first multimodal frontier model, passing the bar exam in the 90th percentile, scoring 5 on AP exams, writing functional code across most languages. No published parameter count. The benchmark jump over GPT-3.5 was large enough that 'AI is actually useful now' became a serious position. Everything changed because the capability threshold had been crossed.

Anthropic releases its first production model. Constitutional AI in practice: a model trained with a set of ethical principles used to self-critique and refine its outputs. Positioned as a safer, more steerable alternative to GPT-4. The beginning of 'which model you use' being a meaningful choice, not just a default.

The meaningful unit of AI integration has shifted from the API call to the context window. What you put in the window — system prompt, tools, examples, state — determines behavior more than model version in most production applications. Fine-tuning is rare. RAG is everywhere. The context engineering skill is now more valuable than model selection.

For most tasks, the bottleneck has inverted: generating tokens is cheap, sampling enough to find a good answer is the expense. o1, R1, and the extended thinking models make this explicit — thinking tokens cost money, but the right answer is worth it. The implication: model providers are now in the business of selling compute-per-correct-answer, not compute-per-token.

The major benchmarks that defined frontier capability for 2022–2024 are saturated or compromised. MMLU is near ceiling. HumanEval was contaminated. SWE-bench Verified is the current gold standard for coding, but it's being gamed. The pattern: a benchmark rises, becomes a training target, loses signal quality within 18 months. Finding ground truth for capability is now a research problem.

The bottleneck in deploying AI agents is not model capability — it's context management, tool reliability, error recovery, and state persistence. The models can reason well enough. What breaks production agents is tool call failures, context window overflow, and unclear task boundaries. The real frontier is agent infrastructure, not MMLU scores.

The gap between open-weight and frontier closed models closed in 2023–2024, then widened again in 2025 as Anthropic and OpenAI shipped reasoning models that relied on proprietary RLHF pipelines. Then narrowed again with DeepSeek-R1 and Llama 4. The pattern is not convergence — it's alternating cycles of parity and divergence, driven by which technique is currently hard to replicate.

Richard Sutton's March 2019 essay distills 70 years of AI research into a single uncomfortable observation: every time researchers have tried to encode human knowledge and domain structure into AI systems, it helped in the short term and hurt in the long run. The approaches that won — always — were those that scaled general-purpose search and learning with more compute. Chess, Go, speech recognition, computer vision: the pattern holds across every domain. The lesson is not that knowledge is useless but that building it into systems as a constraint caps them. Researchers who internalized this in 2019 had a significant head start on understanding why scaling worked so well from 2020 onward.

Ilya Sutskever's 2023 observation — most clearly stated at NVIDIA GTC in March of that year — reframes what language model training is actually doing. Predicting the next token in internet text looks like a statistical compression exercise. But to predict well, the model must implicitly reconstruct the state of the world that produced the text: the author's intent, the physical context, the causal chain behind every word. A model that predicts accurately is, by necessity, modeling the world. This reframing dissolved a persistent objection to LLM capability claims and clarified why scaling toward better prediction keeps producing better reasoning — it is not a coincidence, it is the mechanism.

Sam Altman first used this framing in July 2024 when announcing GPT-4o mini pricing, echoing Lewis Strauss's 1954 promise about nuclear energy — deliberately. The phrase crystallizes a specific economic thesis: that intelligence, like electricity and bandwidth before it, will follow a path from scarce premium resource to infrastructure cost that rounds to zero. Altman elaborated at the Federal Reserve in July 2025: costs had dropped more than 10× per year for five consecutive years. The framing matters because it implies not just cheaper AI but a structural shift in which tasks are worth automating — once intelligence approaches zero marginal cost, the question stops being 'is this task worth doing with AI' and starts being 'why would you do this without AI.'

Vaswani et al. introduce the transformer architecture — self-attention instead of recurrence. Every frontier model running today descends from this paper. The architecture choice that made scaling work: parallelizable, depth-friendly, attention as the core operation. Published at NeurIPS 2017. The thing everyone builds on without always saying so.

Kaplan et al. at OpenAI establish precise power-law relationships between model performance and three variables: the number of parameters, the amount of training data, and the compute budget. The finding that was easy to miss: compute is best spent on larger models trained on fewer tokens than you might expect, not on training small models longer. This paper gave labs a roadmap — not a guess — for how to allocate training runs. It is the theoretical foundation for the billion-dollar bet that bigger always gets better, and it was right enough to drive the entire race to GPT-4 and beyond.

Lewis et al. at Facebook AI Research introduce RAG: rather than storing all knowledge in model weights, retrieve relevant documents at inference time and condition the generation on them. The paper proves the approach on knowledge-intensive NLP tasks where pure parametric models hallucinate or go stale. It became the dominant architecture for grounding LLMs in live, verifiable, or proprietary data — a pattern used by virtually every production AI application built after 2022. RAG is why most enterprise AI deployments do not require fine-tuning.

The GPT-3 paper. Brown et al. show that scaling a language model to 175B parameters produces emergent in-context learning — the model can perform tasks from a few examples in the prompt, without gradient updates. The paper that made 'few-shot prompting' a technique rather than a curiosity, and established that scale itself is a form of capability.

Wei et al. at Google Brain show that prompting a language model with step-by-step reasoning examples — rather than just input-output pairs — dramatically improves performance on math, science, and logic tasks. The key finding: chain-of-thought only emerges reliably in models above roughly 100B parameters, making it the first technique that was demonstrably a property of scale rather than just a prompting trick. It reframed the whole question of what prompting is doing and why it works, and it is the ancestor of every extended thinking and reasoning chain approach in models today.

The InstructGPT paper. Ouyang et al. show that RLHF — fine-tuning on human preference data — produces models that are dramatically more useful and less harmful than raw GPT-3, despite being much smaller. The technique that made ChatGPT possible. The insight: alignment isn't just about capability, it's a training objective.

Bai et al. at Anthropic introduce Constitutional AI: using a fixed set of principles to have the model critique and revise its own outputs, reducing reliance on human labelers for harmlessness training. The paper behind Claude's character — AI feedback at scale as a substitute for human preference data. Also introduces RLAIF, which became an industry-wide technique.

Rafailov et al. show that RLHF can be reformulated as a simpler supervised learning problem — no separate reward model, no RL training loop. DPO matches or exceeds PPO-based RLHF on most tasks at a fraction of the complexity. Became the dominant fine-tuning technique for open-weight models within months of publication. The paper that democratized alignment training.

Dao et al. rewrite attention from scratch with IO awareness — instead of materializing the full attention matrix in HBM, they fuse the computation into a single GPU kernel that fits in SRAM. 2–4× faster than standard attention, with exact (not approximate) output. The engineering paper that made long-context practical. Every frontier inference stack uses it.

Bubeck et al. at Microsoft research GPT-4 for five months before public release and conclude it shows 'sparks of AGI' — performance across math, coding, law, medicine, art, and theory of mind that doesn't fit prior conceptions of narrow AI. The paper that made AGI discourse mainstream inside technical circles. Contested, influential, and impossible to ignore.