Agents in the Wild

What makes an agent

The word "agent" is overloaded. In the broadest sense, any call to a language model is a kind of agent — it receives input and produces output. In the sense that matters for building reliable AI systems, an agent is something more specific: an agent is a system that takes sequences of actions toward a goal, with observable consequences, without requiring human approval at each step.

A single-turn API call that generates a response is not an agent. A model that reads a task, writes a file, runs a test, reads the test output, fixes the error, and runs the test again is an agent. The distinction is the feedback loop: agents observe the results of their actions and update their behavior accordingly. Without that loop, you have a sophisticated autocomplete. With it, you have something that can make progress on real-world tasks that a single text generation can't complete.

The minimal agent

The smallest viable agentic loop has three components: a model that can call tools, a set of tools that interact with the external world (file system, shell, APIs), and an orchestrator that routes tool call results back to the model as context for the next step. Claude Code is this loop: the model reads tasks, calls read/edit/bash tools, observes the results, decides next steps, and continues until it judges the task complete. The orchestration layer can be as simple as a while loop that keeps calling the API until the model returns a final response instead of another tool call.

Agentic vs single-turn tasks

Not every task benefits from an agentic approach. Single-turn generation is faster, cheaper, and easier to debug. Agents are appropriate when:

The task requires actions in the world (write files, call APIs, run code) rather than pure text generation.

The task requires conditional logic: the right action at step 3 depends on the result at step 2, which wasn't known at the start.

The task is too large or multi-part to complete in a single generation window.

The agentic loop

Every agentic system follows the same underlying loop regardless of what framework or orchestration layer wraps it: plan → act → observe → plan. The model decides what to do next, takes an action via a tool call, receives the result as context, and decides again. This continues until the model generates a final response rather than another tool call — or until an external stop condition is hit (max iterations, timeout, human interrupt).

Context accumulates

Each step in the agentic loop adds to the context window: the tool call, the tool result, the model's next reasoning step. A 20-step coding task might consume 30,000–100,000 tokens of context by the time it completes. This is where context compaction matters practically: Claude Code's compaction step summarizes earlier parts of the session when the window fills, preserving the task state while dropping verbatim content that's no longer needed. For long-running sessions, the compaction strategy determines how much earlier context the model retains effectively.

Multi-agent vs single-agent

A single agent handles tasks sequentially. Multi-agent architectures fan out parallel execution: an orchestrator model breaks a large task into independent sub-tasks, spawns sub-agents to execute each in parallel, and synthesizes their outputs.

Claude Opus 4.8's Dynamic Workflows and Claude Code's parallel subagent support both follow this pattern. The practical gain: a 10-file refactor where each file can be analyzed independently takes 10× less wall-clock time in parallel than in sequence. The practical cost: coordination complexity, context management across agents, and debugging failures in a system where multiple agents may have partially completed overlapping work.

Designing for agents

Building tasks that agents execute well is its own design discipline. A task that a human would complete easily — "refactor the authentication module" — may be ambiguous in ways that cause an agent to head confidently in the wrong direction, make irreversible changes, and surface the problem 40 steps later.

Scope tasks clearly

The clearest signal of a well-scoped agentic task: the model can unambiguously declare success or failure. "Refactor the authentication module" has no clear completion condition. "Refactor the authentication module so that all tests in auth.test.ts pass and the JWT handling is extracted to lib/jwt.ts" has one.

Verifiable completion conditions — tests, linting, specific file outputs, API responses — dramatically improve agent reliability because the model can check its own work rather than guessing whether it's done.

Reversibility and checkpoints

Design for reversibility wherever possible. Git-staged commits rather than direct writes let a human review before the change becomes permanent. Dry-run modes on destructive tools (delete, overwrite, send email) prevent a classification error from becoming an incident.

Set checkpoints for long tasks: "after completing steps 1-5, pause and show me the plan for steps 6-10 before proceeding." This converts a single long autonomous run into a supervised sequence, catching errors before they compound across 40 steps.

Tool design

Tools should be atomic: each tool does exactly one thing, returns clear structured output, and doesn't silently swallow errors. A tool that returns partial results with no error signal when it fails will mislead the agent into thinking the action succeeded.

Avoid tools with side effects that are hard to observe. If a tool modifies external state, it should also return what it changed so the model can confirm the action was correct.

What goes wrong

Agentic failures are qualitatively different from single-turn failures. A bad single-turn response is easy to spot and discard. A bad step at iteration 5 of a 40-step task has compounded by iteration 40 into something that looks internally consistent but is wrong in ways that are hard to trace back.

Compounding errors

The most common agentic failure mode: a small error at step N gets treated as ground truth by steps N+1 through M. The model edits file A based on a misunderstanding, then edits files B through E to be consistent with the now-wrong state of A, then writes tests that pass against the wrong state. The result looks coherent — it's internally consistent — but is wrong throughout. This is much worse than a random error in a single-turn response, which would be obvious. Mitigation: build in verification steps after consequential actions, not just at the end.

Hallucinated tool results

If a tool call fails silently or the orchestrator doesn't properly route the error back to the model, the model may proceed as if the tool succeeded — filling in the expected result from its training data. It "remembers" that a successful file read typically returns code that looks like X, so it behaves as if that's what it got. The subsequent steps are all built on fabricated context. Mitigation: never let tool errors silently vanish. Return error messages to the model explicitly; let it decide how to handle the failure.

Stuck loops

An agent that encounters an error may retry the same tool call repeatedly, each time getting the same error, never escaping. This is especially common with rate limits and transient network errors: the model retries, fails, retries, fails, until it exhausts the context budget or the max_iterations limit. The loop termination condition — max iterations, timeout, explicit "give up" instruction — is not optional in production agents. It's the only thing preventing an infinite retry loop from running until you notice.

Specification drift

Over a long session, the model may gradually reinterpret the original task in ways that drift from the user's intent. A "refactor for readability" task may silently expand to include behavioral changes the user didn't ask for, because the model's interpretation of "readability" drifted across 30 steps. Mitigation: include the original task description prominently in the system prompt and instruct the model to re-read it before each major decision.

Measuring what works

The hardest part of building agents isn't the first working prototype — it's knowing whether the changes you're making are actually improving it. Evals are the answer: structured tests that measure agent performance on representative tasks with measurable outcomes. Without them, you're optimizing by feel, and regressions are invisible until they hit users.

What SWE-bench actually measures

SWE-bench Verified is the canonical benchmark for coding agents: given a real GitHub issue on a real open-source project, pass the test suite that defines the fix. The benchmark scores are pass rates on 500 curated issues across 12 Python repositories. A 79.6% score (Claude Sonnet 4.6) means the model resolves 398 of 500 real issues on the first attempt without human guidance. The benchmark is designed so that passing it requires actually understanding the codebase, not just generating plausible-looking patches. It's imperfect — it doesn't measure latency, cost, or multi-turn interactions — but it's the closest thing to a standardized measure of agentic coding capability that currently exists.

Writing your own evals

Benchmark scores measure capability on someone else's tasks. Your users are a different distribution. Write evals against the tasks your agent actually performs:

Representative tasks. Collect 50–100 real task examples from production use (or generate realistic synthetic ones). Each needs an input and a ground-truth success condition.

Measurable outcomes. "Good response" is not an eval condition. "Test suite passes," "output matches schema," "specific value appears in output" are. For tasks where human judgment is unavoidable, use a separate LLM-as-judge call with a rubric rather than manual review — it scales.

Pass@1 vs pass@k. Pass@1 is the fraction of tasks the agent solves on the first attempt. Pass@k measures whether the agent can solve a task if given k tries. For most production use, pass@1 is what matters — users aren't running the same task 10 times. But pass@k is useful for understanding whether a task is solvable at all.

Run on every prompt change. If you change the system prompt, run the eval suite before deploying. Prompt changes that improve one task type regularly regress others. The eval suite is the only thing that catches this systematically.

The minimum viable eval

A working eval harness doesn't require a framework. Three components: (1) a list of tasks as JSON objects (input + expected output / success condition), (2) a runner that calls the agent on each task and captures the result, (3) a scorer that checks whether the result meets the success condition and reports the pass rate. This fits in a single 50-line script. The value is not in the infrastructure — it's in having a stable number you can track over time that tells you whether the agent is getting better or worse.