Real-Time AI Interfaces

Why streaming matters

Streaming is the difference between a UI that feels responsive and one that feels frozen. Without streaming, the user stares at a blank space or spinner for the full duration of model inference — often 5 to 30 seconds for a detailed response — and then the entire response appears at once. With streaming, the first tokens appear within a second of submitting the request, and the response builds in real time. The actual completion time is identical. The perceived experience is entirely different.

Time-to-first-token vs time-to-last-token

These are the two latency metrics that matter for streaming UIs. Time-to-first-token (TTFT) determines how long the user waits before seeing any output. Time-to-last-token (TTLT) determines the total inference time. Streaming does not reduce TTLT — the model still takes the same time to generate all tokens. Streaming reduces the perceived cost of TTLT by starting to show the user output at TTFT.

TTFT for Claude typically ranges from 0.5 to 2 seconds depending on server load and request size. This is well under the 3-second threshold where users start to perceive a system as slow. Non-streaming responses display at TTLT — 5 to 30+ seconds for long responses — which is clearly perceptible as slow. Streaming converts a 15-second TTLT into a 1-second TTFT with progressive display, which feels fast even though the math has not changed.

When not to stream

Streaming adds implementation complexity. For batch processing pipelines where latency does not affect user experience — document processing jobs, offline analysis tasks, scheduled workflows — non-streaming is simpler and equally correct. Streaming is a UI concern. If there is no user watching the response arrive in real time, non-streaming is the right default.

SSE and the Anthropic streaming API

The Anthropic API uses Server-Sent Events (SSE) for streaming. When you enable streaming on a request, the API responds with a series of text events over a persistent HTTP connection. Your client reads and processes these events as they arrive rather than waiting for the full response body.

Event types

message_start — the first event. Contains the message ID and metadata including the model and initial usage stats. Signals that the response has begun.

content_block_start — signals the start of a content block. Each content block has a type: text, tool_use, or thinking. A response can contain multiple content blocks in sequence.

content_block_delta — a token increment within a content block. For text blocks, the delta contains a text fragment to append to the accumulator. For tool use blocks, the delta contains a JSON fragment that builds the tool input incrementally. For thinking blocks, the delta contains a fragment of the model's reasoning.

content_block_stop — signals the end of a content block.

message_delta — contains the final stop reason and usage statistics for the complete message.

message_stop — the last event. Signals that the stream is complete.

Delta accumulation

To reconstruct the full response from a stream, you accumulate deltas. For text content, this is straightforward string concatenation: initialize an empty string, append each delta text fragment as it arrives. For tool use input, the delta contains JSON fragments that build up the tool's input JSON incrementally — you cannot parse the tool input until the content block is complete, because partial JSON is not valid JSON. Accumulate the raw string, then parse it on content_block_stop.

Partial rendering patterns

The challenge of streaming UIs is not reading the stream — the SDK handles that — but deciding how to render partial content to the user as it arrives. Different content types require different partial rendering strategies.

Streaming text

The straightforward case. Append each text delta to the displayed output as it arrives. If the text will be rendered as markdown, decide whether to render incrementally (update the rendered HTML on each delta) or buffer until a structural boundary (a complete paragraph, a complete code block). Incremental markdown rendering can produce visual artifacts as incomplete syntax structures are rendered and then replaced — a single asterisk that flashes briefly before the second asterisk arrives to complete the bold. Buffering to paragraph boundaries reduces visual noise at the cost of slightly worse perceived streaming.

Tool call streaming

When a tool use content block arrives, the model is constructing a tool call. The tool name arrives in the content_block_start event; the input arguments build up over subsequent delta events. You cannot execute the tool until the full input is available at content_block_stop.

For the user, showing a "calling tool: {toolName}..." indicator as soon as thecontent_block_start arrives gives immediate feedback that the agent is doing something. Showing the partial input arguments as they build can provide additional context, though it requires handling the visual noise of incomplete JSON.

Thinking blocks

When extended thinking is enabled, the model produces a thinking block before the response. Thinking blocks can be very long — tens of thousands of tokens — and stream with their own thinking content block type.

For most UIs, showing the thinking block in full is not appropriate — it is the model's internal reasoning process, often verbose and exploratory. The common patterns are: hide it entirely; show a "thinking..." indicator while the thinking block streams, then replace it with the response; or show it in a collapsible "Show reasoning" panel that the user can expand. The collapsible panel is useful for power-user interfaces where reasoning transparency is valuable.

Abort and cancel

Users need to be able to stop a streaming response — they sent the wrong message, the response is going in the wrong direction, or they just changed their mind. Implementing abort correctly is more involved than it might appear.

AbortController

The browser's AbortController is the standard mechanism for cancelling in-flight HTTP requests. Create a controller per streaming request, pass its signal to the fetch call, and call controller.abort() when the user cancels. This closes the connection and triggers an AbortError on the pending read, which you catch and handle.

The Anthropic SDK's stream helper accepts an AbortSignal in the request options. Closing the SDK stream also aborts the underlying request. Use whichever API matches your framework.

Cleanup

When a stream is aborted, clean up the UI state explicitly. Mark the message as cancelled rather than leaving a partial message displayed without any indicator that it was not completed. If the user aborts mid-stream and then sends a new message, the new request should start cleanly — no lingering state from the aborted stream.

In React, use useEffect cleanup to abort streams when the component unmounts or when the streaming state is reset. An aborted stream that is not properly cleaned up will continue delivering events to an unmounted component, producing React state-update warnings and potential memory leaks.

Orphaned requests

An orphaned request is one that is still running on the server after the client has stopped listening. This happens when the user closes the tab, navigates away, or the component unmounts without aborting the underlying request. Orphaned requests waste API quota and compute. Always tie the request lifecycle to the component lifecycle — abort on unmount.

Common mistakes

Streaming implementations tend to fail in the same ways. These are the most common ones.

Buffering defeats streaming

Any buffering layer between the API and the user degrades the streaming experience. Common culprits: an intermediate API server that waits for the full response before forwarding; a state management store that batches updates; a rendering framework that does not commit partial updates to the DOM. If streaming looks slow in production but fast in direct API testing, there is a buffer somewhere in the stack between the API response and the displayed text.

In Next.js, ensure your API route or server action is returning a ReadableStream directly and not buffering the response body. Use the TransformStream API to pipe the Anthropic stream to the HTTP response without accumulation.

Not handling errors mid-stream

A non-streaming API call either succeeds or fails — the error is unambiguous. A streaming call can succeed initially (the first tokens arrive) and then fail mid-stream — a network interruption, a server-side error, a rate limit hit mid-generation. The partial response that arrived before the error is not a valid response. Your code needs to handle the mid-stream error case explicitly: detect the error, stop rendering, show an error state, and offer a retry.

Wrap your stream reading loop in a try-catch. On any error from the stream, set the UI to an error state and discard the partial content. Do not leave a partial response displayed as if it were complete — users will try to use it and encounter confusing or incorrect behavior.

Race conditions on concurrent streams

If the user can send a new message while a stream is in progress — by pressing enter again, clicking a send button that was not disabled, or submitting a form — you can end up with two streams running concurrently, both writing to the same UI state. This produces garbled output and confusing behavior.

Prevent it at the UI level: disable or hide the send button while streaming, and abort the in-progress stream before starting a new one. The user should not be able to create two concurrent streaming states in a single chat thread.