Real-time Response Streaming: WebSockets vs. SSE vs. Chunked Transfer

Hello! Welcome back to your course on preparing for an Anthropic software engineering interview.

In our last lesson, we tackled the critical challenge of memory management for the KV cache. We designed a strategy using PagedAttention, borrowing concepts from operating systems to nearly eliminate memory fragmentation and enable highly efficient, concurrent request processing. With a dynamic batching scheduler and an efficient memory manager, our backend can now generate tokens at high throughput.

This brings us to the final piece of the serving puzzle: how do we deliver these tokens to the user? Waiting for the entire response to be generated would lead to a slow user experience. The goal is to make the interaction feel as real-time as possible. This lesson is dedicated to designing the system that achieves this.

Our learning outcome is to design a system for streaming token-by-token responses, comparing WebSocket, Server-Sent Events (SSE), and chunked transfer. We'll analyze the trade-offs of each technology and determine the industry-standard approach for LLM services.

1. The Need for Streaming

In a traditional request-response model, a client sends a request and then waits for the server to compute and return the entire response. For an LLM generating a long-form answer, this could mean many seconds of dead air, creating high perceived latency.

A naive improvement might be short polling, where the client repeatedly asks, "Are you done yet?". This is highly inefficient.

Let's watch a brief video that illustrates the limitations of polling and sets the stage for more efficient, event-driven communication patterns.

To provide a responsive, "typewriter" effect like you see in ChatGPT or Claude, the server must be able to push data to the client as soon as it's generated. Let's explore the primary technologies that enable this.

2. Core Streaming Technologies: A Comparison

We'll evaluate three main technologies for building our streaming system. For each, we'll consider how it works, its pros and cons, and its suitability for our specific use case of delivering LLM tokens.

A. HTTP Streaming with Chunked Transfer Encoding

This is the most fundamental form of HTTP streaming. In HTTP/1.1, if a server doesn't know the full size of a response ahead of time, it can omit the Content-Length header and instead specify Transfer-Encoding: chunked. This allows the server to send the response in a series of "chunks," each prefixed with its size in hexadecimal. A final, zero-length chunk signals the end of the response. HTTP/2 and HTTP/3 have more advanced native streaming mechanisms (DATA frames), but the principle is the same.

How it works for LLMs:
The server simply writes each new token (or a small batch of tokens) directly to the response stream. The client receives these raw chunks and must have logic to decode and process them.

• Pros:
  • Low-level and Efficient: It's built directly into HTTP, involving minimal overhead.
  • Flexible: The client and server can define any data format they want for the stream (e.g., plain text, NDJSON).
• Cons:
  • Lacks Standardization: There's no built-in protocol for message framing, event typing, or error handling. You have to invent your own (e.g., using newlines as delimiters).
  • Manual Client Implementation: The client must manually handle the raw byte stream, decode it (e.g., using TextDecoder), and implement any reconnection logic if the connection drops.

This approach is excellent for server-to-server communication or custom clients where you have full control over both ends and prioritize raw performance.

B. Server-Sent Events (SSE)

SSE is a standardized protocol built on top of HTTP streaming. It's designed specifically for servers to push updates to clients. The client initiates a connection, and the server keeps it open, sending messages in a specific text/event-stream format.

Key Features:

• Unidirectional: Communication is one-way, from server to client.
• Standardized Event Format: Events are simple text blocks, typically starting with data:. This provides a clear structure that chunked transfer lacks.
• Native Browser API (EventSource): This is the killer feature. Modern browsers have a built-in EventSource API that handles all the complexity of connection management, message parsing, and, crucially, automatic reconnection. If the connection is lost, the browser will automatically try to reconnect, sending the ID of the last event it received so the server can resume the stream.

Let's watch a video that explains SSE and explicitly connects it to its use in ChatGPT.

Given these features, it's no surprise that SSE is the de-facto standard for streaming LLM responses to web-based clients.

• Pros:
  • Perfect Fit for LLMs: Unidirectional push is exactly what's needed for streaming tokens.
  • Simple Client-Side: The EventSource API makes consumption trivial in a web browser.
  • Robust: Automatic reconnection significantly improves reliability.
• Cons:
  • Unidirectional: Not suitable if the client also needs to send real-time messages to the server (beyond the initial request).
  • Text-Only: SSE is designed for UTF-8 text data.
  • Connection Limits: In HTTP/1.1, browsers limit the number of concurrent SSE connections per domain (usually 6), though this is largely mitigated by HTTP/2's multiplexing.

C. WebSockets

WebSockets provide a fully bi-directional, full-duplex communication channel over a single, long-lived TCP connection. Unlike SSE, which is just a specially formatted HTTP response, WebSockets are a separate protocol (ws:// or wss://) initiated by an HTTP Upgrade request.

How it works for LLMs:
The client would establish a WebSocket connection, send the prompt as a message, and the server would send each token back as a separate message over the same connection.

• Pros:
  • Bi-directional & Low Latency: Ideal for truly interactive applications like online games, collaborative editors, or financial trading platforms where both client and server need to push data at any time.
• Cons:
  • Overkill for Token Streaming: The bi-directional capability is not needed for a standard prompt-response interaction, adding unnecessary complexity to the server (managing stateful connections) and the client.
  • Infrastructure Complexity: WebSockets can sometimes be more difficult to configure with firewalls and proxies than standard HTTP traffic.
  • No Automatic Reconnection: Like raw chunked streaming, you have to implement reconnection logic yourself.

3. Trade-offs and The Final Design

Now, let's formalize the comparison to make a clear design choice. The following article provides a Node.js implementation of both SSE and chunked streaming endpoints for an LLM, along with a direct comparison table.

Here is a summary of the trade-offs:

Feature	HTTP Chunked Streaming	Server-Sent Events (SSE)	WebSockets
Directionality	Server → Client	Server → Client	Bi-directional
Protocol	Standard HTTP	Standard HTTP	Upgraded from HTTP to WS/WSS
Client API	Manual (fetch + ReadableStream)	Native (EventSource)	Library or manual implementation
Reconnection	Manual	Automatic (in browsers)	Manual
Complexity	Low on server, medium on client	Low on both (for web clients)	High on both
Best For	Server-to-server; custom clients	Web UI token streaming (Industry Standard)	Highly interactive apps (e.g., chat, games)

Conclusion: For streaming LLM responses to a standard web client, Server-Sent Events (SSE) is the clear winner. It provides the ideal feature set (unidirectional push, robust reconnection) with the lowest implementation complexity, which is why it's the technology of choice for Anthropic, OpenAI, and others.

4. System Architecture

Let's visualize how this fits into our overall LLM serving system. Your experience with Rust and Tokio makes this architecture particularly relevant.

The end-to-end flow looks like this:

1. Client Request: A web client sends a request to the API, e.g., POST /v1/chat/completions, with a body including {"stream": true}. The request headers include Accept: text/event-stream.
2. API Gateway: The request hits a load balancer and is routed to a front-end API service. This service handles authentication, rate limiting, and request validation.
3. Dispatch to Inference: The API service forwards the prompt to the core inference engine (which uses the dynamic batching and PagedAttention memory management we designed).
4. Token Generation: The inference engine generates tokens one by one for this request.
5. Streaming Back: As each token is generated, it's sent back to the API service that is holding the original client connection open.
6. SSE Formatting: The API service formats the token into a valid SSE message. For example: data: {"id": "comp-123", "token": " Hello"}\n\n.
7. Push to Client: The formatted message is written to the HTTP response stream. The client's EventSource receives the event and updates the UI.
8. End of Stream: Once the model finishes generating, the inference engine signals completion. The API service sends a final event, often data: [DONE]\n\n, and closes the connection.

This architecture decouples the complex inference logic from the client-facing communication protocol, allowing each component to be optimized independently.

Conclusion

In this lesson, we designed a robust system for delivering LLM-generated tokens to a client in real-time. We analyzed the primary technologies for server-push communication and justified our design choices based on a clear understanding of their trade-offs.

Key Takeaways:

• Streaming is essential for a responsive user experience in LLM applications, avoiding the high perceived latency of a traditional request-response model.
• We compared three core technologies:
  • HTTP Chunked Streaming: A low-level, efficient mechanism, best for server-to-server communication.
  • WebSockets: A powerful bi-directional protocol that is overkill and overly complex for simple token streaming.
  • Server-Sent Events (SSE): The ideal solution for this use case. It's a standard HTTP-based protocol with a simple, unidirectional model and crucial features like automatic reconnection, making it the industry standard for LLM streaming to web clients.
• The final system architecture involves a client-facing service that manages the SSE connection while the backend inference engine focuses on high-throughput token generation.

Next Lesson Preview

Our serving system is now functionally complete. It can batch requests, manage memory efficiently, and stream responses back to the user. But how do we ensure we have the right amount of expensive GPU resources to meet fluctuating demand without breaking the bank? In the next lesson, we will address this by learning how to design an autoscaling policy for the inference fleet based on token-aware queue depth, not just GPU utilization.