Streaming LLM Responses with Server-Sent Events

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back! In our last lesson, we implemented semantic caching to reduce latency and cost by avoiding redundant LLM calls. This is a powerful server-side optimization. Today, we'll tackle performance from a different angle: user-perceived latency. Even when an LLM call is necessary, we don't want the user staring at a blank screen.

This lesson focuses on Server-Sent Events (SSE), the industry-standard technology used to stream LLM responses token-by-token. You've seen this in action in applications like ChatGPT. By the end of this session, you will be able to configure a backend service to stream LLM responses and understand how a frontend client consumes that stream. Mastering this is essential for building the responsive, modern AI applications that users expect.

The Problem: The Waiting Game

When a request to an LLM can take several seconds to complete, the user experience suffers. Users are left waiting, unsure if the application is working or frozen. Streaming solves this by sending the response in small chunks as it's generated.

Let's watch a quick demonstration that vividly illustrates the difference between a standard blocking response and a streaming response.

Streaming LLM Responses with FastAPI

This video by Code with Irtiza provides a powerful visual comparison of the user experience with and without streaming LLM responses in a FastAPI application.

Watch the first part of the video, from the beginning to the comparison. Notice the significant delay in the non-streaming version versus the immediate feedback provided by the streaming approach. This is the core problem we are solving today.

The difference is stark. The perceived responsiveness of the streaming version is vastly superior, even if the total generation time is the same. As you can see in the system architecture for a large-scale application like ChatGPT, SSE is a fundamental part of the design.

This diagram shows a high-level architecture for a major AI application. Note the "SSE events" arrow, indicating that streaming is the method used to deliver responses from the system back to the 200 million daily active users.

Understanding Server-Sent Events (SSE)

So, what exactly is SSE? It's a web technology that allows a server to push data to a client over a single, long-lived HTTP connection. To get a solid grasp of the fundamentals, the following video is an excellent primer.

Server-Sent Events Crash Course

This "Crash Course" by Hussein Nasser explains SSE from first principles, covering the protocol, the necessary HTTP headers, and the native browser API for consuming events.

First, watch the segment explaining what SSE is. Pay attention to the headers mentioned: Content-Type: text/event-stream and Transfer-Encoding: chunked. These are the keys to enabling SSE on the server. Next, watch the coding demo. This will show you a bare-bones implementation in Node.js and how the browser's built-in EventSource API is used to listen for messages. Note the simple data: ...\n\n format of each message.

The key takeaway is that for a use case like LLM streaming, where the data flow is one-way (server to client), SSE is simpler and more efficient than a full bidirectional protocol like WebSockets. This is a crucial trade-off to articulate in a system design interview.

Backend Implementation: Streaming from the Server

Now let's dive into the implementation. Since you are proficient in Go, we'll start with a resource that provides a production-grade example using Go's standard library. This will feel very natural given your background.

#6 - Server-Sent Events in Go: Building Real-Time AI Streaming Without WebSockets - DEV Community

This article provides a superb, detailed walkthrough of implementing an LLM streaming service in Go. It covers everything from making the streaming request to the LLM provider to forwarding the events to the client.

Start by reading the section Why SSE over WebSockets. This reinforces the architectural choice we just discussed. Next, focus on the implementation of the LLM client in the section The LiteLLM Client. Pay close attention to the readSSE function. It demonstrates how to: Use bufio.NewReader to read the response body line-by-line. Parse the data: <json> lines from the LLM provider. Handle the special data: [DONE] message that signals the end of the stream. Use a goroutine and a channel (<-chan port.StreamEvent) to process events concurrently. Now, look at the server side in the sections The SSE Response Helpers and The Chat Handler. This is where your backend exposes the stream to the client. Note the critical steps: Setting the SSE headers (Content-Type: text/event-stream, Connection: keep-alive, etc.). Using the http.Flusher interface to force buffered data to be sent to the client immediately after each write. Ranging over the event channel from the LLM client and writing each event to the http.ResponseWriter.

This Go example showcases a robust, low-level implementation. For contrast, it's useful to see how a high-level framework in Python, the most common language in AI, handles this. The FastAPI video you watched earlier explains this pattern well.

Streaming LLM Responses with FastAPI

This segment explains how FastAPI and libraries like LangChain abstract the streaming process.

Rewatch the core explanation of the streaming implementation, from this timestamp. Notice the key differences from the Go approach: Instead of manually reading from a buffer, the code uses llm.stream(), which returns a generator. Instead of manually flushing, the entire generator is passed to FastAPI's StreamingResponse object, which handles the headers and chunked sending automatically.

Both the Go and Python examples achieve the same result, but through different levels of abstraction. Understanding both gives you a more complete picture.

Production Hardening and Frontend Consumption

A working implementation is just the first step. A production-ready system must handle edge cases gracefully. Your experience as a software engineer makes you well-equipped to appreciate these details.

Handling Client Disconnects

What happens if a user closes their browser tab mid-stream? You must stop the entire pipeline to avoid wasting resources and generating tokens that will never be seen.

#6 - Server-Sent Events in Go: Building Real-Time AI Streaming Without WebSockets - DEV Community

This section of the Go article explains how Go's context.Context is used to propagate cancellation signals through the entire application stack.

Read the section Context Cancellation. This is a perfect example of robust, resource-safe engineering. When the net/http server detects the disconnect, it cancels the request's context, which triggers a clean shutdown of the goroutine and the upstream LLM API call.

Proxy Buffering

A common and frustrating issue is when a reverse proxy (like Nginx) or a CDN buffers your response, waiting for it to complete before sending it to the client. This completely defeats the purpose of streaming.

The Complete Guide to Streaming LLM Responses in Web ...

This article discusses several critical details for a production SSE setup, including the dreaded proxy buffering problem.

Read the section Critical Implementation Details. Pay special attention to the part about Proxy and Load Balancer Buffering. The X-Accel-Buffering: no header is often the magic fix required to make streaming work in a production environment. The discussion on heartbeats and backpressure are also excellent points for building highly reliable systems.

Frontend Consumption

Finally, to complete the end-to-end picture, let's briefly look at how a modern web frontend consumes the SSE stream. Since you're proficient in JavaScript, this React example will be easy to follow.

The Complete Guide to Streaming LLM Responses in Web ...

This final section provides a complete frontend implementation, including a custom React hook for managing the streaming state.

Skim through Part 2: Frontend Implementation with React. Focus your attention on the useStreamingChat.ts custom hook. You'll see how it uses the fetch API to make the request, gets a ReadableStream from the response body, and reads and decodes the chunks in a while loop. It parses the data: lines and updates the component's state, causing the UI to re-render with the new content.

Conclusion

In this lesson, you've moved from understanding why streaming is critical for user experience to implementing it on the backend and seeing how it's consumed on the frontend. This is a fundamental skill for any AI engineer building interactive applications.

Key Takeaways:

Server-Sent Events (SSE) is the ideal protocol for the unidirectional flow of LLM responses, offering a simpler alternative to WebSockets.
The backend implementation requires setting specific HTTP headers (Content-Type: text/event-stream) and, crucially, flushing the response buffer after writing each chunk to ensure immediate delivery.
Frameworks like FastAPI can abstract away some of this complexity with helpers like StreamingResponse, while a lower-level language like Go gives you more direct control with tools like http.Flusher.
Production-grade streaming must account for client disconnects (using mechanisms like context.Context in Go) and proxy buffering (using headers like X-Accel-Buffering).

We have now optimized our AI system for both server efficiency with semantic caching and user-perceived performance with streaming. However, some agentic tasks are so complex that they can take minutes to run, which is too long even for a streaming connection. In our next lesson, we will tackle this by designing an asynchronous architecture using a message queue, moving beyond the request-response model to handle long-running background tasks.