Streaming Responses from Claude API

Hello again! In our last lesson, you successfully gave your bot a memory using grammY's session plugin, enabling it to hold coherent, multi-turn conversations. While the bot is now much more intelligent, you've likely noticed a delay between sending a prompt and receiving the full response. For longer answers, this waiting period can make the bot feel slow and unresponsive.

Today, we will directly address this by implementing one of the most impactful user experience enhancements for AI bots: streaming. You will learn how to modify your bot to receive Claude's response as a stream of text chunks and progressively display it to the user in real-time. This technique makes the bot feel significantly faster and more interactive, mimicking the experience of modern AI chat interfaces.

The Power of Streaming

Currently, our bot works like this:

1. User sends a message.
2. We send the entire conversation history to the Claude API.
3. We wait for the Claude API to generate the entire response.
4. We receive the full block of text and send it as a single Telegram message.

The bottleneck is step 3. Instead of waiting for the whole response, we can ask the API to send us the answer piece by piece as it's being generated. This is called streaming.

The Anthropic TypeScript SDK provides excellent support for this. Let's start by looking at the official documentation to understand the mechanism.

As the documentation shows, getting a stream from the API is straightforward. The real challenge is presenting this stream of text chunks elegantly within a Telegram chat.

The "Placeholder and Edit" Pattern

We can't just send a new Telegram message for every text chunk we receive—that would be chaotic and spam the user with notifications. Instead, we'll use a common and effective pattern:

1. Immediately after the user sends a message, our bot replies with a temporary placeholder message (e.g., "...").
2. As we receive text chunks from the Claude API stream, we accumulate them in a buffer.
3. Periodically, we edit the placeholder message to display the buffered text.
4. Once the stream ends, we perform a final edit to ensure the complete message is displayed.

This creates the illusion of the bot "typing" its response in real-time, directly inside a single message bubble. The key API method we'll need for this is editMessageText, which is available in grammY as ctx.api.editMessageText().

A blog post by René Zander provides a fantastic, concise implementation of this exact pattern. It also introduces a critical real-world consideration: API rate limits.

This pattern of sending a placeholder, buffering text, and using a throttled edit loop is the professional way to handle streaming in a Telegram bot. Let's integrate this into our project.

For a visual demonstration of the SDK's streaming capabilities in action, the following video is a helpful reference.

Updating the Bot for Streaming

Now, let's update our bot.on("message:text") handler in src/bot.ts. We'll replace the single call to claude.messages.create with the new streaming logic, while keeping the session management from our previous lesson.

Here is the updated code for src/bot.ts:

// src/bot.ts

import { Bot, Context, session, SessionFlavor } from "grammy";
import { claude } from "./services/claude";

// --- Type Definitions ---
interface Message {
  role: "user" | "assistant";
  content: string;
}

interface SessionData {
  messages: Message[];
}

type MyContext = Context & SessionFlavor<SessionData>;

// --- Bot Initialization ---
const bot = new Bot<MyContext>(process.env.TELEGRAM_BOT_TOKEN!);

bot.use(session({
  initial: (): SessionData => ({ messages: [] }),
}));

// --- Bot Handlers ---
bot.command("start", (ctx) => {
  ctx.session.messages = []; // Clear history on /start
  return ctx.reply("Hello! I'm a streaming AI assistant. I can remember our conversation. Send /start to begin a new one.");
});

bot.on("message:text", async (ctx) => {
  try {
    const userPrompt = ctx.message.text;
    ctx.session.messages.push({ role: "user", content: userPrompt });

    // 1. Send a placeholder message and start the "typing" action
    const placeholder = await ctx.reply("...");
    await ctx.replyWithChatAction("typing");

    let responseText = "";
    let lastEdit = Date.now();
    const EDIT_INTERVAL = 800; // milliseconds

    // 2. Call the Claude API with streaming enabled
    const stream = claude.messages.stream({
      model: "claude-3-5-sonnet-20240620",
      max_tokens: 1024,
      messages: ctx.session.messages,
    });

    // 3. Process the stream of text chunks
    for await (const chunk of stream) {
      if (chunk.type === "content_block_delta" && chunk.delta.type === "text_delta") {
        responseText += chunk.delta.text;

        // 4. Throttle edits to avoid hitting Telegram's rate limits
        if (Date.now() - lastEdit > EDIT_INTERVAL && responseText.length > 0) {
          await ctx.api.editMessageText(
            ctx.chat.id,
            placeholder.message_id,
            responseText
          );
          lastEdit = Date.now();
        }
      }
    }

    // 5. Perform a final edit to ensure the full message is sent
    await ctx.api.editMessageText(
      ctx.chat.id,
      placeholder.message_id,
      responseText
    );

    // 6. Save the final assistant response to the session history
    ctx.session.messages.push({ role: "assistant", content: responseText });

  } catch (error) {
    console.error("Error in streaming response:", error);
    await ctx.reply("I'm sorry, but I encountered an error while generating a response. Please try again.");
  }
});

// --- Error Handling and Startup ---
bot.catch((err) => {
  // ... (existing error handler)
});

bot.start();
console.log("Streaming bot started...");

Let's break down the changes in the message:text handler:

1. Placeholder: We immediately send ctx.reply("...") to get a message_id we can edit later.
2. Streaming Call: We now use claude.messages.stream() which returns an async iterable. The conversation history from ctx.session.messages is still passed in, maintaining the bot's memory.
3. Stream Processing: The for await...of loop iterates through events from the stream. We check for content_block_delta events, which contain the text chunks (chunk.delta.text).
4. Throttled Edits: Text chunks are appended to the responseText buffer. The if (Date.now() - lastEdit > EDIT_INTERVAL) condition ensures we only call ctx.api.editMessageText at most once every 800ms, respecting Telegram's rate limits.
5. Final Edit: After the loop finishes, there might be remaining text in the buffer that hasn't been sent. A final call to editMessageText flushes the buffer and displays the complete message.
6. Session Update: Only after the full response is constructed do we save it to the session history.

Now, run your bot with bun start. Send it a prompt that requires a longer answer, like "Explain the difference between TCP and UDP". You should see the ... appear almost instantly, then be replaced by the response as it's generated, word by word. The user experience is dramatically improved.

Conclusion

In this lesson, you've transformed your bot from a simple request-response machine into a dynamic, real-time conversationalist. By implementing streaming, you have significantly enhanced the perceived performance and interactivity of your AI assistant.

Key Takeaways:

• Streaming responses from an LLM API is crucial for a good user experience, making the bot feel much faster.
• The Anthropic SDK makes this easy with the client.messages.stream() method.
• In Telegram, the correct way to display a stream is the "placeholder and edit" pattern using ctx.reply() followed by ctx.api.editMessageText().
• It is essential to throttle calls to editMessageText to avoid hitting Telegram's API rate limits.
• The complete assistant message should be saved to the session history after the stream has concluded to maintain conversational context for the next turn.

Your bot is now not only context-aware but also highly responsive. However, our error handling is still quite basic. What happens if the Claude API returns an error mid-stream, or if a user's prompt is blocked by a content filter? In our next lesson, we will dive into robust error handling for the Claude API, learning how to manage rate limits, token limits, and other API-specific issues gracefully.