Structured Reasoning for Multi-Step Problems

Introduction

In our last lesson, we focused on guiding Large Language Models by providing explicit input-output examples through few-shot prompting. This is highly effective for teaching a model a new format or a classification scheme. However, when a problem requires multiple logical steps—like a math problem, a planning task, or a complex data query—simply showing the final answer isn't enough. The model can often fail by trying to "jump" to a conclusion without performing the necessary intermediate reasoning.

This lesson addresses that gap by introducing chain-of-thought (CoT) prompting and other structured reasoning patterns. Your objective is to learn how to use these techniques to elicit step-by-step thinking from an LLM, dramatically improving its performance on multi-step problems. This is a foundational technique that moves us from simple instruction-following to the more complex reasoning required for the agentic systems that are your ultimate goal.

The Problem with Direct Answers: Pattern Matching vs. Reasoning

Given your background in computer science, you're familiar with the difference between a simple lookup and a multi-step algorithm. LLMs, at their core, are powerful pattern-matchers. For many queries, this is sufficient. But for problems requiring sequential logic, forcing a model to produce an answer in a single step is like asking a program to return the result of a complex calculation without executing the intermediate lines of code. It often fails.

Chain-of-thought prompting solves this by giving the model "space to think." Instead of demanding an immediate answer, we instruct it to articulate its reasoning process first. This aligns the prompt with the model's natural, sequential token-generation process, allowing it to perform a series of logical operations instead of compressing them into a single, often-flawed, forward pass.

Let's begin by watching a short video that explains the core concept of "test-time compute" and how CoT prompting leverages it.

As the video illustrates, by prompting the model to "show its work," we are essentially allowing it to perform more computation before arriving at a final answer, which is key to solving more complex problems.

Implementing Chain-of-Thought: From Zero to Few-Shot

There are two primary ways to elicit a chain of thought, which vary in their level of directiveness.

1. Zero-Shot CoT: This is the simplest method. For capable models, you can often trigger reasoning behavior by simply adding a phrase like "Let's think step by step" to the end of your prompt. The model, having been trained on vast amounts of text, recognizes this as a cue to break down its thinking process.

2. Few-Shot CoT: This is a direct extension of our previous lesson. Instead of providing only input-output pairs, your examples now include the intermediate reasoning steps. This is more explicit and reliable, especially for domain-specific or novel reasoning patterns.

The following article gives a clear overview of both approaches and the fundamental gap they are designed to bridge.

Here's an example of a few-shot CoT prompt for a simple logic problem:

Prompt:

Question: A farmer had 15 sheep. All but 8 died. How many are left?
Answer: The question is a trick question. "All but 8 died" means that 8 sheep are the ones that survived. So, there are 8 sheep left.




###

Question: If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
Answer:

This prompt structure not only shows the correct answer but demonstrates the reasoning required to avoid a common pitfall. The model will now follow this pattern for the new question.

Advanced Reasoning Patterns: Building More Robust Systems

A single chain of thought can still be wrong. More advanced techniques build on CoT to create more robust and powerful reasoning structures. These methods often involve generating multiple possibilities and evaluating them, a process analogous to exploring different paths in a search algorithm.

Let's explore some of these advanced patterns.

• Self-Consistency: Instead of generating one chain of thought, you ask the model to generate several (by making multiple API calls with a non-zero temperature). You then select the final answer that appears most frequently across the different reasoning paths. This is an ensemble method that improves reliability by mitigating the impact of a single flawed reasoning chain.
• Tree of Thoughts (ToT): This pattern generalizes CoT by allowing the model to explore multiple reasoning paths at each step. The model generates several possible "next thoughts," evaluates their viability, and proceeds down the most promising branches. This is computationally expensive but very powerful for problems that require exploration or strategic planning, like solving a puzzle or generating a complex design.
• Least-to-Most Prompting: This involves decomposing a problem into a sequence of simpler sub-problems and solving them in order. The solution to each sub-problem is fed into the prompt for the next one. This is a natural fit for your software engineering background, as it mirrors the process of breaking down a complex feature into smaller, manageable functions or modules.

The same article from Comet provides an excellent summary of these techniques.

From Chain-of-Thought to Agentic Behavior

The true power of these reasoning patterns becomes apparent when they are used to build AI agents. An agent doesn't just think; it acts. The ReAct (Reason + Act) framework is a seminal pattern that directly combines CoT with action.

In a ReAct loop, the model follows a "Thought -> Act -> Observation" cycle:

1. Thought: The LLM generates a chain of thought, reasoning about what it needs to do next to solve the user's request.
2. Act: Based on its thought, the model decides to take an action, such as calling a tool (e.g., a search engine API, a calculator, or your own codebase).
3. Observation: The result of the action (e.g., the search results or the calculator's output) is fed back into the model's context.
   The loop then repeats, with the model reasoning about the observation and planning its next thought and action.

This is the fundamental building block of agentic AI, and it's a direct application of CoT.

The following reading explains how CoT is the cornerstone of these systems.

For a deeper dive into how ReAct is implemented, this resource provides a concise explanation and a conceptual code workflow.

Conclusion

Today we've moved beyond simple instruction-following to explore how to make LLMs "think." By encouraging or explicitly demonstrating step-by-step reasoning, we can unlock their ability to solve complex, multi-step problems that are otherwise intractable.

Here are the key takeaways:

• Chain-of-Thought (CoT) prompting bridges the gap between pattern matching and true reasoning by forcing the model to articulate its intermediate steps.
• It can be implemented via zero-shot (e.g., "Let's think step by step") or few-shot (providing examples with reasoning) methods.
• Advanced patterns like Self-Consistency and Tree of Thoughts (ToT) build on CoT to improve robustness and handle problems requiring exploration, though they come with higher computational costs.
• CoT is not just a prompting trick; it is the foundational mechanism for agentic frameworks like ReAct, which enables an LLM to reason, act, and observe, forming the basis of autonomous agents.

You now have a solid understanding of the primary reasoning structures used in advanced prompting. In our next lesson, we will shift our focus to another critical component of prompt engineering: crafting effective system prompts to define a model's long-term persona, constraints, and overall behavior.