Behavior Trees vs. Finite State Machines

The Limits of a State-Based Mindset

FSMs force us to think in terms of states. An NPC is in the Patrol state or in the Chase state. Every behavior is defined by the current state, and logic is primarily focused on transitioning between these states. As we saw, even with HFSMs, managing all the possible transitions can become a challenge as AI complexity grows.

What if we framed the problem differently? Instead of asking, "What state am I in?", what if the AI constantly asked, "What is the most important thing I should be doing right now?" This question is the philosophical heart of Behavior Trees.

What is a Behavior Tree?

A Behavior Tree (BT) is a hierarchical model, much like an HFSM, but it's used to determine which action to execute from a wide range of possibilities. Instead of states and transitions, it's composed of a tree of nodes that represent tasks, conditions, and logical operations.

To get a solid overview of what a Behavior Tree is and how it's structured, please watch the first few minutes of the following video.

Key Components of a Behavior Tree

As the video explained, a BT is made of a few simple, powerful node types that you can combine to create complex logic. Let's recap them:

• Execution Nodes (or Leaf Nodes): These are the actual actions the AI performs (AttackPlayer, MoveToWaypoint) or conditions it checks (IsPlayerVisible?, IsHealthLow?).
• Composite Nodes: These are parent nodes that control the flow of execution to their children. The two most common types are:
  • Sequence (→): Executes its children in order. If any child fails, the entire sequence fails immediately. Think of it as an "AND" gate—all children must succeed. Example: To open a door, you must MoveToDoor AND TurnHandle AND PushDoor.
  • Selector (?): Executes its children in order until one succeeds. If a child succeeds, the selector succeeds and stops. Think of it as an "OR" gate or a list of backup plans. Example: To deal with a threat, try to Attack, OR if that fails, try to Flee.
• Decorator Nodes: These have only one child and modify its behavior or the result it returns. Common examples include an Inverter (flips Success to Failure and vice versa) or a Limiter (only allows its child to run a certain number of times).

Here's a simple visual example of these nodes in action:

How It Works: The "Tick" and The "Blackboard"

A BT operates on a "tick." Many times per second, a signal (the tick) is sent from the root of the tree. It travels down the branches according to the logic of the composite nodes, eventually reaching an action to execute.

Each node, when ticked, returns one of three statuses:

• Success: The task was completed successfully.
• Failure: The task could not be completed.
• Running: The task is in progress and needs more time (e.g., moving to a location).

This constant, top-down re-evaluation on every tick makes BTs highly reactive. A high-priority branch of the tree can instantly interrupt a lower-priority one if conditions change.

To share information between nodes (like the player's last known position or the NPC's current health), BTs often use a Blackboard, which is essentially a shared data dictionary that any node in the tree can read from and write to.

Behavior Trees vs. Finite State Machines

Now for the main question: why choose one over the other? The problems that FSMs face at scale are exactly what led to the development of BTs in the game industry.

To understand these problems and how BTs solve them, watch the next two video clips.

Key Advantages of Behavior Trees

The videos highlighted several key advantages that make BTs so powerful for complex AI:

1. Modularity and Reusability: A branch of a BT that defines a specific behavior (e.g., "find cover") can be saved as a sub-tree and easily reused in different parts of the main tree or even for different AI characters. In an FSM, state logic is often tightly coupled to its specific transitions, making it harder to reuse.
2. Scalability and Maintainability: Adding a new behavior to a BT is often as simple as adding a new branch. In an FSM, adding a new state might require you to define transitions to and from many other existing states. This visual comparison makes the difference very clear:

3. Reactivity: Because the tree is re-evaluated from the top down on each tick, it can react instantly to changes in the game world. A higher-priority behavior (like fleeing from a grenade) can easily interrupt a lower-priority one (like patrolling) without needing explicit transition logic from every possible state.

Here is a summary of the core differences:

When to Use an FSM vs. a BT

So, should you throw away FSMs and use BTs for everything? Absolutely not. They are different tools for different jobs.

Based on everything we've covered, here is the guiding principle:

• Use a Finite State Machine (or HFSM) when:

  • You have a small, manageable number of behaviors.
  • The logic is genuinely state-based, with clear, rigid transitions.
  • Examples:
    • Game State: Managing high-level game flow like Lobby → Countdown → InRound → PostRound. The transitions are fixed and sequential.
    • UI Flow: Switching between distinct screens like MainMenu → Settings → MainMenu.
    • Simple NPCs: A door that is either Open or Closed. A simple enemy that only cycles between Patrol and Chase.

• Use a Behavior Tree when:

  • You need to model complex AI with many interacting and interruptible behaviors.
  • You need a highly reactive AI that can prioritize actions based on a dynamic environment.
  • You want a modular system where behaviors can be easily reused and modified, sometimes even by designers without coding.
  • Examples:
    • Complex NPC AI: A soldier who needs to patrol, find cover, reload, target specific enemies, throw grenades, and flee, all while reacting to player actions. The "what to do next" is a complex decision, not a simple state change.

Conclusion

In this lesson, we've taken a high-level look at Behavior Trees, a powerful alternative to Finite State Machines for authoring AI. You've learned that BTs structure decision-making as a tree of tasks, using nodes like Sequences and Selectors to create reactive and modular behavior.

Key Takeaways:

• Behavior Trees are hierarchical decision-making structures that excel at managing complex, dynamic, and reusable AI behaviors.
• They function via a "tick" that evaluates the tree from the top down, allowing high-priority actions to interrupt low-priority ones.
• The choice between an FSM and a BT is about using the right tool for the job. FSMs are perfect for clear, rigid state-based logic (like game rounds or UI), while BTs are superior for complex, reactive agents with many possible actions (like modern NPC AI).

Congratulations on completing this course! You began with no prior knowledge of this topic and have progressed from the basic definitions of states and transitions to implementing your own reusable Hierarchical FSM in Luau. You've applied these concepts to practical problems and now have the context to know when an FSM is the right choice, and when you might want to explore other patterns like Behavior Trees. You now possess a solid foundation for building more sophisticated and intelligent systems in your Roblox games. Well done