Introduction to Reinforcement Learning: States, Actions, Rewards, and Policies

Hello! Welcome to your next lesson in the "Artificial Intelligence: Search and Reasoning" module.

In our last session, we formally defined problems involving uncertainty using the language of Markov Decision Processes (MDPs). You learned how to model a scenario by defining its states, actions, transition probabilities, and rewards.

Today, we will solidify your understanding of these core components, as they form the absolute foundation of Reinforcement Learning (RL). This lesson is about building the fundamental vocabulary you'll need for the rest of our journey into RL. The goal is to explain the core reinforcement learning concepts: states, actions, rewards, and policies. Mastering these ideas will bridge the gap between defining a problem (what we did with MDPs) and solving it with an RL algorithm.

1. The Reinforcement Learning Cycle

At the heart of Reinforcement Learning is a constant interaction between two main entities: the Agent and the Environment.

The Agent is the learner or decision-maker. It's the AI you are training, like the game-playing character or the robot arm.
The Environment is the world the agent exists in and interacts with. It's everything outside the agent.

This interaction happens in a loop over a sequence of time steps:

The agent observes the current State of the environment.
Based on this state, the agent selects an Action.
The environment reacts to the action, transitions to a new state, and provides a numerical Reward to the agent.
The agent uses this reward to update its decision-making strategy, which is called a Policy.
The cycle repeats.

Reinforcement Learning Model Diagram — This diagram shows the fundamental cycle of Reinforcement Learning. The agent observes a state (St) and reward (Rt), performs an action (At), and the environment responds with a new state (S(t+1)) and reward (R(t+1)).

To get a more formal definition of these components, let's start with a short reading.

Reinforcement Learning Made Simple (Part 1)

This article, 'Reinforcement Learning Made Simple', provides a clear breakdown of the fundamental components of a Reinforcement Learning problem, framing them within the MDP structure we discussed previously.

Please read the section titled 'To use RL, structure your problem as a Markov Decision Process'. Focus on the definitions of Agent, Environment, State, Action, and Reward. The example of a robot navigating terrain will help you visualize these concepts.

2. A Deeper Dive into the Core Concepts

The reading gave you the formal definitions. Now, let's build a strong intuition for these concepts using a more dynamic example.

RL-1B: State, Action, Reward, Policy, State Transition

The concepts you just read about can be understood more intuitively by seeing them in action. This video from Shusen Wang uses the classic Super Mario game to explain States, Actions, Rewards, and Policies in a very accessible way.

Watch the video from the beginning up to 05:40. Pay close attention to: How a screenshot of the game is considered a 'state'. What the possible 'actions' for Mario are. How 'rewards' are designed to guide Mario's behavior (e.g., +1 for a coin, -10000 for dying). What a 'policy' is and how it decides Mario's next move.

Let's break down and solidify what you saw in the video.

State ( $s$ )

A state is a complete description of the world at a specific point in time. It's a snapshot that contains all the information the agent needs to make a decision. In the Super Mario example, the state is the collection of all pixel values on the screen. For a chess-playing agent, the state would be the position of all pieces on the board.

A key assumption we often make (as discussed in the last lesson) is the Markov Property: the current state captures all relevant information from the past.

Action ( $a$ )

An action is a choice the agent can make. The set of all possible actions is the action space. This can be:

Discrete: A finite set of moves, like {left, right, up} for Mario.
Continuous: A range of values, like the angle to turn a steering wheel (from -45° to +45°).

Reward ( $r$ )

A reward is a numerical feedback signal that the environment sends to the agent. This signal is fundamental because it defines the agent's goal.

Positive rewards are like "pleasure" and encourage the behavior that led to them.
Negative rewards are like "pain" and discourage the behavior.

The design of the reward function is one of the most critical aspects of applying RL. As the video pointed out, if the reward for winning the game isn't large enough, Mario might learn a policy of just collecting coins forever and never actually finishing the level. This is an example of reward hacking, where the agent cleverly maximizes the reward in a way the designer didn't intend.

Policy ( $π$ )

The policy is the agent's brain; it's the strategy the agent uses to select an action given its current state. In RL, our goal is to find an optimal policy ( $π^{*}$ )—a policy that maximizes the total reward the agent can expect to receive over the long run.

As the video explained, policies can be:

Deterministic: The policy always outputs the same action for a given state. E.g., "If you see a Goomba, always jump."
Stochastic (or Probabilistic): The policy outputs a probability distribution over actions. E.g., "If you see a Goomba, there's a 90% chance of jumping and a 10% chance of moving right." Stochastic policies are crucial in games like Rock-Paper-Scissors, where a predictable, deterministic strategy would be easily defeated.

Test your understanding!

Imagine you're designing an RL agent to control a self-driving car at a traffic intersection. Define a possible State, a set of Actions, and a simple Reward function for this agent.

Show answer

There are many correct answers, but here is one possible model:

State (s): A collection of key pieces of information, such as:
- The car's current speed.
- The color of the traffic light (Red, Yellow, Green).
- The distance to the intersection.
- Whether pedestrians are waiting to cross.
Actions (a): A set of discrete moves the agent can take:
- {Accelerate, Maintain Speed, Decelerate, Stop}
Reward (r): A function designed to promote safe and efficient driving:
- +100 for successfully crossing the intersection without incident.
- -1000 for colliding with another car or a pedestrian.
- -1 for every second spent waiting at the intersection (to encourage efficiency).
- -50 for running a red light.

The agent's policy would learn rules like: IF the traffic_light is Red, THEN the best action is Stop to avoid a large negative reward.

3. The Goal: Maximizing Cumulative Reward

An RL agent is "far-sighted." It doesn't just want to maximize its next immediate reward; it wants to maximize the total, cumulative reward over its entire lifetime. We call this total reward the Return.

However, simply adding up rewards can be problematic, especially for tasks that could go on forever. Furthermore, a reward you get right now is often more valuable than a reward you might get far in the future. To handle this, we use a discount factor, $γ$ (gamma), which is a number between 0 and 1.

This next video explains this concept clearly.

An Intro to Reinforcement Learning for Beginners in Python 🐍 (Open AI Gym)

The agent's goal is to get the most reward possible, not just in the next step but over the long run. This video explains how we formalize this idea of 'total reward' and introduces the crucial discount factor.

Please watch from 04:55 to 07:58. Focus on why simply summing rewards can be problematic and how the 'discount factor' (gamma) helps us prioritize immediate rewards while still considering the future.

As the video showed, the discounted return ( $G_{t}$ ) at a time step $t$ is calculated as:

G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots

Where:

$R_{t + 1}$ is the immediate reward.
$γ R_{t + 2}$ is the discounted reward from the step after that.
$γ$ controls how much we value future rewards. If $γ = 0$ , the agent is "myopic" and only cares about the immediate reward. If $γ$ is close to 1, the agent is "far-sighted" and takes future rewards heavily into account.

Conclusion

You have now revisited and solidified your understanding of the essential building blocks of Reinforcement Learning. These are not just abstract terms; they are the practical components you will use to define and solve complex decision-making problems.

Key Takeaways:

RL problems are framed as an Agent interacting with an Environment.
The State is a snapshot of the environment's current situation.
The Action is a choice the agent makes.
The Reward is the numerical feedback that guides the agent's learning.
The Policy is the agent's strategy for choosing actions based on states.
The ultimate goal of the agent is to learn an optimal policy that maximizes the cumulative discounted reward (Return).

Preview of the Next Lesson:

Now that you have mastered the what (the core concepts), we can begin to explore the how. How does an agent actually learn a good policy? There are two main families of algorithms for this. In the next lesson, we will distinguish between value-based and policy-based reinforcement learning methods, setting the stage for understanding specific algorithms like Q-learning and Policy Gradients.

Recommended resources

1. The Reinforcement Learning Cycle

2. A Deeper Dive into the Core Concepts

State (s)

Action (a)

Reward (r)