Understanding Value Functions: V(s) and Q(s,a)

Introduction

Welcome to your third lesson on the foundations of Reinforcement Learning. In our previous lessons, we established two cornerstones:

The Markov Decision Process (MDP), which defines the environment our agent operates in—its states, actions, transition rules, and rewards.
The Stochastic Policy ( $π (a ∣ s)$ ), which defines the agent's strategy or "brain," telling it the probability of taking any action $a$ in a given state $s$ .

Now, we face a critical question: how do we know if a policy is good or bad? Simply acting in the world isn't enough; we need a way to quantify the long-term benefit of following a particular strategy. This is where value functions come into play. They are the yardstick we use to measure the desirability of states and actions.

Today's learning goal is to define the state-value function V(s) and the action-value function Q(s,a). These two functions are perhaps the most important concepts in value-based reinforcement learning and are essential for understanding advanced algorithms like PPO and GRPO later in the course.

1. The Value of a Strategy: Expected Return

Remember the concept of the return, $G_{t}$ , which is the discounted sum of all future rewards from a time step $t$ :

G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots = k = 0 \sum \infty γ^{k} R_{t + k + 1}

The discount factor, $γ$ , ensures that rewards received sooner are valued more highly.

Total Discounted Rewards (Return) in Reinforcement Learning — This diagram illustrates how the total return, $G_t$, is calculated by summing up rewards from future time steps, with each reward being discounted by a factor of $\gamma$.

Because both the agent's policy and the environment's transitions can be stochastic (probabilistic), the return $G_{t}$ you get from a single episode is a random variable. If you run the same policy from the same starting state twice, you might get two different trajectories and two different total returns.

To get a reliable measure of how "good" a situation is, we can't just rely on a single outcome. Instead, we need to calculate the expected return. This is the core idea behind value functions.

To begin, please read this short introduction that frames the purpose of value functions.

Action/State Value Functions, Bellman Equations, Optimal Action/State Value Functions

The article 'Action/State Value Functions...' by Gurpreet's AI Blog clearly introduces the purpose of value functions in RL.

Read the sections 'Article Goal' and '1. Rewards'. Focus on the distinction between the cumulative reward and the discounted future reward, and understand why the discount factor \gamma is used.

2. The State-Value Function: V(s)

The first type of value function we'll define is the state-value function, denoted $V^{π} (s)$ . It answers the question: "If I am in state $s$ and follow policy $π$ from now on, what is the expected total future reward I will get?"

Mathematically, it's defined as the expected return given the current state:

V^{π} (s) = E_{π} [G_{t} ∣ S_{t} = s]

The subscript $π$ on the expectation $E$ is crucial: it signifies that this expectation is taken over trajectories generated by the agent following policy $π$ . The value of a state is therefore policy-dependent. A state that is very valuable under a good policy (e.g., a state one step away from the goal) might be worthless under a bad policy that moves away from the goal.

The following video provides an intuitive walkthrough of what state-value functions represent.

Bellman Equation - Explained!

This video from CodeEmporium uses a simple grid-world to explain the concept of value functions.

Watch from the beginning to 04:13. This will introduce the two types of value functions and then use a grid-world example to provide a concrete intuition for the state-value function V(s).

As the video illustrates, $V^{π} (s)$ quantifies how "good" it is for the agent to be in a particular state while committed to a specific strategy $π$ .

3. The Action-Value Function: Q(s,a)

While knowing the value of a state is useful, it doesn't directly tell the agent which action is best to take. For that, we need a more granular function: the action-value function, also known as the Q-function, denoted $Q^{π} (s, a)$ .

It answers the question: "If I am in state $s$ , take a specific action $a$ , and then follow policy $π$ from that point onwards, what is my expected total future reward?"

The mathematical definition is very similar to the state-value function, but it's conditioned on both the state and the initial action:

Q^{π} (s, a) = E_{π} [G_{t} ∣ S_{t} = s, A_{t} = a]

Let's return to the video to see how this works with the same grid-world example.

Bellman Equation - Explained!

Let's continue with the CodeEmporium video to understand the Q-function.

Watch from 05:57 to 07:59. This segment explains the state-action value function Q(s,a) and relates it back to the grid-world example.

The Q-function is immensely powerful because it allows an agent to compare actions directly. If $Q^{π} (s, action_{1}) > Q^{π} (s, action_{2})$ , it means that, under the current policy $π$ , taking $action_{1}$ is expected to yield a higher long-term return than taking $action_{2}$ .

This image provides a clear visual summary of the Q-function's definition.

Action-Value Function (Qπ(s, a)) Definition — This diagram breaks down the definition of the action-value function, showing that it's the expected return after starting in state $s$, taking a specific action $a$, and then following policy $\pi$.

4. The Relationship Between V and Q

The state-value and action-value functions are not independent; they are intimately related. You can express one in terms of the other.

From Q to V

The value of being in a state, $V^{π} (s)$ , is the average of the action-values for all possible actions from that state, where each action-value is weighted by the probability of taking that action according to the policy $π$ .

In other words, you sum up the value of each possible action multiplied by its probability:

V^{π} (s) = a \in A \sum π (a ∣ s) Q^{π} (s, a)

From V to Q

The value of taking an action $a$ in a state $s$ , $Q^{π} (s, a)$ , is the immediate reward you expect to get, plus the discounted value of whatever state $s^{'}$ you land in. Since the next state is uncertain, we take an expectation over all possible next states and rewards.

Q^{π} (s, a) = s^{'}, r \sum p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]

This relationship is a precursor to the Bellman equations, which we'll study next. For now, let's watch a clear explanation of this link.

Bellman Equations, Dynamic Programming, Generalized Policy Iteration | Reinforcement Learning Part 2

The 'Mutual Information' channel provides a structured overview of the Bellman equations, and in doing so, clearly explains the relationship between V and Q.

Watch from 08:40 to 10:10. This segment visually lays out how the state-value and action-value functions are defined in terms of each other, forming a system of equations.

This recursive relationship is the engine that drives many RL algorithms. It allows us to compute the value of the current state/action based on the values of future states/actions.

Conclusion

In this lesson, you've learned to define the two most fundamental concepts for evaluating an agent's policy in reinforcement learning.

Key Takeaways:

Value functions measure the "goodness" of states or actions by calculating the expected, discounted, future reward.
The State-Value Function, $V^{π} (s)$ , is the expected return when starting in state $s$ and following policy $π$ . It tells you how good it is to be in a state. $V^{π} (s) = E_{π} [G_{t} ∣ S_{t} = s]$
The Action-Value Function, $Q^{π} (s, a)$ , is the expected return after taking action $a$ in state $s$ and then following policy $π$ . It tells you how good it is to take an action in a state. $Q^{π} (s, a) = E_{π} [G_{t} ∣ S_{t} = s, A_{t} = a]$
These two functions are directly related: $V^{π} (s)$ is the policy-weighted average of the $Q^{π} (s, a)$ values for that state.

Preview of the Next Lesson:

We have defined what value functions are, but how do we actually calculate them for a given policy? In the next lesson, we will formalize the recursive relationships we touched on today by deriving the Bellman equations. These equations form a system that can be solved to find the value functions, a critical step in policy evaluation and improvement. We will then explore the Bellman optimality equations, which provide a direct path to finding the best possible policy.

Recommended resources