How Discounting Shapes the Effective Planning Horizon

The Myopic-to-Farsighted Spectrum

The discount factor $\gamma$, a value between 0 and 1, directly controls the trade-off between immediate and future rewards. Let's start by considering the two extremes:

• Myopic Agent ($\gamma =0$): If $\gamma =0$, the formula for the discounted return $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$ collapses to $G_t=R_{t+1}$. The agent only cares about the very next reward. This creates a purely "greedy" or shortsighted agent that will always choose the action leading to the best immediate outcome, even if it leads to disastrous results later on.
• Farsighted Agent ($\gamma \to 1$): As $\gamma$ approaches 1, the agent becomes increasingly farsighted. The weights ${\gamma }^k$ applied to future rewards decrease very slowly, meaning rewards far in the future are valued almost as much as immediate ones. The agent is willing to endure short-term pain for long-term gain.

David Silver's lecture provides a concrete example of how the values of states change based on the discount factor. The "Student MRP" example shows a dramatic shift in the desirability of different states when moving from a myopic to a farsighted perspective.

A Probabilistic Interpretation of Discounting

While discounting is mathematically necessary for convergence, it also has a powerful and intuitive interpretation: it can be seen as a model of survival.

Imagine that at every time step, there is a probability $\gamma$ that the process continues and a probability $(1-\gamma )$ that it terminates (or "dies"). A reward $R_{t+k+1}$ received $k$ steps in the future can only be collected if the process has "survived" for those $k$ steps. The probability of surviving $k$ consecutive steps is ${\gamma }^k$.

Therefore, the discounted return $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$ can be interpreted as the expected total reward, where the expectation accounts for the possibility of termination at each step. This provides a compelling justification for why future rewards are worth less: there is an inherent uncertainty about whether we will ever reach the point to collect them.

The following reading briefly introduces this "morbid" but powerful way of thinking about the discount factor.

The Effective Horizon: $H\approx 1/(1-\gamma )$

This probabilistic interpretation leads us to a quantitative relationship between the discount factor and the agent's planning horizon. If the probability of termination at any given step is $p=1-\gamma$, then the process follows a geometric distribution for its lifetime. The expected number of steps before termination is $1/p$.

This gives us the concept of the effective horizon, $H$. It is the characteristic timescale over which an agent using discount factor $\gamma$ is planning. We can approximate this horizon as:
$H\approx \frac{1}{1-\gamma }$
This formula provides a rule of thumb for setting $\gamma$. If you believe the significant events in your problem happen within about 50 steps, you might choose a $\gamma$ such that $1/(1-\gamma )=50$, which gives $\gamma =0.98$.

The following lecture notes formalize this idea and examine its mathematical underpinnings.

How "Effective" is the Effective Horizon?

Let's investigate the claim that $H=1/(1-\gamma )$ is a meaningful horizon. How much is a reward received exactly at time step $H$ worth today? Its present value is ${\gamma }^H{R}_H$. The weight on this reward is ${\gamma }^H={\gamma }^{1/(1-\gamma )}$.

Given your background, you'll recognize the form of the famous limit for $e$. For values of $\gamma$ close to 1, let $ϵ=1-\gamma$. Then $\gamma =1-ϵ$ and $H=1/ϵ$. The weight becomes $(1-ϵ{)}^{1/ϵ}$. As $\gamma \to 1$, $ϵ\to 0$, and we have:
${\lim }_{ϵ\to 0^{+}}(1-ϵ{)}^{1/ϵ}=e^{-1}\approx 0.368$
This is a remarkable result. It means that for any sufficiently high $\gamma$, a reward received at the effective horizon is discounted to roughly 37% of its original value. Rewards beyond this point are diminished much more significantly.

Let's check this for a few values:

• $\gamma =0.9⟹H=10$. Weight ${\gamma }^H=0.9^{10}\approx 0.349$.
• $\gamma =0.99⟹H=100$. Weight ${\gamma }^H=0.99^{100}\approx 0.366$.
• $\gamma =0.999⟹H=1000$. Weight ${\gamma }^H=0.999^{1000}\approx 0.368$.

This confirms that $H=1/(1-\gamma )$ is a very natural definition for the planning horizon. It's the point where rewards retain a substantial, yet significantly diminished, portion of their value.

Total Value and the Horizon

There's one more elegant connection. In the last lesson, you saw that if an agent receives a constant maximum reward $R_{\text{max}}$ forever, the total discounted return is $V_{\text{max}}=R_{\text{max}}/(1-\gamma )$.

If we substitute our definition of the effective horizon, $H=1/(1-\gamma )$, we get:
$V_{\text{max}}=H\cdot R_{\text{max}}$
This provides a final, compelling intuition. The total value an agent can ever hope to accumulate in a discounted infinite-horizon problem is equivalent to the value it would get by receiving the maximum possible reward for a finite period equal to the effective horizon, with no discounting at all.

This idea is wonderfully illustrated in a lecture from MIT, which demonstrates the calculation for an infinite stream of rewards.

Conclusion

This lesson concludes our foundational tour of Markov Decision Processes. We've moved beyond a simple definition of $\gamma$ as a mathematical trick and established its role as the master controller of an agent's planning strategy.

Here are the key takeaways:

• The discount factor $\gamma$ places an agent on a spectrum from purely myopic ($\gamma =0$) to infinitely farsighted ($\gamma \to 1$).
• $\gamma$ can be intuitively understood as a continuation probability, where $(1-\gamma )$ is the probability of the process terminating at each step.
• This interpretation leads to the definition of an effective planning horizon $H\approx 1/(1-\gamma )$, which is the expected lifetime of the process.
• At this effective horizon, rewards are discounted to approximately $e^{-1}\approx 37\%$ of their nominal value, providing a quantitative anchor for what "horizon" means.
• The maximum achievable discounted return is equivalent to receiving the maximum reward for a duration of $H$ undiscounted steps.

We have now formally defined all the components of a finite MDP. In the next module, we will put this theory into action. You will learn how to translate a real-world sequential decision problem into the MDP formalism and, crucially, how to represent the states, actions, transitions, and rewards in Python using NumPy arrays. This will be our first step towards building agents that can actually solve these problems.