Formal Foundations of Markov Decision Processes

Distinguish a Markov chain, a Markov reward process, and a Markov decision process.

Specify a finite MDP using state and action sets, a transition kernel, a reward convention, a discount factor, and an initial-state condition.

Determine whether a proposed state representation satisfies the Markov property.

Distinguish deterministic, stochastic, stationary, and time-dependent policies.

Compute the probability of a trajectory under a specified policy, including its initial-state probability.

Compute finite-horizon and discounted returns for a given reward sequence.

Explain how the discount factor changes the effective planning horizon.