Multi-step Transitions with Chapman-Kolmogorov

Welcome back. In our previous lesson, we focused on the crucial first step of setting up multi-state models: defining states and constructing the one-step transition probability matrices that govern movement between them. This provided us with the fundamental building block for modeling risk over time.

This lesson takes the logical next step. We will explore how to use these one-step matrices to calculate probabilities over multiple time periods. You'll learn how to answer practical questions like, "Given a policyholder is healthy today, what is the probability they will be in the 'sick' state three years from now?". Since you already have a strong grasp of the theoretical underpinnings of the Chapman-Kolmogorov equations, our focus will be squarely on their application and computational implementation in the context of the actuarial models we introduced.

The Logic of Multi-Step Transitions

At its core, calculating a multi-step transition probability involves considering all the possible paths a subject can take. To go from state $i$ to state $j$ in two steps, the subject must pass through some intermediate state $k$ at the end of the first step. The total probability is the sum of the probabilities of all possible intermediate paths.

This diagram illustrates the calculation for the total probability of transitioning from state H at time t=0 to state P at time t=2. It involves summing the probabilities of all possible paths through the intermediate states at t=1. This is the fundamental idea behind the Chapman-Kolmogorov equations.

Let's represent the probability of moving from state $i$ to state $j$ in one time step as $p_{ij}$ . The probability of being in state $j$ at time $n + 2$ , given we started in state $i$ at time $n$ , can be written by conditioning on the state at time $n + 1$ :

p_{ij}^{(2)} = Pr (X_{n + 2} = j ∣ X_{n} = i) = k \in S \sum Pr (X_{n + 2} = j, X_{n + 1} = k ∣ X_{n} = i)

Using the definition of conditional probability and the Markov property, this simplifies to:

p_{ij}^{(2)} = k \in S \sum Pr (X_{n} = i, X_{n + 1} = k) Pr (X_{n + 2} = j ∣ X_{n + 1} = k) = k \in S \sum p_{ik} p_{k j}

where $S$ is the set of all states. This summation is precisely the calculation for the $(i, j)$ -th entry when multiplying the one-step transition matrix by itself.

From Manual Calculation to Matrix Power

This path-summation logic can be seen in action in cohort simulations. The following video demonstrates how a cohort of 1,000 healthy individuals is tracked over several cycles in a simple 'healthy-diseased-dead' model. Pay close attention to how the number of people in each state for "Cycle 2" is calculated.

Markov modelling for Pharmacoeconomics.....Transition matrix simplified

This segment from "Markov modelling for Pharmacoeconomics" by Biomedical AI Basics provides a concrete numerical example of a two-step transition for an entire cohort.

Please watch the section on simulating Cycle 2. Notice how the number of people ending up in the 'Healthy' state at the end of Cycle 2 is a sum of contributions from those who were 'Healthy' at Cycle 1 and those who were in other states (if recovery were possible). This is a direct numerical application of the path-summation principle.

While this step-by-step approach is intuitive, it quickly becomes cumbersome for transitions over many time steps or for models with many states. A much more efficient method leverages matrix algebra. The Chapman-Kolmogorov equations, which you are familiar with, provide the formal justification for this.

The following video lecture provides a crisp connection between the path-summation logic and the far more powerful method of matrix multiplication.

Markov Processes (2025): Transition Probabilities and the Chapman-Kolmogorov Equations (Lecture 2)

This video from the "A Probability Space" channel expertly bridges the gap between the intuitive calculation of multi-step probabilities and the formal matrix-based approach.

First, watch the segment that demonstrates calculating a two-step probability. The lecturer shows how conditioning on the intermediate state leads to a calculation that is identical to a row-by-column multiplication in matrices. Next, watch the segment that formalizes this connection, showing that the n-step transition matrix is the one-step matrix raised to the nth power. This is the central computational tool for homogeneous Markov chains.

Calculating Multi-Step Probabilities in Actuarial Models

We can now formalize this powerful result. The key is to distinguish between homogeneous and non-homogeneous models, as we discussed in the last lesson.

[PDF] Multi-state transition models for actuarial applications - SOA

This section of the study note by James W. Daniel provides the definitive theorem for calculating longer-term transition probabilities in both homogeneous and non-homogeneous actuarial models.

Please read the following parts of Section 1.2: First, familiarize yourself with the notation for longer-term probabilities in Notation 1.16. Next, read Example 1.17, which walks through the logic of calculating a two-step probability for a simple homogeneous chain. Then, read Theorem 1.18. This is the main result we will use. It states that for a homogeneous chain, the k-step transition matrix is Q^k. For a non-homogeneous chain, the matrix for transitions from time n to n+k is the product Q_n \times Q_{n+1} \times \dots \times Q_{n+k-1}. Finally, see this applied in Example 1.20 for a homogeneous case (Q^2) and Example 1.21 for a non-homogeneous case (Q_2 Q_3 Q_4).

Let's summarize the key formulas from the reading.

1. Homogeneous Markov Chains
For a homogeneous chain, the one-step transition matrix $Q$ is constant over time. The probability of moving from state $i$ to state $j$ in $k$ steps is given by the $(i, j)$ -th entry of the matrix $Q^{k}$ .

_{k} Q = Q^{k}

Example: Consider the critical illness model from the slide deck Multiple State Models, with states 1 (Healthy), 2 (Critically Ill), and 3 (Dead).
The one-step transition matrix is:

Q = 0.92 0.00 0.00 0.05 0.76 0.00 0.03 0.24 1.00

To find the probability that a healthy person is critically ill after two years, we need to calculate the (1, 2) entry of $Q^{2}$ .

Q^{2} = Q \times Q = 0.92 0.00 0.00 0.05 0.76 0.00 0.03 0.24 1.00 0.92 0.00 0.00 0.05 0.76 0.00 0.03 0.24 1.00

Q^{2} = (0.92) (0.92) ⋮ \dots (0.92) (0.05) + (0.05) (0.76) ⋱ \dots \dots = 0.8464 00 0.084 0.5776 0 0.0696 0.4224 1

The probability is $_{2} Q^{(1, 2)} = 0.084$ .

2. Non-Homogeneous Markov Chains
For a non-homogeneous chain, the transition matrix $Q_{n}$ depends on the time $n$ . The matrix of transition probabilities from time $n$ to time $n + k$ , denoted $_{k} Q_{n}$ , is the product of the individual one-step matrices for each period.

_{k} Q_{n} = Q_{n} \times Q_{n + 1} \times \dots \times Q_{n + k - 1}

Note the order of multiplication is important.

Example: Consider the driver rating model from the slide deck, where an insured can be Preferred (State 0) or Standard (State 1). The transition matrix for year $t$ is given by:

Q_{t} = (0.65 0.50 0.35 0.50) + \frac{1}{t + 1} (0.15 - 0.20 - 0.15 0.20)

Let's find the probability that an insured who is Preferred at the start of year 1 (time $t = 1$ ) will be Standard at the start of year 3 (time $t = 3$ ). This is a 2-step transition from $t = 1$ to $t = 3$ . We need to calculate $_{2} Q_{1} = Q_{1} \times Q_{2}$ .

First, we find $Q_{1}$ (for $t = 1$ ) and $Q_{2}$ (for $t = 2$ ):

Q_{1} = (0.65 0.50 0.35 0.50) + \frac{1}{2} (0.15 - 0.20 - 0.15 0.20) = (0.725 0.400 0.275 0.600)

Q_{2} = (0.65 0.50 0.35 0.50) + \frac{1}{3} (0.15 - 0.20 - 0.15 0.20) \approx (0.700 0.433 0.300 0.567)

Now, we multiply them:

_{2} Q_{1} = Q_{1} \times Q_{2} = (0.725 0.400 0.275 0.600) (0.700 0.433 0.300 0.567) \approx (0.626 0.540 0.373 0.460)

The desired probability is the (0, 1) entry of this matrix: $_{2} Q_{1}^{(0, 1)} \approx 0.373$ .

Conclusion

In this lesson, we have moved from setting up one-step transition matrices to using them for multi-step predictions. The Chapman-Kolmogorov equations, while theoretically deep, lead to a straightforward and powerful computational method: matrix multiplication.

Key Takeaways:

The probability of transitioning from state $i$ to state $j$ over $k$ steps is found in the $(i, j)$ -th entry of the $k$ -step transition matrix.
For a homogeneous Markov chain with transition matrix $Q$ , the $k$ -step matrix is simply $Q^{k}$ .
For a non-homogeneous Markov chain, the $k$ -step matrix for a transition starting at time $n$ is the ordered product of the one-step matrices: $Q_{n} \times Q_{n + 1} \times \dots \times Q_{n + k - 1}$ .
This matrix approach is computationally efficient and forms the basis for nearly all practical calculations involving discrete-time Markov chains.

In our next lesson, we will build on this to address another critical set of questions in actuarial modeling. Using our ability to calculate multi-step probabilities, we will determine the probability of eventually being absorbed into states like 'Dead' or 'Lapsed', and calculate the expected time until that absorption occurs.

Recommended resources

The Logic of Multi-Step Transitions

From Manual Calculation to Matrix Power

Calculating Multi-Step Probabilities in Actuarial Models

Conclusion