Kaplan-Meier Confidence Intervals: Greenwood's Formula

Quantifying Uncertainty: The Role of Standard Error

Just as we compute a confidence interval for a sample mean using its standard error, we can compute a pointwise confidence interval for the survival probability $\hat{S}(t)$ at any specific time $t$. For a 95% confidence level, the structure is familiar:

$\text{CI}(t)=\hat{S}(t)\pm 1.96\times \text{SE}[\hat{S}(t)]$

The main task, therefore, is to find the standard error, $\text{SE}[\hat{S}(t)]$, which is the square root of the variance, $\text{Var}[\hat{S}(t)]$. The most widely used formula for this variance is Greenwood's formula, sometimes called Greenwood's approximation.

Greenwood's Formula

Greenwood's formula provides an estimate for the variance of the Kaplan-Meier estimator $\hat{S}(t)$. It is derived using the delta method, a concept you will be familiar with from probability theory, applied to the product-limit form of the KM estimator.

The formula for the estimated variance is:

$\widehat{\text{Var}}[\hat{S}(t)]=[\hat{S}(t){]}^2{\sum }_{j:t_j\le t}\frac{d_j}{n_j(n_j-d_j)}$

Let's dissect this:

• $[\hat{S}(t){]}^2$: The variance is proportional to the square of the survival estimate itself. As survival probability decreases, so does the absolute size of the variance.
• ${\sum }_{j:t_j\le t}$: The summation is over all distinct event times $t_j$ up to or at time $t$. This means the variance accumulates over time.
• $\frac{d_j}{n_j(n_j-d_j)}$: This is the component of variance added at each event time $t_j$. It depends on $d_j$, the number of events, and $n_j$, the number at risk.

The standard error is simply the square root of this value:
$\text{SE}[\hat{S}(t)]=\sqrt{\widehat{\text{Var}}[\hat{S}(t)]}=\hat{S}(t)\sqrt{{\sum }_{j:t_j\le t}\frac{d_j}{n_j(n_j-d_j)}}$

The following video provides a clear, step-by-step walkthrough of calculating the variance and the resulting confidence interval for a small dataset. This will help you see the formula in action.

Practical Issues and Transformations

As the video and your statistical intuition might suggest, there's a practical problem with the straightforward confidence interval $\hat{S}(t)\pm 1.96\times \text{SE}[\hat{S}(t)]$. Since the survival function $S(t)$ is a probability, it must lie within the range [0, 1]. However, this "plain" or "linear" confidence interval can produce an upper bound greater than 1 or a lower bound less than 0, especially when $\hat{S}(t)$ is close to 1 or 0, or with small sample sizes.

To address this, it is standard practice to use transformations. We compute the confidence interval on a transformed scale where the range is unrestricted, and then transform the interval's endpoints back to the original [0, 1] scale. This ensures the resulting interval for $S(t)$ is always valid.

The two most common methods are:

1. Log Transformation: This method constructs the confidence interval for $\log (\hat{S}(t))$. Since $\log (S(t))$ ranges from $-\infty$ to 0, the back-transformed interval will be bounded by 0 and $\infty$. We then truncate any upper bound greater than 1. This prevents negative lower bounds. The resulting 95% CI for $S(t)$ is:
   $\left[\hat{S}(t)/\exp \left(1.96\frac{\text{SE}[\hat{S}(t)]}{\hat{S}(t)}\right),\hat{S}(t)\cdot \exp \left(1.96\frac{\text{SE}[\hat{S}(t)]}{\hat{S}(t)}\right)\right]$
   Notice that $\frac{\text{SE}[\hat{S}(t)]}{\hat{S}(t)}=\sqrt{{\sum }_{j:t_j\le t}\frac{d_j}{n_j(n_j-d_j)}}$, which is the standard error of $\log (\hat{S}(t))$.

2. Log-Log Transformation: This method uses the transformation $\log (-\log (S(t)))$, which maps the [0, 1] interval for $S(t)$ to the entire real line ($-\infty ,\infty$). A confidence interval constructed on this scale and transformed back will always lie within (0, 1), without any need for truncation. This is often the preferred method in software.

The following reading provides a clear summary of these three approaches ("plain", "log", and "log-log") and discusses their relative merits.

Worked Example Continued

Let's continue the example from our last lesson to solidify these concepts.

Data (n=21): 6, 6, 6, 6+, 7, 9+, 10, 10+, 11+, 13, 16, 17+, 19+, 20+, 22, 23, 25+, 32+, 32+, 34+, 35+

In the previous lesson, we started building the KM table. Let's add columns for the variance calculation. The key column we need is for the term $\frac{d_j}{n_j(n_j-d_j)}$.

Let's calculate the 95% "plain" confidence interval for the survival probability at t = 16.

1. Find the KM estimate: From the table, $\hat{S}(16)=0.627$.
2. Find the variance sum: The cumulative sum of the variance terms up to $t=16$ is 0.03305.
3. Calculate the Standard Error:
   $\text{SE}[\hat{S}(16)]=\hat{S}(16)\sqrt{\sum \dots }=0.627\times \sqrt{0.03305}=0.627\times 0.1818=0.1140$
4. Construct the "Plain" CI:
   $\text{CI}(16)=0.627\pm 1.96\times 0.1140=0.627\pm 0.2234=[0.404,0.850]$

Now, let's calculate the log-transformed 95% CI at t = 16.

1. Find the SE of the log-survival:
   $\text{SE}[\log (\hat{S}(16))]=\sqrt{\sum \dots }=\sqrt{0.03305}=0.1818$
2. Construct CI on log scale:
   $\log (0.627)\pm 1.96\times 0.1818=-0.4668\pm 0.3563=[-0.8231,-0.1105]$
3. Transform back:
   $[\exp (-0.8231),\exp (-0.1105)]=[0.439,0.895]$

Notice the asymmetry of the log-transformed interval compared to the plain one. As mentioned in the resources, this is the default method in R's survival package (conf.type='log').

The following video shows how different confidence interval types (plain, log, log-log) can be specified and plotted in R. This provides excellent context for how these theoretical options translate into practical data analysis.

Conclusion

In this lesson, we have added a crucial layer of interpretation to the Kaplan-Meier curve by quantifying its uncertainty. You are now equipped to not only estimate a survival function but also to provide a statistically sound range for its true value.

Key Takeaways:

• The uncertainty of the Kaplan-Meier estimate $\hat{S}(t)$ is captured by a pointwise confidence interval.
• Greenwood's formula is the industry standard for estimating the variance of $\hat{S}(t)$. The variance is calculated cumulatively at each event time.
• The "plain" or "linear" confidence interval, $\hat{S}(t)\pm 1.96\times \text{SE}[\hat{S}(t)]$, can yield bounds outside the valid [0, 1] range.
• To solve this, transformations (log, and especially log-log) are used to construct the interval on an unbounded scale before transforming the endpoints back. These methods are the defaults in most statistical software packages.

In our next lesson, we will explore a different but related non-parametric approach: the Nelson-Aalen estimator. Instead of estimating the survival function $S(t)$ directly, it estimates the cumulative hazard function $H(t)$. This will provide another valuable tool for your survival analysis toolkit.