Asymptotic Distributions of Transformed Estimators: The Delta Method

The Univariate Delta Method

Intuition from Taylor Series

The core idea behind the Delta Method is surprisingly straightforward and relies on the Taylor series expansion, a concept fundamental in physics and mathematics.

Suppose we have an estimator $Y_n$ that is asymptotically normal and converges in probability to a parameter $\theta$. For large $n$, $Y_n$ will be close to $\theta$. If we are interested in a transformed parameter $g(\theta )$, estimated by $g(Y_n)$, we can use a first-order Taylor expansion of $g(Y_n)$ around the point $\theta$:

$g(Y_n)\approx g(\theta )+g^{\prime }(\theta )(Y_n-\theta )$

This linear approximation suggests that the variability of $g(Y_n)$ is simply the variability of $Y_n$ scaled by the factor $|g^{\prime }(\theta )|$. Since $Y_n$ is asymptotically normal, this linear function of it should also be asymptotically normal. The Delta Method formalizes this intuition.

Formal Statement and Proof

To make this rigorous, we need a result that we'll explore in depth in our next lesson: Slutsky's Theorem. For our purposes today, we can state it as follows:

• Slutsky's Theorem (preview): If $X_n\stackrel{d}{\to }X$ and $Y_n\stackrel{p}{\to }c$ (where $c$ is a constant), then $X_n{Y}_n\stackrel{d}{\to }cX$.

With this tool, we can formally state and prove the Delta Method.

Theorem (The Delta Method):
Let $\{Y_n\}$ be a sequence of random variables such that
$\sqrt{n}(Y_n-\theta )\stackrel{d}{\to }N(0,{\sigma }^2)$
Let $g$ be a function that is differentiable at $\theta$ with $g^{\prime }(\theta )\ne 0$. Then,
$\sqrt{n}(g(Y_n)-g(\theta ))\stackrel{d}{\to }N(0,[g^{\prime }(\theta ){]}^2{\sigma }^2)$

This implies that for large $n$, the distribution of $g(Y_n)$ can be approximated by:
$g(Y_n)\approx N\left(g(\theta ),\frac{[g^{\prime }(\theta ){]}^2{\sigma }^2}{n}\right)$

The following video provides a clear statement of the theorem, a rigorous proof using the Mean Value Theorem (a close relative of the Taylor expansion) and Slutsky's Theorem, and a concrete example.

For a concise written summary of the theorem and its proof, you can refer to the following resource.

Applying the Univariate Delta Method

Let's work through a common application: finding the asymptotic distribution of the log-odds.

Suppose we conduct a survey of $n$ individuals and find that $X$ of them support a policy. The number of supporters $X$ follows a binomial distribution, $X\sim Bin(n,p)$. Our estimator for the population proportion $p$ is the sample proportion, $\hat{p}=X/n$.

From the CLT, we know the asymptotic distribution of $\hat{p}$:
$\sqrt{n}(\hat{p}-p)\stackrel{d}{\to }N(0,p(1-p))$

Now, suppose we are interested in the log-odds of success, $\eta =\log \left(\frac{p}{1-p}\right)$. Our estimator for $\eta$ is $\hat{\eta }=g(\hat{p})=\log \left(\frac{\hat{p}}{1-\hat{p}}\right)$. What is the asymptotic distribution of $\hat{\eta }$?

We can apply the Delta Method:

1. Identify components:

   • Our estimator is $Y_n=\hat{p}$.
   • The parameter is $\theta =p$.
   • The asymptotic variance of $\sqrt{n}(Y_n-\theta )$ is ${\sigma }^2=p(1-p)$.
   • The transformation is $g(y)=\log \left(\frac{y}{1-y}\right)=\log (y)-\log (1-y)$.

2. Calculate the derivative:
   $g^{\prime }(y)=\frac{1}{y}-\frac{-1}{1-y}=\frac{1}{y}+\frac{1}{1-y}=\frac{1}{y(1-y)}$

3. Evaluate the derivative at the parameter $\theta =p$:
   $g^{\prime }(p)=\frac{1}{p(1-p)}$

4. Apply the Delta Method formula:
   The asymptotic variance of $\sqrt{n}(\hat{\eta }-\eta )$ is $[g^{\prime }(p){]}^2{\sigma }^2$.
   $[g^{\prime }(p){]}^2{\sigma }^2={\left(\frac{1}{p(1-p)}\right)}^2\cdot p(1-p)=\frac{1}{p(1-p)}$

5. State the result:
   $\sqrt{n}\left(\log \left(\frac{\hat{p}}{1-\hat{p}}\right)-\log \left(\frac{p}{1-p}\right)\right)\stackrel{d}{\to }N\left(0,\frac{1}{p(1-p)}\right)$
   This result is fundamental in logistic regression and the analysis of contingency tables. It allows us to compute standard errors and confidence intervals for odds ratios.

The Multivariate Delta Method

The Delta Method can be extended to functions of multiple asymptotically normal random variables. This is particularly useful when a parameter of interest is a function of several other parameters, such as a correlation coefficient or the ratio of two means.

The intuition is the same, but the derivative is now a Jacobian matrix, and the scaling involves a quadratic form. This is directly analogous to the formula for propagation of uncertainty used in experimental physics.

Theorem (Multivariate Delta Method):
Let $\{{\mathbf{Y}}_n\}$ be a sequence of $k$-dimensional random vectors such that
$\sqrt{n}({\mathbf{Y}}_n-\mathbf{\theta })\stackrel{d}{\to }{N}_k(0,\mathbf{\Sigma })$
where $\mathbf{\Sigma }$ is the $k\times k$ covariance matrix.
Let $g:{\mathbb{R}}^k\to {\mathbb{R}}^m$ be a function with continuous first partial derivatives. Let $\nabla g(\mathbf{\theta })$ be the $m\times k$ Jacobian matrix of $g$ evaluated at $\mathbf{\theta }$. Then,
$\sqrt{n}(g({\mathbf{Y}}_n)-g(\mathbf{\theta }))\stackrel{d}{\to }{N}_m(0,\nabla g(\mathbf{\theta })\mathbf{\Sigma }[\nabla g(\mathbf{\theta }){]}^T)$

The term $\nabla g(\mathbf{\theta })\mathbf{\Sigma }[\nabla g(\mathbf{\theta }){]}^T$ is the new $m\times m$ asymptotic covariance matrix.

The following resource provides a concise statement of the multivariate theorem and a solved example.

Application: Ratio of Two Means

Let's consider an example. Suppose we have pairs of observations $(X_i,Y_i)$ and we are interested in the ratio of the population means, $R={\mu }_x/{\mu }_y$. We estimate this with $\hat{R}={\bar{X}}_n/{\bar{Y}}_n$.

From the multivariate CLT, we have:
$\sqrt{n}\left(\begin{array}{c}{\bar{X}}_n-{\mu }_x\\ {\bar{Y}}_n-{\mu }_y\end{array}\right)\stackrel{d}{\to }{N}_2\left(\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}{\sigma }_x^2& {\sigma }_{xy}\\ {\sigma }_{xy}& {\sigma }_y^2\end{array}\right)\right)$

1. Identify components:

   • ${\mathbf{Y}}_n=({\bar{X}}_n,{\bar{Y}}_n{)}^T$
   • $\mathbf{\theta }=({\mu }_x,{\mu }_y{)}^T$
   • $\mathbf{\Sigma }=\left(\begin{array}{cc}{\sigma }_x^2& {\sigma }_{xy}\\ {\sigma }_{xy}& {\sigma }_y^2\end{array}\right)$
   • The function is $g(x,y)=x/y$.

2. Calculate the Jacobian (gradient vector in this case, since $m=1$):
   $\nabla g(x,y)=\left[\frac{\partial g}{\partial x}\frac{\partial g}{\partial y}\right]=\left[\frac{1}{y}-\frac{x}{y^2}\right]$

3. Evaluate at $\mathbf{\theta }$:
   $\nabla g({\mu }_x,{\mu }_y)=\left[\frac{1}{{\mu }_y}-\frac{{\mu }_x}{{\mu }_y^2}\right]$

4. Apply the formula $\nabla g\mathbf{\Sigma }(\nabla g{)}^T$:
   The asymptotic variance of $\sqrt{n}(\hat{R}-R)$ is:

   $$\left[\begin{array}{cc}\frac{1}{{\mu }_y}& -\frac{{\mu }_x}{{\mu }_y^2}\end{array}\right]\left[\begin{array}{cc}{\sigma }_x^2& {\sigma }_{xy}\\ {\sigma }_{xy}& {\sigma }_y^2\end{array}\right]\left[\begin{array}{c}\frac{1}{{\mu }_y}\\ -\frac{{\mu }_x}{{\mu }_y^2}\end{array}\right]$$

   Performing the matrix multiplication gives:

   $$=\frac{1}{{\mu }_y^2}{\sigma }_x^2-\frac{2{\mu }_x}{{\mu }_y^3}{\sigma }_{xy}+\frac{{\mu }_x^2}{{\mu }_y^4}{\sigma }_y^2={\left(\frac{{\mu }_x}{{\mu }_y}\right)}^2\left(\frac{{\sigma }_x^2}{{\mu }_x^2}-\frac{2{\sigma }_{xy}}{{\mu }_x{\mu }_y}+\frac{{\sigma }_y^2}{{\mu }_y^2}\right)$$

   This final expression gives the asymptotic variance for the ratio estimator, allowing for the construction of confidence intervals.

Conclusion

The Delta Method is an indispensable tool in the statistician's toolkit, acting as a bridge between the Central Limit Theorem and a vast range of practical estimation problems.

Key Takeaways:

• The Delta Method provides the asymptotic normal distribution for a differentiable function of an asymptotically normal estimator.
• The method is fundamentally a rigorous application of a first-order Taylor series approximation.
• In the univariate case, the new asymptotic variance is the original asymptotic variance multiplied by the square of the function's derivative.
• In the multivariate case, the new asymptotic covariance matrix is found by pre- and post-multiplying the original covariance matrix by the Jacobian matrix and its transpose, a procedure analogous to error propagation in physics.

Preview of the Next Lesson:

In our derivation of the Delta Method, we used a powerful result called Slutsky's Theorem to combine limits. This theorem is a key piece of machinery in asymptotic theory. In the next lesson, we will formally define Slutsky's Theorem and explore its applications in combining sequences of random variables that converge in different modes (in distribution and in probability).