Method of Moments Estimation for Gamma Distribution Parameters

Introduction

Welcome to the next lesson in our exploration of statistical inference. In our previous discussion, we introduced the Method of Moments (MoM) as an intuitive "recipe" for creating estimators: you simply equate the theoretical properties of a distribution (its population moments) to the measured properties of your data (the sample moments) and solve for the unknown parameters. We saw how this works for simple one-parameter distributions like the Poisson.

Today, we will apply this technique to a more complex and highly practical case: the Gamma distribution. This two-parameter distribution is frequently used in engineering and science to model waiting times, component reliability, or service times—scenarios that may be familiar from your work in the EV industry.

Our learning goal is to rigorously derive the Method of Moments estimators for the two parameters of the Gamma distribution. This will involve:

Setting up a system of two equations based on the first and second moments.
Performing the algebraic manipulation to solve this system.
Translating the final theoretical formulas into a practical Python implementation.

This lesson directly addresses your request for detailed mathematical derivations, moving from the general concept to a concrete, multi-parameter application.

1. The Gamma Distribution: A Quick Refresher

The Gamma distribution is defined by two parameters: a shape parameter (let's use $k$ , or $α$ ) and a scale parameter (let's use $θ$ , or $1/ β$ ). Its probability density function (PDF) is given by:

f (x; k, θ) = \frac{1}{Γ ( k ) θ ^{k}} x^{k - 1} e^{- x / θ} for x > 0

where $Γ (k)$ is the Gamma function.

The shape and scale parameters give this distribution great flexibility, allowing it to model a wide variety of phenomena.

Gamma Distribution Probability Density Functions with Varying Parameters — This image shows how the shape and scale parameters alter the form of the Gamma distribution's PDF. For our derivation, we will be estimating these parameters directly from data.

To apply the Method of Moments, we need the first two population moments, which are functions of $k$ and $θ$ . For a Gamma-distributed random variable $X$ :

First Population Moment (Mean): $μ_{1}^{'} = E [X] = k θ$
Variance: $V a r (X) = k θ^{2}$

We can use these to find the second population moment. Recall the relationship:

V a r (X) = E [X^{2}] - (E [X])^{2}

Rearranging this gives us the second raw moment:

μ_{2}^{'} = E [X^{2}] = V a r (X) + (E [X])^{2} = k θ^{2} + (k θ)^{2}

These two equations, for $μ_{1}^{'}$ and $μ_{2}^{'}$ , are the starting point for our derivation.

2. Deriving the Method of Moments Estimators

The core task is to solve the following system of equations for $k$ and $θ$ :

$μ_{1}^{'} = k θ$
$μ_{2}^{'} = k θ^{2} + k^{2} θ^{2}$

This process is a great example of the general strategy for any multi-parameter distribution. The following video provides a clear, step-by-step walkthrough of this exact derivation.

1. Method of Moments Estimator

This video from the channel Chessability works through the complete derivation for the Gamma distribution's MoM estimators. It's a great visual guide to the algebraic steps involved.

Watch the segment from 02:17 to 08:18. The video uses 'a' for the shape parameter and 'lambda' for the rate parameter, where rate λ = 1/θ (scale). The logic is identical, so focus on how the system of equations is set up and solved algebraically. We will formalize the result using our k and θ notation afterward.

Now, let's formalize the derivation using our $k$ and $θ$ notation, following the rigorous four-step process.

Lecture 2 Method of Moments

This PDF of lecture notes from UC Berkeley provides a very concise and formal derivation of the estimators we are seeking. It's an excellent resource for seeing the mathematical argument laid out cleanly.

Please read Section 2.2, which outlines the formal four-step procedure for MoM, and then Section 2.4, which applies this procedure directly to the Gamma distribution. This reading solidifies the steps shown in the video.

Let's re-trace the key algebraic steps from the resources to solve for $k$ and $θ$ :

Isolate $θ$ : We can express the variance, $σ^{2}$ , in terms of our moments: $σ^{2} = μ_{2}^{'} - (μ_{1}^{'})^{2}$ . We also know that $σ^{2} = k θ^{2}$ .
Now, notice that $σ^{2} = k θ \cdot θ$ . Since $μ_{1}^{'} = k θ$ , we can substitute it in:
$σ^{2} = μ_{1}^{'} θ ⟹ θ = \frac{σ ^{2}}{μ _{1}^{'}}$
Substituting the moment expression for variance, we get:
$θ = \frac{μ _{2}^{'} - ( μ _{1}^{'} ) ^{2}}{μ _{1}^{'}}$
Isolate $k$ : Now we use the first equation, $μ_{1}^{'} = k θ$ , and solve for $k$ :
$k = \frac{μ _{1}^{'}}{θ}$
Substitute the expression for $θ$ we just found:
$k = \frac{μ _{1}^{'}}{( μ _{2}^{'} - ( μ _{1}^{'} ) ^{2} ) / μ _{1}^{'}} = \frac{( μ _{1}^{'} ) ^{2}}{μ _{2}^{'} - ( μ _{1}^{'} ) ^{2}}$
Find the Estimators: Finally, we obtain the MoM estimators $k^$ and $θ^$ by replacing the population moments $μ_{1}^{'}$ and $μ_{2}^{'}$ with their corresponding sample moments, $μ^_{1}^{'}$ and $μ^_{2}^{'}$ :
- First sample moment: $μ^_{1}^{'} = X ˉ = \frac{1}{n} \sum_{i = 1}^{n} X_{i}$
- Second sample moment: $μ^_{2}^{'} = \frac{1}{n} \sum_{i = 1}^{n} X_{i}^{2}$

This gives us our final estimators:

k^_{M o M} = \frac{X ˉ ^{2}}{\frac{1}{n} \sum X _{i}^{2} - X ˉ ^{2}}

θ^_{M o M} = \frac{\frac{1}{n} \sum X _{i}^{2} - X ˉ ^{2}}{X ˉ}

Notice that the term $\frac{1}{n} \sum X_{i}^{2} - X ˉ^{2}$ is simply the (biased) sample variance. This provides a nice, compact way to write the estimators:
$k^_{M o M} = \frac{mean ^{2}}{variance} and θ^_{M o M} = \frac{variance}{mean}$

3. (Optional but Recommended) Deriving the Moments from Scratch

In the derivation above, we took the formulas for the mean and variance of the Gamma distribution as given. For a deeper understanding, as you requested, we can derive these moments from first principles using the Moment Generating Function (MGF).

The MGF, $M_{X} (t) = E [e^{tX}]$ , is a powerful tool. The k-th population moment can be found by taking the k-th derivative of the MGF and evaluating it at $t = 0$ .

μ_{k}^{'} = E [X^{k}] = \frac{d ^{k}}{d t ^{k}} M_{X} (t)_{t = 0}

The following article walks through the calculus required to first find the MGF of the Gamma distribution and then use it to derive the moments we used. I highly recommend working through this to satisfy your interest in the underlying mathematics.

How to Use Method of Moments Like a Pro

This article provides a complete, self-contained derivation, starting from the Gamma PDF to find its MGF, and then differentiating the MGF to obtain the first and second moments. This is the mathematical foundation for the formulas we used earlier.

Read the first two sections, 'Obtain the moment generating function' and 'Derive the moments'. Pay close attention to the integral trick used to find the MGF and the application of the chain rule to find its derivatives.

4. From Theory to Practice: Python Implementation

Now that we have derived the formulas for $k^_{M o M}$ and $θ^_{M o M}$ , let's see them in action. The same article provides a clean Python implementation that directly translates our derived formulas into code.

How to Use Method of Moments Like a Pro

This reading shows how to implement the estimators we just derived and test them on simulated data.

Now, read the sections 'Obtain parameter estimators' and 'Calculate the estimates of the parameters'. Note how the Python functions sample_moment and estimate_pars are direct translations of the mathematical formulas we derived.

As the article demonstrates, when you generate a large sample from a Gamma distribution with known parameters (e.g., $k = 3, θ = 0.2$ ), the MoM estimators calculated from that sample ( $k^\approx 3.05, θ^\approx 0.197$ ) are quite close to the true values. This is a practical demonstration of the consistency of MoM estimators.

Conclusion

In this lesson, we moved from the general principle of the Method of Moments to a concrete, rigorous application. We successfully derived the estimators for the two parameters of a Gamma distribution by setting up and solving a system of two moment equations.

Key Takeaways:

The MoM "recipe" extends naturally to distributions with multiple parameters: for p parameters, you need to set up and solve a system of p equations using the first p moments.
The derivation often involves clever algebraic substitution to isolate each parameter in terms of the population moments.
The final estimators are found by "plugging in" the sample moments, which are always computable from the data.
The derived formulas can be directly and easily implemented in code to estimate parameters from a real dataset.

Self-Test

To check your understanding of the procedure, try to derive the Method of Moments estimators for the two parameters, $a$ and $b$ , of a Uniform distribution, $U (a, b)$ .

Hint 1: The first two population moments are $E [X] = \frac{a + b}{2}$ and $E [X^{2}] = \frac{a ^{2} + ab + b ^{2}}{3}$ .
Hint 2: It may be easier to first solve for the mean $μ = \frac{a + b}{2}$ and variance $σ^{2} = \frac{( b - a ) ^{2}}{12}$ , and then solve for $a$ and $b$ .

Preview of the Next Lesson:

We have now seen the MoM in detail. While it is simple and intuitive, it is not always the "best" method in terms of statistical properties like efficiency. In our next lesson, we will dive deep into Maximum Likelihood Estimation (MLE). We will derive the MLEs for several common distributions and, as you requested, prove the famous result that for a linear model with Gaussian noise, the MLE is equivalent to the Ordinary Least Squares (OLS) solution. This will provide a powerful link between probabilistic modeling and linear algebra.

Recommended resources