Moments and Cumulants via Generating Functions

Welcome to our third lesson. In our previous session, we established a profound connection between information theory and statistical physics by showing how the Gibbs distribution, $p_{i} \propto e^{- β E_{i}}$ , arises from the principle of maximum entropy. We identified the Lagrange multiplier $β$ as the inverse temperature and discovered that the partition function, $Z$ , could be used to find the average energy of the system through the relation $U = - \frac{\partial l n Z}{\partial β}$ . This hinted that $Z$ might be more than just a normalization constant.

Today, we will formalize and expand upon that hint. Our learning goal is to master the computation of moments and cumulants using moment-generating and characteristic functions. We will see that the partition function is a specific physical instance of a moment-generating function, a powerful mathematical tool that "generates" not just the mean value of a quantity like energy, but its variance, skewness, and all higher-order statistical moments. This will equip us with the formal machinery to analyze fluctuations and other statistical properties of physical systems with rigor.

1. Moments and the Moment-Generating Function (MGF)

A probability distribution is not solely defined by its average value. Its full character is captured by an infinite set of quantities called moments. The $n$ -th moment of a random variable $X$ is defined as the expectation value of its $n$ -th power, $⟨ X^{n} ⟩$ .

The 1st moment, $⟨ X ⟩$ , is the mean.
The 2nd moment, $⟨ X^{2} ⟩$ , is used to find the variance: $Var (X) = ⟨ X^{2} ⟩ - ⟨ X ⟩^{2}$ .
Higher moments like skewness ( $⟨(X - ⟨ X ⟩)^{3} ⟩$ ) and kurtosis describe the asymmetry and "tailedness" of the distribution.

Calculating these moments one by one can be tedious. The moment-generating function (MGF) provides an elegant and unified way to access all of them. For a random variable $X$ , the MGF is a function of a dummy variable $t$ , defined as:

M_{X} (t) = ⟨ e^{tX} ⟩

The following video from Harvard's Statistics 110 course provides an excellent introduction to what an MGF is and why it is so useful.

Lecture 17: Moment Generating Functions | Statistics 110

This video introduces the Moment Generating Function (MGF), explains why it's named as such through its Taylor series expansion, and lists its three most important properties.

First, watch the definition of the MGF and the condition for its existence from the definition. Next, follow the crucial part where the lecturer shows how the Taylor series of the MGF contains all the moments of the distribution. This is the origin of its name. Watch from this segment. Finally, pay close attention to the summary of the three key properties of MGFs from this part. These are: It generates moments. It uniquely determines the distribution. It simplifies the analysis of sums of independent random variables.

As the video explains, the MGF "generates" moments in two equivalent ways. If we expand $M_{X} (t)$ as a Taylor series (Maclaurin series) around $t = 0$ :

M_{X} (t) = ⟨ n = 0 \sum \infty \frac{( tX ) ^{n}}{n !} ⟩ = n = 0 \sum \infty \frac{t ^{n}}{n !} ⟨ X^{n} ⟩

We can see that the $n$ -th moment, $⟨ X^{n} ⟩$ , is simply the coefficient of the $t^{n} / n!$ term. Alternatively, we can generate the moments by taking derivatives with respect to $t$ and then setting $t = 0$ :

⟨ X^{n} ⟩ = \frac{d ^{n} M _{X} ( t )}{d t ^{n}}_{t = 0}

2. The Characteristic Function and Cumulants

The MGF is a powerful tool, but it has one drawback: the expectation $⟨ e^{tX} ⟩$ may not be finite for all $t$ . A more general tool that always exists is the characteristic function, $ϕ_{X} (k)$ . You can think of this as an MGF with an imaginary argument, $t = ik$ , where $k$ is a real variable. Given your physics background, you will immediately recognize its form: it is the Fourier transform of the probability density function (PDF), $p (x)$ .

ϕ_{X} (k) = ⟨ e^{ik X} ⟩ = \int_{- \infty}^{\infty} e^{ik x} p (x) d x

From this, one can define another set of important quantities: the cumulants, denoted $⟨ X^{n} ⟩_{c}$ . They arise from the Taylor expansion of the logarithm of the generating function. The function $K_{X} (t) = ln M_{X} (t)$ or $K_{X} (k) = ln ϕ_{X} (k)$ is called the cumulant-generating function (CGF).

The following notes from an MIT Statistical Mechanics course provide the rigorous definitions and relationships.

Statistical Mechanics I

These lecture notes provide a concise and mathematically rigorous treatment of moments, characteristic functions, and cumulants.

Please read the sections titled "One Random Variable" starting on page 26. Begin with the definition of the nth moment, mn. Then, read the definitions of the characteristic function and how it generates moments. Note its definition as a Fourier transform. Finally, study the definition of the cumulant generating function and the relations between the first four cumulants and moments.

The first few cumulants are particularly important and have intuitive meanings:

$⟨ X ⟩_{c} = ⟨ X ⟩$ (the mean)
$⟨ X^{2} ⟩_{c} = ⟨ X^{2} ⟩ - ⟨ X ⟩^{2} = Var (X)$ (the variance)
$⟨ X^{3} ⟩_{c} = ⟨(X - ⟨ X ⟩)^{3} ⟩$ (the skewness)
$⟨ X^{4} ⟩_{c} = ⟨(X - ⟨ X ⟩)^{4} ⟩ - 3 (Var (X))^{2}$ (the excess kurtosis)

So, why introduce cumulants when we already have moments? Cumulants have a crucial property known as additivity or extensivity. If $X$ and $Y$ are independent random variables, then the cumulants of their sum are simply the sum of their individual cumulants:

⟨(X + Y)^{n} ⟩_{c} = ⟨ X^{n} ⟩_{c} + ⟨ Y^{n} ⟩_{c}

This property does not hold for moments (except for the mean). This extensivity makes cumulants the natural language for describing macroscopic observables in statistical mechanics, which are often sums over many microscopic, nearly independent parts.

The following notes from a CERN lecture make this point about extensivity exceptionally clear.

[PDF] Lecture on the cumulants (for XQCD2025) - CERN Indico

These notes, prepared for a high-energy physics school, give a very clear physicist's perspective on cumulants, focusing on their extensivity.

First, quickly review the definitions in Section 1, "Relation between moments and cumulants," which reinforces what you read in the MIT notes. Next, read Section 2, "Extensivity of cumulants," very carefully. The derivation showing that K_{\text{sum}}(\theta) = K_1(\theta) + K_2(\theta) is the heart of the argument.

3. Application to Statistical Mechanics

Now we can connect these formal tools back to the partition function. Recall the Gibbs distribution $p_{i} = e^{- β E_{i}} / Z$ and the canonical partition function $Z (β) = \sum_{i} e^{- β E_{i}}$ .

Let's find the MGF for the energy, $E$ .

M_{E} (t) = ⟨ e^{tE} ⟩ = i \sum p_{i} e^{t E_{i}} = \frac{1}{Z ( β )} i \sum e^{- β E_{i}} e^{t E_{i}} = \frac{1}{Z ( β )} i \sum e^{- (β - t) E_{i}}

Recognizing the sum in the numerator, we get:

M_{E} (t) = \frac{Z ( β - t )}{Z ( β )}

The cumulant-generating function for energy is therefore:

K_{E} (t) = ln M_{E} (t) = ln Z (β - t) - ln Z (β)

This is a powerful result. All statistical information about energy is encoded in $ln Z$ . Let's generate the first two cumulants by taking derivatives with respect to $t$ and evaluating at $t = 0$ .

First Cumulant (Mean Energy):

⟨ E ⟩_{c} = ⟨ E ⟩ = \frac{d K _{E} ( t )}{d t}_{t = 0} = \frac{- Z ^{'} ( β - t )}{Z ( β - t )}_{t = 0} = - \frac{Z ^{'} ( β )}{Z ( β )} = - \frac{\partial}{\partial β} ln Z (β)

This exactly reproduces the result from our last lesson!

Second Cumulant (Energy Variance):

⟨ E^{2} ⟩_{c} = Var (E) = \frac{d ^{2} K _{E} ( t )}{d t ^{2}}_{t = 0} = \frac{\partial ^{2}}{\partial t ^{2}} (ln Z (β - t))_{t = 0} = \frac{\partial ^{2}}{\partial β ^{2}} ln Z (β)

This is a new and extremely important result. The variance of the energy—a measure of the system's energy fluctuations—can be found by taking the second derivative of $ln Z$ . Using $β = 1/ (k_{B} T)$ and the chain rule, this can be related to the heat capacity:

Var (E) = ⟨ E^{2} ⟩ - ⟨ E ⟩^{2} = k_{B} T^{2} C_{V}

This is a fundamental fluctuation-dissipation relation, directly connecting microscopic fluctuations ( $Var (E)$ ) to a macroscopic response function ( $C_{V}$ ).

The same logic applies to other ensembles. For instance, in the grand canonical ensemble, the grand partition function $Ξ$ acts as a generating function for the cumulants of the particle number $N$ , with the conjugate variable being $μ / T$ . The CERN notes briefly show this.

[PDF] Lecture on the cumulants (for XQCD2025) - CERN Indico

This short section explicitly shows how the grand partition function generates the cumulants of particle number.

Please read the subsection "Cumulants of particle number N" in Section 4. This directly parallels our derivation for energy in the canonical ensemble.

4. Worked Example: The Gaussian Distribution

To solidify your computational skills, let's work through finding the MGF and cumulants for the most important distribution in statistical mechanics: the Normal or Gaussian distribution.

A Gaussian PDF with mean $λ$ and variance $σ^{2}$ is given by:

p (x) = \frac{1}{2 π σ ^{2}} exp (- \frac{( x - λ ) ^{2}}{2 σ ^{2}})

To find its MGF, we must compute the integral $M_{X} (t) = \int_{- \infty}^{\infty} e^{t x} p (x) d x$ . This involves the classic technique of completing the square in the exponent. The videos from Harvard and Steve Brunton both walk through this calculation, but the MIT notes present it most concisely.

Statistical Mechanics I

This section provides a compact derivation of the MGF for a Gaussian and then immediately extracts the cumulants.

In section "II.C Some Important Probability Distributions," focus on the subsection for the Gaussian. Follow the derivation of the characteristic function (eq. II.16), which is equivalent to the MGF with k \to it, and see how the cumulants are identified from its logarithm (eq. II.17).

The result of the calculation is remarkably simple. The MGF is:

M_{X} (t) = exp (λ t + \frac{σ ^{2} t ^{2}}{2})

From this, the cumulant-generating function is trivial:

K_{X} (t) = ln M_{X} (t) = λ t + \frac{σ ^{2} t ^{2}}{2}

Since this is just a quadratic in $t$ , its Taylor series terminates.

The coefficient of $t^{1} /1!$ is $λ$ , so $⟨ X ⟩_{c} = λ$ .
The coefficient of $t^{2} /2!$ is $σ^{2}$ , so $⟨ X^{2} ⟩_{c} = σ^{2}$ .
All higher coefficients are zero, so $⟨ X^{n} ⟩_{c} = 0$ for $n \geq 3$ .

This is a defining feature of the Gaussian distribution: it is entirely specified by its first two cumulants (mean and variance). The vanishing of all higher cumulants makes it a crucial reference point in many theoretical calculations, as you've seen in your QFT course.

Conclusion

In this lesson, we have built a formal framework for characterizing probability distributions beyond their mean values.

Key Takeaways:

Moments ( $⟨ X^{n} ⟩$ ) and cumulants ( $⟨ X^{n} ⟩_{c}$ ) provide a complete statistical description of a random variable.
The Moment-Generating Function ( $M_{X} (t) = ⟨ e^{tX} ⟩$ ) and Characteristic Function ( $ϕ_{X} (k) = ⟨ e^{ik X} ⟩$ ) provide a compact way to generate all moments via differentiation or Taylor expansion.
The Cumulant-Generating Function ( $K_{X} (t) = ln M_{X} (t)$ ) generates cumulants, which possess the key property of additivity (extensivity) for independent variables.
The partition functions of statistical mechanics (e.g., $Z$ , $Ξ$ ) are physical examples of generating functions. Their logarithms, which correspond to thermodynamic potentials like the free energy, are cumulant-generating functions for conserved quantities like energy and particle number.

We are now equipped with a powerful set of mathematical tools. In our next lesson, we will use the characteristic function to prove one of the most important theorems in all of science: the Central Limit Theorem. We will see why the Gaussian distribution appears so frequently and understand how the collective behavior of many microscopic components gives rise to simple, predictable macroscopic laws.

Recommended resources

1. Moments and the Moment-Generating Function (MGF)

2. The Characteristic Function and Cumulants

3. Application to Statistical Mechanics

4. Worked Example: The Gaussian Distribution

Conclusion