Shannon Entropy & Maximum Entropy Principle

Welcome to your first lesson in the Foundations of Statistical Mechanics. This course is designed to provide you with a rigorous, conceptually deep understanding of the principles that bridge the microscopic world of particles with the macroscopic world of thermodynamics. We will approach this from a theorist's perspective, emphasizing mathematical structure and foundational logic, which will prepare you for advanced topics in condensed matter and statistical field theory.

This first module focuses on the informational and probabilistic underpinnings of statistical mechanics. Today, we will address the most fundamental question: on what basis do we assign probabilities to the microscopic states of a system? To answer this, we will define Shannon entropy as a measure of uncertainty and then derive the Principle of Maximum Entropy (MaxEnt). This powerful principle, first articulated by E.T. Jaynes, provides a rational and unbiased method for constructing equilibrium probability distributions based solely on macroscopic observations.

1. Quantifying Uncertainty: Shannon Entropy

Before we can build probability distributions, we need a way to measure what we don't know. The concept of "entropy" in statistical mechanics is profoundly connected to the idea of "missing information" from information theory. The key insight, developed by Claude Shannon, is that we can quantify this missing information in a mathematically precise way.

The derivation starts with a simple, intuitive idea: the "surprise" of an event. If an event is very likely, we are not surprised when it occurs. If it is very unlikely, its occurrence is surprising. Shannon proposed that any reasonable measure of surprise, $s (p)$ , for an event with probability $p$ should satisfy a few simple axioms:

It should be a continuous function of the probability $p$ .
It should be a decreasing function of $p$ : lower probability means higher surprise.
For two independent events, the total surprise should be the sum of their individual surprises.

Remarkably, these simple axioms are enough to uniquely determine the functional form of surprise.

Information Theory, Lecture 1: Defining Entropy and Information - Oxford Mathematics 3rd Yr Lecture

To see how these axioms lead to a unique mathematical form, please watch the following segment from a lecture on Information Theory by Prof. Sam Cohen at the University of Oxford.

Watch the portion of the video where the concept of "surprise" is introduced and its functional form is derived from the axioms. This is a beautiful example of how simple, logical requirements can lead to a powerful mathematical result. Please watch from the axiomatic derivation.

As you've seen, the only function that satisfies these properties is $s (p) = - lo g (p)$ (up to a constant multiplicative factor).

Shannon then defined entropy as the average surprise over all possible outcomes of a random variable. If a random variable $X$ can take on values ${x_{i}}$ with probabilities ${p_{i}}$ , its entropy $H (X)$ is:

H (X) = ⟨ s (p_{i})⟩ = i \sum p_{i} s (p_{i}) = - i \sum p_{i} lo g p_{i}

The base of the logarithm is a matter of convention. In information theory, base 2 is common, and the unit of entropy is the "bit." In physics, we use the natural logarithm (base $e$ ) and include Boltzmann's constant $k_{B}$ as the multiplicative factor for dimensional consistency. This gives us the Gibbs entropy:

S = - k_{B} i \sum p_{i} ln p_{i}

This quantity is maximized when our uncertainty is greatest (all $p_{i}$ are equal, a uniform distribution) and is zero when there is no uncertainty (one $p_{i}$ is 1 and all others are 0).

The following clip continues the Oxford lecture, formally defining entropy and illustrating it with the simple, yet fundamental, example of a biased coin.

Information Theory, Lecture 1: Defining Entropy and Information - Oxford Mathematics 3rd Yr Lecture

This part of the video formally introduces the entropy formula and explores its behavior for a simple two-state system.

Please watch from the formal definition of entropy at the definition, and then the example of the biased coin from the example.

2. The Principle of Maximum Entropy (MaxEnt)

Now that we have a tool to quantify missing information, we can address the central problem. Imagine a physical system (like a gas in a box) that can be in one of a vast number of microstates $i$ (specified by the positions and momenta of all particles). We typically don't know the exact microstate. Instead, we know a few macroscopic quantities, which are averages over the microstates. For example, we might know the average energy $⟨ E ⟩$ :

⟨ E ⟩ = i \sum p_{i} E_{i} = U

where $p_{i}$ is the probability of being in microstate $i$ and $E_{i}$ is the energy of that state.

The principle of maximum entropy states that the probability distribution ${p_{i}}$ we should use is the one that maximizes the Gibbs entropy $S$ , subject to the constraints of our knowledge (like the fixed average energy $U$ ).

Why is this the right thing to do? Any other choice of ${p_{i}}$ would imply that we have more information than we've actually stated, effectively making an unwarranted assumption. By maximizing entropy, we choose the distribution that is maximally non-committal about the information we don't have. It's the most honest representation of our state of knowledge.

To find this distribution, we use the method of Lagrange multipliers, a standard technique for constrained optimization that you'll be familiar with from your mechanics and mathematics courses. We want to maximize $S [p_{i}]$ subject to constraints, like $\sum p_{i} = 1$ and $\sum p_{i} E_{i} = U$ .

The following video provides a clear, step-by-step derivation on a whiteboard, showing how this optimization problem is set up and solved.

Stanford CS229: Machine Learning | Summer 2019 | Lecture 19 - Maximum Entropy and Calibration

This lecture from a Stanford course on Machine Learning walks through the mathematical derivation of the MaxEnt probability distribution. While the context is machine learning, the mathematical principle is identical to its use in statistical physics.

First, watch the instructor set up the problem: maximizing entropy subject to linear constraints derived from data (or, in our case, macroscopic measurements). Pay attention to how the constraints are formulated. Watch from the setup. Next, follow the core of the derivation using the Lagrangian method. The key moment is seeing how the logarithmic form of entropy naturally leads to an exponential form for the resulting probability distribution. The derivation starts at the Lagrangian and concludes with the final form of the distribution at the final form.

The crucial result of this derivation is that the probability distribution that maximizes entropy subject to constraints on the average values of some quantities $f_{k} (i)$ always takes the form of an exponential family distribution:

p_{i} = \frac{1}{Z} exp (- k \sum λ_{k} f_{k} (i))

Here, the $λ_{k}$ are the Lagrange multipliers associated with each constraint, and $Z$ is a normalization constant called the partition function:

Z = i \sum exp (- k \sum λ_{k} f_{k} (i))

This exponential form is one of the most fundamental and recurring structures in all of statistical mechanics.

3. MaxEnt and the Ensembles of Statistical Mechanics

This result may seem abstract, but it has immediate and profound physical consequences. The different "ensembles" of statistical mechanics are nothing more than the MaxEnt distributions that arise from different sets of physical constraints.

Let's see this in action by reading a concise summary from a set of lecture notes by M. Evans.

Statistical Physics—Section 2: Assignment of Probabilities ...

This text applies the general MaxEnt procedure you've just seen to derive the probability distributions for the standard statistical mechanical ensembles.

Please read the first two sections of the document. In Section 2.1, "Assignment of Probability", focus on the procedure of using Lagrange multipliers. Then, follow Example 1, where the only constraint is normalization. This shows that in the absence of any other information, MaxEnt gives a uniform probability distribution. This is the Microcanonical Ensemble assumption, or the "principle of equal a priori probabilities." Next, examine Example 2 and Section 2.2, "Formulation of Statistical Mechanics". Here, constraints on average energy \langle E \rangle and average particle number \langle N \rangle are imposed. Notice how this immediately yields the familiar forms of the Canonical and Grand Canonical distributions, which you've likely encountered in your prior coursework.

As the text shows, the MaxEnt framework provides a unified and logical foundation for all of equilibrium statistical mechanics. It replaces postulates with a principle of rational inference. The different ensembles are not arbitrary constructs; they are the inevitable consequence of applying this principle to systems with different physical constraints (isolated, closed, or open).

Conclusion

In this lesson, we have established the informational basis for assigning probabilities in statistical mechanics.

Key Takeaways:

Shannon (or Gibbs) Entropy $S = - k_{B} \sum p_{i} ln p_{i}$ provides a rigorous measure of uncertainty or "missing information" associated with a probability distribution.
The Principle of Maximum Entropy (MaxEnt) directs us to choose the probability distribution that maximizes this entropy, subject to the constraints imposed by our macroscopic knowledge. This yields the most unbiased inference possible.
Applying MaxEnt via the method of Lagrange multipliers invariably leads to a distribution from the exponential family, $p_{i} \propto exp (- \sum λ_{k} f_{k} (i))$ .
This single framework successfully derives the probability distributions for all the standard equilibrium ensembles (Microcanonical, Canonical, and Grand Canonical) based on the relevant physical constraints.

In our next lesson, we will solidify these connections. We will take a closer look at the general form of the exponential family of distributions and, crucially, identify the physical meaning of the Lagrange multipliers ( $λ_{k}$ ). This will allow us to build the "bridge equations" that explicitly connect the microscopic partition function $Z$ to macroscopic thermodynamic potentials like free energy, fully merging the languages of information theory and thermodynamics.

Recommended resources

1. Quantifying Uncertainty: Shannon Entropy

2. The Principle of Maximum Entropy (MaxEnt)

3. MaxEnt and the Ensembles of Statistical Mechanics

Conclusion