Information, Probability, and Asymptotic Methods

Define Shannon entropy and derive the principle of maximum entropy (MaxEnt) for constructing equilibrium probability distributions.

Derive the general form of the exponential family of distributions (e.g., Gibbs distribution) from the MaxEnt principle subject to macroscopic constraints.

Compute moments and cumulants of distributions using moment-generating and characteristic functions.

Derive and apply the central limit theorem to sums of independent random variables.

Evaluate integrals asymptotically using the saddle-point (steepest descent) approximation.

Define the large deviation principle and the rate function, stating Cramér's theorem for sums of IID random variables.