Weight Initialization: Xavier, He, and Beyond

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our last lesson, we examined Dropout and Batch Normalization, two powerful techniques for improving the training of deep neural networks. We saw that Batch Normalization, by stabilizing the distribution of layer inputs, significantly reduces the network's sensitivity to the initial scale of its weights. This allows for faster and more stable training.

However, even with Batch Normalization, the fundamental problem of how to set the initial weights remains. A poor choice can still hinder learning, especially in architectures without normalization layers. If weights are too small, the signal propagating through the network can shrink to nothing; if too large, it can explode to infinity or cause activation functions to saturate. This leads to the infamous vanishing and exploding gradient problems.

Today, we will tackle this issue from first principles. Our goal is to derive a principled method for initializing weights that promotes a stable flow of information. We will analyze how the variance of activations and gradients changes as it propagates through the network and use this analysis to derive two cornerstone initialization strategies: Xavier/Glorot initialization and He initialization.

The Core Problem: Preserving Signal Variance

The central challenge in training deep networks is ensuring that information, in the form of both activations (during the forward pass) and gradients (during the backward pass), can flow effectively through all layers.

A conceptual illustration of the vanishing and exploding gradient problems. In a deep network, gradients computed via backpropagation can either shrink exponentially towards zero (left), starving early layers of updates, or grow exponentially (right), leading to unstable updates.

Let's consider two naive initialization schemes:

Zero Initialization: If we initialize all weights to zero, every neuron in a given layer will compute the exact same output and receive the exact same gradient during backpropagation. The neurons will remain identical throughout training, completely negating the benefit of having a deep, wide network. This is known as the symmetry problem.
Large Random Initialization: If we initialize weights with large random values, the weighted sum of inputs ( $z$ ) can become very large. For activation functions like tanh or sigmoid, this pushes the activations into the flat, saturated regions of the curve. In these regions, the derivative is nearly zero, causing the gradients to vanish and learning to stall.

The ideal initialization strategy should break symmetry without causing activations or gradients to either vanish or explode. The key insight, developed by researchers like Glorot, Bengio, and He, is to treat this as a problem of variance control. We want to initialize the weights such that the variance of the outputs of each layer remains close to the variance of its inputs.

A First-Principles Analysis of Variance

Let's analyze how variance propagates through a single linear layer. Consider the pre-activation $z$ for a single neuron in a layer, which is the weighted sum of $n_{in}$ inputs from the previous layer:

z = i = 1 \sum n_{in} w_{i} x_{i}

Here, $x_{i}$ are the activations from the previous layer and $w_{i}$ are the weights connecting them to our neuron.

To derive a condition for stable variance, we'll make a few reasonable assumptions for the start of training:

The weights $w_{i}$ and inputs $x_{i}$ are independent and identically distributed (i.i.d.).
Both weights and inputs have a mean of zero, i.e., $E [w_{i}] = 0$ and $E [x_{i}] = 0$ .

Our goal is to ensure the variance of the output, $Var (z)$ , is equal to the variance of the input, $Var (x)$ .

Now, I have a question for you. Using the basic properties of variance, namely $Var (A + B) = Var (A) + Var (B)$ for independent variables, and $Var (c A) = c^{2} Var (A)$ for a constant $c$ , how can you express $Var (z)$ in terms of $n_{in}$ , $Var (w)$ , and $Var (x)$ ?

Think about the variance of the product of two independent, zero-mean random variables. It can be shown that $Var (w_{i} x_{i}) = Var (w_{i}) Var (x_{i})$ .

Let's work through the derivation:

The variance of the sum is the sum of the variances (due to independence): $Var (z) = Var (i = 1 \sum n_{in} w_{i} x_{i}) = i = 1 \sum n_{in} Var (w_{i} x_{i})$
Using the property for the variance of a product of independent, zero-mean variables: $Var (z) = i = 1 \sum n_{in} Var (w_{i}) Var (x_{i})$
Since the weights and inputs are i.i.d., we can replace $Var (w_{i})$ with $Var (w)$ and $Var (x_{i})$ with $Var (x)$ : $Var (z) = n_{in} \cdot Var (w) \cdot Var (x)$

This equation is the key. It tells us that the output variance is scaled by the number of input connections ( $n_{in}$ ) and the variance of the weights.

To preserve the variance (i.e., make $Var (z) = Var (x)$ ), we must satisfy the condition:

n_{in} \cdot Var (w) = 1 ⟹ Var (w) = \frac{1}{n _{in}}

This is our first major result: to keep the signal variance constant during the forward pass, the variance of the weights should be inversely proportional to the number of input neurons (the "fan-in").

This core idea is explained very clearly in the following video from Andrew Ng.

Weight Initialization in a Deep Network (C2W1L11)

This video, "Weight Initialization in a Deep Network," offers a concise and intuitive explanation for this principle.

Watch from the beginning to the single-neuron example. This will solidify your intuition for why the variance of the weights should be scaled by 1/n.

Xavier/Glorot Initialization: The Two-Way Compromise

The derivation above only considered the forward pass. However, training also involves a backward pass where gradients are propagated from the output layer back to the input layer. Using a similar analysis, to keep the variance of the gradients stable during backpropagation, one can derive a second condition:

Var (w) = \frac{1}{n _{out}}

where $n_{out}$ is the number of output neurons (the "fan-out").

We now have two conflicting requirements unless the fan-in and fan-out are equal ( $n_{in} = n_{out}$ ), which is not always the case. In their 2010 paper, Xavier Glorot and Yoshua Bengio proposed a simple and effective compromise: take the harmonic mean of the two constraints. This leads to the Xavier/Glorot initialization formula:

Var (w) = \frac{2}{n _{in} + n _{out}}

This approach balances the requirements of both the forward and backward passes, aiming to keep variance stable in both directions.

This derivation assumes that the activation function is linear around the origin, making it well-suited for tanh and sigmoid activations.

The following resource provides an excellent, in-depth derivation and discussion of this principle.

Weight Initialization: Xavier, He & Variance Preservation for Deep ...

This article from mbrenndoerfer.com provides a rigorous mathematical breakdown of weight initialization.

First, read the section Variance Analysis. This will walk you through the derivation we just did, solidifying your understanding of how fan-in affects output variance. Next, read the section on Xavier/Glorot Initialization. Focus on how it arrives at the compromise formula by considering both the forward and backward pass requirements.

The image below, taken from the original Glorot and Bengio paper, vividly demonstrates the effect of their proposed initialization. The top row shows a 5-layer network initialized with a standard normal distribution. Activations quickly saturate to -1 and 1, and gradients vanish in the early layers. The bottom row shows the same network with Xavier initialization: both activations and gradients maintain a healthy distribution across all layers.

Comparison of network dynamics with standard initialization (top) and Xavier initialization (bottom). Left plots show the evolution of activations over training epochs. Right plots show histograms of back-propagated gradients for each layer. With Xavier initialization, activations and gradients remain well-distributed across all layers, preventing vanishing/saturation issues.

He Initialization: Adapting for ReLU

Xavier initialization was a major breakthrough, but it was designed for activations like tanh that are symmetric and zero-centered. With the rise of the Rectified Linear Unit (ReLU), a new problem emerged.

Let's consider the ReLU function, $ReLU (z) = max (0, z)$ . Here is my second question for you: if the input $z$ to a ReLU unit is from a symmetric distribution centered at zero (like a Gaussian), what happens to the variance of the output activation?

The ReLU function clamps all negative values to zero. For a symmetric input distribution, this means roughly half of the inputs become zero. This act of "killing" half the distribution effectively halves the variance.

This causes a problem for Xavier initialization. With ReLU, the activation variance would shrink by a factor of 2 at every layer, leading to vanishing signals in deep networks. To compensate for this, Kaiming He et al. (2015) proposed a simple adjustment. We go back to our forward pass condition:

Var (activation) = Var (x)

But now, we account for the variance reduction from ReLU:

Var (activation) = Var (ReLU (z)) \approx \frac{1}{2} Var (z) = \frac{1}{2} n_{in} \cdot Var (w) \cdot Var (x)

To satisfy our goal, we now need:

\frac{1}{2} n_{in} \cdot Var (w) = 1 ⟹ Var (w) = \frac{2}{n _{in}}

This is He initialization. It doubles the output variance of the weights to counteract the halving effect of ReLU, ensuring the signal variance remains stable. The He paper found that for deep ReLU networks, satisfying the forward pass condition was most critical, so the formula typically only uses fan-in.

The following video and reading will complete your understanding of both Xavier and He initialization.

L11.6 Xavier Glorot and Kaiming He Initialization

This video by Sebastian Raschka provides a clear, practical explanation of both methods.

Watch the segment on Xavier initialization for a great summary of its mechanism and the concept of fan-in. Then, watch the section on He initialization, which explains precisely why ReLU requires a different approach and how the formula is adjusted.

Weight Initialization: Xavier, He & Variance Preservation for Deep ...

We'll return to the same article to study the derivation for ReLU.

Please read the section titled He Initialization. Pay close attention to the argument about "variance halving" and how it directly leads to the modified formula.

Conclusion

In this lesson, we moved from observing a problem—unstable training in deep networks—to a solution derived from first principles. By analyzing how signal variance propagates, we have uncovered the motivation behind modern weight initialization schemes.

Key Takeaways:

The Goal: Proper weight initialization aims to maintain the variance of activations and gradients as they flow through the network, preventing them from vanishing or exploding.
The Principle: The variance of a layer's weights should be inversely proportional to the number of its connections.
Xavier/Glorot Initialization:
- Formula: $Var (w) = \frac{2}{n _{in} + n _{out}}$ .
- Use Case: Designed for symmetric, zero-centered activation functions like tanh and sigmoid. It balances the needs of both the forward and backward passes.
He Initialization:
- Formula: $Var (w) = \frac{2}{n _{in}}$ .
- Use Case: Designed specifically for ReLU and its variants. It compensates for the halving of variance caused by ReLU setting negative inputs to zero.

Preview of the Next Lesson:

We have now assembled a powerful toolkit for training deep networks: a solid understanding of backpropagation, advanced optimizers like Adam, regularization techniques like Dropout, and principled weight initialization. However, one crucial hyperparameter remains: the learning rate. A static learning rate can be suboptimal, starting too large and ending too small. In our next lesson, we will explore methods for dynamically adjusting the learning rate during training using learning rate schedules, a simple yet highly effective technique for improving convergence.