Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 2

Calculating Router Probabilities from Token Representations and Learned Weights

Recommended resources

Welcome back. In the previous lesson, you traced a token through attention, the MoE router, selected experts, and the block’s residual connection. The router was still a black box: it produced one score per expert, but we did not calculate those scores.

This lesson opens that box. You will use a token representation and learned router weights to calculate logits, then apply softmax to obtain a probability distribution over experts. The example uses deliberately small numbers so you can check every dot product and normalization by hand. Allow about 40 minutes, including the short video, reading, and numerical walkthrough.


What the router computes

Let be one token’s vector entering the MoE sublayer. In the pre-normalization block from the previous lesson, this is : it already reflects the attention sublayer’s contextual computation. With available experts, a simple bias-free router has a learned matrix . Treating as a row vector, its scores are

Each column of supplies the weights for one expert’s score. The result has entries, called router logits. A logit is an unconstrained score, not yet a probability: it can be negative, and the logits do not need to sum to one. A router with a bias would instead compute ; we will use the bias-free version for the calculation.

The diagram isolates these operations within a single MoE layer. Its denotes scores produced using router weights , while denotes the values after softmax. The highlighted FFN illustrates a subsequent expert choice; calculating comes before that choice.

A token representation \(X\) enters a router with weights \(W\); its scores \(H(x)\) are converted by softmax into expert probabilities \(G(x)\). The highlighted FFNN depicts a selected expert after routing.

A Visual Guide to Mixture of Experts (MoE) in LLMs

Watch “A Visual Guide to Mixture of Experts (MoE) in LLMs” by Maarten Grootendorst for a visual account of the matrix multiplication and softmax.

Watch the router calculation. Follow the distinction between the router’s output H(x) and the probabilities G(x). The subsequent expert choice is context for the calculation, not yet our subject.

For a longer explanation, read the linear-router and softmax portions of the following resource. Its diagrams and prose complement the explicit arithmetic below.

Gating Networks: Router Architecture in Mixture of Experts - Interactive

Read “Gating Networks: Router Architecture in Mixture of Experts” from mbrenndoerfer.com to connect the router’s learned projection with the softmax distribution.

In “The Linear Router,” begin where the text says the standard router uses a single linear layer without bias. Read the linear-router explanation through the end of that subsection; focus on why the projection produces one score per expert. Then, in “Routing Score Computation,” read the “Softmax Gating” subsection from its opening discussion of raw logits through its explanation of the denominator: the softmax explanation. Keep the two stages—scores and normalized values—separate.



Calculate four logits by hand

Suppose one token has three features and the MoE layer has four experts, indexed through . Use

The matrix has three rows because the token has three features, and four columns because the router must score four experts. Multiplying means taking the dot product of with each column:

Thus the router’s logit vector is

Expert has the highest score for this token. Notice, though, that the score of for expert is perfectly valid. It expresses a lower relative preference; it is not a negative probability. Also, these weights are learned router parameters, not a direct measurement of how accurate each expert would be on the token.

There is no separate router decision for each feature. All three features contribute to each expert’s score, and the four scores are calculated for this one token.



Turn scores into probabilities

Softmax makes the scores positive and normalizes them across all four experts:

Substituting our logits gives the denominator

Dividing each term by yields:

Expert Logit Softmax probability

The rounded probabilities sum to . Every entry is positive, including the one corresponding to a negative logit. Softmax alone has not made the computation sparse: it has assigned a relative weight to every expert. Selecting which experts actually run is a separate operation.

Softmax preserves the ranking of the logits, but its exponential also gives score differences a useful interpretation. Here, expert ’s logit exceeds expert ’s by , so their probability ratio is

The shared denominator cancels. These probabilities describe the router’s normalized preferences, not necessarily calibrated odds that an expert will produce the best answer. Nor does a probability of mean a deterministic top-expert router randomly chooses expert on roughly 64% of identical passes. Under deterministic selection, the highest-scoring expert wins each time for the same scores.



A stable calculation and a token-wise check

In code, softmax is normally evaluated with a numerically stable form. Subtract the largest logit from every logit before exponentiating:

For our example, , so the exponentials are approximately , , , and . Dividing by their sum gives the same probabilities as the table. Subtracting the same value from every score does not change any probability, but avoids needlessly large exponentials when logits have large magnitudes.

Here is a compact PyTorch check using the same matrix orientation as our hand calculation:

import torch

x = torch.tensor([[1.0, 2.0, -1.0]])  # shape (1, 3)
W = torch.tensor([
    [0.0, 1.0, 1.0, 0.0],
    [1.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 1.0, 1.0],
])                                     # shape (3, 4)

logits = x @ W                         # shape (1, 4)
probs = torch.softmax(logits, dim=-1)

print(logits)                           # tensor([[ 2.,  1.,  0., -1.]])
print(probs)                            # approximately [[.6439, .2369, .0871, .0321]]

The final dimension is the expert dimension, so dim=-1 normalizes the four expert scores for each token independently. If a second token has a different representation, it gets its own row of four logits and its own probability distribution, using the same learned .

One implementation detail will matter later: torch.nn.Linear(d, E, bias=False) stores its weights with shape , the transpose of the displayed here. It performs the equivalent projection; only the storage convention differs.



Takeaway

For each token, a linear router combines its features with learned weights to produce logits. Softmax converts those scores into positive probabilities that sum to one. The calculation is token-wise, and probabilities are not the same thing as selected experts: at this point, every expert still has a nonzero softmax value.

The next lesson uses these logits and probabilities to perform top- selection. It will also distinguish keeping the selected values from renormalizing them after the other experts are excluded.