Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 10

Weighted Expert Output Combination with Indexed Accumulation

Recommended resources

Hello. In the previous lesson, you grouped token assignments by expert and kept their original token positions, top- slots, and gate weights aligned. That metadata now has a second job: once the experts produce outputs in expert-grouped order, it tells you where each weighted result belongs. By the end of this lesson, you will have a PyTorch combination function that returns one output vector per token in the original order.


What combination must compute

Let be the expert output for dispatched row , its saved original token ID, and its saved gate weight. The MoE output for token is

There is one summand per selected-expert assignment. With top-2 routing, two different dispatched rows normally contribute to the same output row. This is why combination is accumulation, not simply putting each expert result into an output position.

The top- slot does not appear explicitly in this sum because dispatch already used it to attach the correct weight to each row. It remains valuable metadata for checking that attachment: d.weights[a] should equal the original gate_weight[d.token_ids[a], d.slot_ids[a]]. Whether those weights are full-softmax probabilities or have been renormalized over the selected experts, combination uses the weights the router supplied; it must not normalize them again.

Watch the expert-processing portion of Mixture of Experts: Explained & Implemented from NeuralNine. Its loop finds token and slot indices directly rather than using your saved Dispatch object, but the multiplication and indexed accumulation implement the same operation.

Mixture of Experts: Explained & Implemented

NeuralNine’s “Mixture of Experts: Explained & Implemented” shows expert outputs being paired with their router weights and added to token-indexed output rows.

Watch the combination loop. Focus on why the weight lookup needs both a token index and a selected-expert slot, and why multiple contributions are added at a token index. In our implementation, dispatch has already performed that lookup and saved the aligned weights.



Indexed addition is the reverse operation

Your dispatch code used x.index_select(0, d.token_ids) to make one input row per assignment. The return trip has a different shape: several assignment rows may share the same token ID, so they must contribute to the same output row. PyTorch’s index_add_ expresses that directly.

torch.Tensor.index_add_

Read the PyTorch reference for the indexed accumulation operation used below.

In the “torch.Tensor.index_add_” entry, read the operation contract, beginning with the description of adding source into indexed rows and ending with the dimension requirements. Then read the 3-D illustration, the CUDA note beneath it, and the example under “Parameters,” from x = torch.ones(5, 3) through the output of the call with alpha=-1. Pay particular attention to dim=0: each source row is added to the destination row named by the corresponding index.

For our two-dimensional tensors, dim=0 means “index output rows.” The source will contain weighted expert outputs of shape ; the index will contain original token IDs; and the destination will have shape . Repeated IDs are expected.

Avoid replacing this with output[d.token_ids] += contributions. Advanced indexing with repeated indices does not reliably accumulate every contribution into a repeated destination. index_add_ is the operation whose purpose is to do so.



Implement the combiner

Use the Dispatch type and dispatch function from the previous lesson. The contract for expert_out is important: row must be the result of running the expert assigned to d.values[a], without changing the order of dispatched rows.

import torch


def combine(
    expert_out: torch.Tensor,  # [T * k, d], still in dispatched order
    d: Dispatch,
    num_tokens: int,
) -> torch.Tensor:
    if expert_out.ndim != 2 or expert_out.shape != d.values.shape:
        raise ValueError("Expected one expert output per dispatched row")
    if d.token_ids.numel() != expert_out.shape[0]:
        raise ValueError("Expected one token ID per expert output")
    if d.weights.numel() != expert_out.shape[0]:
        raise ValueError("Expected one gate weight per expert output")
    if d.token_ids.device != expert_out.device:
        raise ValueError("Token IDs and expert outputs must share a device")
    if d.weights.device != expert_out.device:
        raise ValueError("Gate weights and expert outputs must share a device")

    weighted = expert_out * d.weights.unsqueeze(-1)  # [T * k, d]
    output = expert_out.new_zeros((num_tokens, expert_out.shape[1]))
    output.index_add_(0, d.token_ids, weighted)
    return output

unsqueeze(-1) gives each scalar gate weight a feature dimension, so it multiplies every feature of its expert output. new_zeros starts an output tensor on the expert outputs’ device and with their dtype. The underscore in index_add_ denotes an in-place update; here it updates a freshly created output tensor, not the original token inputs.

To connect this to actual expert modules, process each expert’s contiguous slice and retain slice order:

# experts[e] is the module for expert e.
parts = []
for e, expert in enumerate(experts):
    start, end = d.offsets[e:e + 2].tolist()
    if start < end:
        parts.append(expert(d.values[start:end]))

expert_out = torch.cat(parts, dim=0)
output = combine(expert_out, d, num_tokens=x.shape[0])

An expert with an empty slice contributes no rows. Concatenating the nonempty results in expert-ID order preserves the dispatched row order. Gate weighting happens after expert evaluation, not while constructing the expert inputs.



Follow three tokens back home

Reuse the previous lesson’s top-2 routing example. To make the arithmetic visible, suppose experts 0, 1, and 2 multiply every input feature by , , and , respectively. These stand in for expert computations; the combiner does not depend on what functions the experts implement.

x = torch.tensor([
    [10.0, 0.0],
    [20.0, 0.0],
    [30.0, 0.0],
])

expert_idx = torch.tensor([
    [2, 0],
    [1, 2],
    [2, 1],
], dtype=torch.long)

gate_weight = torch.tensor([
    [0.7, 0.3],
    [0.6, 0.4],
    [0.8, 0.2],
])

d = dispatch(x, expert_idx, gate_weight, num_experts=3)

expert_scale = torch.tensor([1.1, 2.0, 3.0])
expert_out = d.values * expert_scale[d.expert_ids].unsqueeze(-1)
output = combine(expert_out, d, num_tokens=x.shape[0])

print(output)
# tensor([[24.3000,  0.0000],
#         [48.0000,  0.0000],
#         [84.0000,  0.0000]])

The dispatched rows are grouped by expert, but the result is ordered by original token ID. For token 0, expert 0 returns with weight , while expert 2 returns with weight . Its first output feature is therefore . Token 1 receives ; token 2 receives .

Notice that d.token_ids contains repeated destinations: [0, 1, 2, 0, 1, 2]. The first and fourth dispatched rows both add to output row 0. Neither needs to be moved into token-major order before the index_add_ call.

A compact check catches a mistaken destination or weight alignment in this example:

expected = torch.tensor([
    [24.3, 0.0],
    [48.0, 0.0],
    [84.0, 0.0],
])
torch.testing.assert_close(output, expected)

If the MoE layer began with a sequence tensor of shape , output is still flattened; after combination, reshape it to using the original and . Do not reshape the expert-grouped rows as though they were in sequence order.



The completed boundary

Dispatch creates one row for each selected assignment and records its origin. Expert execution transforms those rows without changing their order. Combination multiplies each result by its aligned gate weight and uses indexed addition to gather repeated assignments into one row per original token. That is the complete dropless output path for this sparse expert sublayer.

The next lesson will check the sparse layer against a reference that evaluates every expert and masks unselected contributions, including checks of outputs and gradients. That comparison is a stronger test of the whole route–dispatch–combine implementation than output shape alone.