Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 12

Replacing a Feed-Forward Sublayer with a Sparse Expert Layer

Recommended resources

Welcome back. In the previous lesson, you checked that your sparse expert layer matches a dense, masked reference in both outputs and gradients. Now you will put that tested layer where it belongs: in place of one feed-forward sublayer of a tiny decoder-only transformer. The attention, residual connections, embeddings, and output head will stay in place. By the end, you will have a runnable model and checks that the integration preserves causality and the sparse layer’s behavior.

Allow about 40 minutes, including time to run and inspect the code. This lesson reuses the TinyMoE class from the previous lesson; run its class-definition cell before the code below.


Locate the replacement boundary

In a pre-normalization decoder block, attention and the feed-forward sublayer each produce an update that is added to a residual stream. We will keep that structure and replace only the function that produces the feed-forward update:

The MoE receives the same hidden width and returns the same hidden width as the dense feed-forward network. Internally, it routes each token to its selected experts. From the block’s perspective, its input and output still have shape , for batch size , sequence length , and hidden width .

The diagram shows a transformer block with attention and residual connections, comparing a conventional feed-forward network with an MoE replacement. Its illustrated GLU experts differ from the GELU feed-forward experts in our tiny implementation; the replacement boundary is the point to notice.

For a short architectural walkthrough, watch the indicated passage from Sparse Mixture of Experts by Neural Breakdown with AVB. It demonstrates the feed-forward replacement using a dense MoE; we will make the same architectural substitution with the sparse top- layer already tested.

Sparse Mixture of Experts - The transformer behind the most efficient LLMs (DeepSeek, Mixtral)

Watch this passage to identify which part of a decoder changes when its feed-forward sublayer becomes an MoE.

In the section beginning with the dense-MoE code demonstration, watch the replacement boundary. Focus on what stays unchanged around the feed-forward sublayer and on its unchanged input and output dimensions. Do not copy its all-experts computation into the sparse implementation.



Give the tested layer a block-shaped interface

The previous TinyMoE.forward accepts a two-dimensional tensor of shape , where is the number of tokens. The decoder block will supply . A thin adapter can flatten the batch and sequence dimensions, call the existing, tested implementation, and restore the shape:

import copy

import torch
from torch import nn
from torch.nn import functional as F


class MoEFeedForward(nn.Module):
    def __init__(self, layer: nn.Module, reference: bool = False):
        super().__init__()
        self.layer = layer
        self.reference = reference

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: [batch, sequence, hidden width]
        batch, sequence, width = x.shape
        flat = x.reshape(batch * sequence, width)
        flat_output = self.layer(flat, reference=self.reference)
        return flat_output.reshape(batch, sequence, width)

reshape changes how we view token positions for routing; it does not change their order. The output is restored in that same order. In particular, do not rebuild the flattened tensor with torch.tensor(flat): that would create a copy detached from the computation graph. The reference switch is here solely to reuse last lesson’s compute-heavy oracle during testing.

This adapter imposes no capacity limit and drops no assignments. It also adds no load-balancing loss. Those training controls come later; for now, keeping the layer unchanged makes an integration error easier to identify.



Build a tiny decoder, then change one sublayer

The following model has two pre-normalization decoder blocks. Each begins with a conventional feed-forward sublayer. Attention uses a boolean causal mask: positions above the diagonal are disallowed, so a token cannot attend to later tokens.

class TinyDecoderBlock(nn.Module):
    def __init__(self, d: int, n_heads: int, hidden: int):
        super().__init__()
        self.norm1 = nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(
            embed_dim=d,
            num_heads=n_heads,
            dropout=0.0,
            batch_first=True,
        )
        self.norm2 = nn.LayerNorm(d)
        self.ffn = nn.Sequential(
            nn.Linear(d, hidden),
            nn.GELU(),
            nn.Linear(hidden, d),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        sequence = x.shape[1]
        causal_mask = torch.ones(
            sequence, sequence, device=x.device, dtype=torch.bool
        ).triu(1)

        h = self.norm1(x)
        attention_output, _ = self.attn(
            h, h, h, attn_mask=causal_mask, need_weights=False
        )
        x = x + attention_output
        x = x + self.ffn(self.norm2(x))
        return x


class TinyDecoder(nn.Module):
    def __init__(
        self,
        vocab: int = 16,
        context: int = 8,
        d: int = 8,
        n_heads: int = 2,
        hidden: int = 16,
        n_blocks: int = 2,
    ):
        super().__init__()
        self.token_embedding = nn.Embedding(vocab, d)
        self.position_embedding = nn.Embedding(context, d)
        self.blocks = nn.ModuleList([
            TinyDecoderBlock(d, n_heads, hidden)
            for _ in range(n_blocks)
        ])
        self.final_norm = nn.LayerNorm(d)
        self.output_head = nn.Linear(d, vocab)

    def forward(self, token_ids: torch.Tensor) -> torch.Tensor:
        batch, sequence = token_ids.shape
        if sequence > self.position_embedding.num_embeddings:
            raise ValueError("Sequence exceeds the configured context length.")

        positions = torch.arange(sequence, device=token_ids.device)
        x = (
            self.token_embedding(token_ids)
            + self.position_embedding(positions)
        )  # [batch, sequence, d]

        for block in self.blocks:
            x = block(x)

        return self.output_head(self.final_norm(x))

Now make the actual substitution. TinyMoE is the class you tested in the previous lesson, with renormalized top- gates and experts that map width back to width .

torch.manual_seed(12)

model = TinyDecoder()
model.blocks[0].ffn = MoEFeedForward(
    TinyMoE(d=8, hidden=16, n_experts=4, k=2)
)

assert isinstance(model.blocks[0].ffn, MoEFeedForward)
assert isinstance(model.blocks[1].ffn, nn.Sequential)

token_ids = torch.tensor([
    [1, 4, 2, 7, 9],
    [3, 8, 6, 10, 5],
])

logits = model(token_ids)
assert logits.shape == (2, 5, 16)

Only block zero’s feed-forward sublayer was replaced. Block one remains dense, and both blocks still have attention. The router and all four experts are stored as model parameters, although each token evaluates only its selected two experts in the replaced block.

Create the optimizer after the replacement. An optimizer created beforehand would hold references to the old feed-forward parameters, not the newly installed router and experts.



Check the integration against the reference path

A successful shape assertion does not prove that the adapter preserved token order or gradients. Make an identical model whose MoE adapter calls the all-experts, masked reference from the previous lesson. Its logits and task-loss gradients should match those of the sparse model.

reference_model = copy.deepcopy(model)
reference_model.blocks[0].ffn.reference = True

sparse_logits = model(token_ids)
reference_logits = reference_model(token_ids)

torch.testing.assert_close(
    sparse_logits, reference_logits, rtol=1e-5, atol=1e-6
)


def next_token_loss(logits: torch.Tensor) -> torch.Tensor:
    # Predict each token from the preceding positions.
    predictions = logits[:, :-1, :]
    targets = token_ids[:, 1:]
    return F.cross_entropy(
        predictions.reshape(-1, logits.shape[-1]),
        targets.reshape(-1),
    )


next_token_loss(sparse_logits).backward()
next_token_loss(reference_logits).backward()

for (name, parameter), (ref_name, ref_parameter) in zip(
    model.named_parameters(),
    reference_model.named_parameters(),
):
    assert name == ref_name

    # An expert with no selected tokens may have grad=None in the sparse
    # path and an explicit zero gradient in the masked reference.
    gradient = (
        torch.zeros_like(parameter)
        if parameter.grad is None else parameter.grad
    )
    ref_gradient = (
        torch.zeros_like(ref_parameter)
        if ref_parameter.grad is None else ref_parameter.grad
    )
    torch.testing.assert_close(
        gradient, ref_gradient, rtol=1e-5, atol=1e-6
    )

print("Full-model logits and parameter gradients match the reference.")

This is a stronger check than the layer-only test: it exercises the flatten-and-restore adapter, both residual paths, attention, and the output head. If it fails, first inspect whether each flattened token returns to its original location. Then check that the sparse and reference paths use the same renormalized-gate convention.

The test does not say that the sparse model should match the original dense transformer’s logits. Replacing a learned feed-forward function with newly initialized router and expert weights changes the model’s function; matching the MoE reference is the relevant correctness criterion.



Check causality and take one training step

Routing is token-wise inside the feed-forward sublayer. It should not give an earlier position access to a later token. With dropout disabled in this tiny model, changing tokens at positions three and four should leave logits at positions zero through two unchanged:

model.eval()

changed_ids = token_ids.clone()
changed_ids[:, 3:] = torch.tensor([
    [11, 12],
    [13, 14],
])

with torch.no_grad():
    original = model(token_ids)
    changed = model(changed_ids)

torch.testing.assert_close(
    original[:, :3, :],
    changed[:, :3, :],
    rtol=1e-5,
    atol=1e-6,
)
print("Earlier logits do not depend on later tokens.")

Finally, verify that the integrated model can perform an ordinary optimization step. Re-create gradients for this step rather than reusing those from the reference comparison:

model.train()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
optimizer.zero_grad(set_to_none=True)

loss = next_token_loss(model(token_ids))
loss.backward()

router_gradient = model.blocks[0].ffn.layer.router.weight.grad
assert router_gradient is not None
assert torch.isfinite(router_gradient).all()
assert torch.isfinite(loss)

optimizer.step()
print(f"One training step completed; loss before update: {loss.item():.4f}")

With top- renormalized gates, the task loss can provide a gradient through the selected gate weights as well as through evaluated experts. An expert that receives no tokens in this small batch may have no gradient; that alone is not an integration failure. Nor does one successful step establish useful expert specialization or balanced use—it establishes that the model’s basic forward and backward paths are connected.



Key takeaways

Replacing a dense feed-forward sublayer does not require rewriting the decoder. The block still accepts and returns ; the adapter presents its tokens as to the tested sparse layer and restores their positions afterward. Full-model agreement with a masked reference checks the integration’s outputs and gradients, while the future-token test checks that the surrounding decoder remains causal.

You now have a minimal MoE decoder that takes a training step. The next module adds the controls this deliberately simple version lacks: measuring expert usage, encouraging load balance, and handling limited expert capacity.