LLM Architecture: Open-weight vs. Proprietary Models

The Modern "Standard": A Llama-like Baseline

Let's begin by establishing a baseline. The original 2017 Transformer was an encoder-decoder model. Today, the dominant paradigm for generative text models is the decoder-only architecture. Models from the Llama family are excellent examples of this modern standard, incorporating many of the optimizations we've studied.

A typical modern decoder-only LLM architecture, exemplified by models like Llama 3, includes:

• Pre-Normalization: Using RMSNorm before the attention and FFN layers for training stability.
• Rotary Positional Embeddings (RoPE): For robust length extrapolation.
• Grouped-Query Attention (GQA): To reduce the KV-cache size and improve inference speed.
• SwiGLU Activation: In the feed-forward network for enhanced expressiveness.

This combination represents a powerful and efficient baseline. Now, let's explore the major architectural divergences and specializations built upon this foundation.

A Tour of Modern LLM Architectures

To see how these components are combined in practice, we'll turn to a comprehensive comparison of recent LLM architectures. The following resource by Sebastian Raschka provides an excellent, in-depth analysis of the design choices behind many leading models.

You can either watch the selected segments of the video or read the corresponding sections in the article—both cover the same material. The video offers a narrated walkthrough, while the article may be faster for targeted reading.

Alternatively, you can read the corresponding sections from the article version.

Synthesizing the Key Architectural Choices

Having reviewed those models, let's distill the information into a clear comparative framework. The design of a modern LLM involves a series of strategic choices along several key axes.

1. Sparsity: Mixture-of-Experts (MoE) vs. Dense

This is arguably the most significant architectural fork in the road for large-scale models.

• Dense Models (e.g., Llama 3 8B/70B, Qwen3-dense): Every parameter is used for every token. They are generally simpler to train and fine-tune but become computationally expensive at massive scale.
• Sparse MoE Models (e.g., DeepSeek V3, Mixtral, Llama 4, GPT-4, Claude 3): Only a fraction of the model's parameters (the "experts") are activated for each token. This allows for a huge total parameter count (and thus, more stored knowledge) while keeping inference costs manageable.
  • Trade-off: MoE introduces training complexity (e.g., router stability, load balancing) but dramatically reduces inference FLOPs.

2. Attention Mechanism and Context Window

The quadratic complexity of attention is a primary engineering challenge, leading to several solutions.

• Grouped-Query Attention (GQA): The current industry standard (Llama 3, Mistral, Qwen3). It offers a great balance between the performance of Multi-Head Attention (MHA) and the extreme efficiency of Multi-Query Attention (MQA).
• Sliding Window Attention (SWA): Used in models like Gemma 3 and gpt-oss. It restricts attention to a local window, drastically reducing KV-cache size for long sequences. Often used in a hybrid approach with a few full attention layers to maintain global context.
• Multi-Head Latent Attention (MLA): A DeepSeek innovation. Instead of reducing heads (like GQA) or context (like SWA), it compresses the key and value tensors before caching, trading a small amount of extra compute for significant memory savings.

3. Model Shape: Depth vs. Width

Given a fixed parameter budget, engineers can choose to build a model that is "deep" (more layers) or "wide" (larger embedding and hidden dimensions).

• Deeper Models (e.g., Qwen3): More sequential operations, potentially leading to higher latency. However, more layers can allow for more complex transformations and feature extraction with fewer parameters.
• Wider Models (e.g., GPT-OSS, Mistral Small): Fewer layers but more parameters per layer. This structure is often more friendly to parallelization, potentially resulting in higher token-per-second throughput at the cost of higher memory usage per layer.

Summary of Architectural Choices

Here is a simplified table comparing some of the models you've studied.

Conclusion

This lesson has been a capstone for our study of LLM architecture. You've seen that modern models are not monolithic creations but are assembled from a toolbox of sophisticated components. The final design is a carefully balanced equation of trade-offs, optimized for a specific set of goals.

Key Takeaways:

• There is a "standard" modern architecture (Llama-like), but significant variations exist to optimize for specific goals.
• Mixture-of-Experts (MoE) is the key technology for scaling model size while keeping inference computationally feasible.
• Attention variants (GQA, SWA, MLA) are crucial for managing the KV-cache, enabling longer context windows and faster inference.
• The shape of a model—depth versus width—is a fundamental design choice that impacts latency, memory, and parallelization.
• Leading models like Llama, Mistral, DeepSeek, and others each represent a unique point in this complex design space, reflecting different engineering priorities.

What's Next?

You have now completed the module on Transformer architectures and optimizations. You have a solid grasp of how these models are built, from the individual sub-layers to the high-level design philosophies of state-of-the-art systems.

In our next module, "Training and Scaling Large Language Models," we will shift our focus from what the models are to how they are made. We'll begin with the very first step in the data pipeline: tokenization. You'll learn about methods like Byte-Pair Encoding (BPE) and understand how the way we chop up text has a profound impact on a model's performance, vocabulary, and ability to handle diverse languages.