Estimating FLOPs and Parameters for GPT-2 and ResNet-50

Estimating Model Compute

The total compute for a deep learning model is typically measured in FLOPs. A single forward pass through a model performs a specific number of FLOPs, which we can estimate by summing the contributions of each layer.

A key point to remember is the relationship between the forward and backward passes during training. The backward pass, which calculates gradients via backpropagation, is computationally more intensive than the forward pass.

As the reading suggests, a common and effective rule of thumb is that FLOPs_backward ≈ 2 * FLOPs_forward. Therefore, a full training step (one forward pass + one backward pass) requires approximately 3 * FLOPs_forward. For the remainder of this lesson and the next, we will focus on calculating FLOPs_forward, as this is the fundamental unit of work that determines inference latency and is the basis for analyzing training throughput.

The number of parameters determines the model's size in memory, while the FLOPs count determines the amount of raw computation. For many common layers, these two quantities are related. A multiply-accumulate (MAC) operation, which forms the core of linear algebra routines, involves one multiplication and one addition. It is standard practice to count this as 2 FLOPs. Consequently, for a layer with $N$ parameters (weights), the forward pass often involves approximately $2\times N$ FLOPs.

Let's now look at the specific formulas for the layers that dominate our case study models.

Case Study 1: Deconstructing ResNet-50

ResNet-50 is a classic convolutional neural network for image classification. Its architecture is built from repeating "bottleneck" residual blocks, stacked in stages. Let's examine its structure.

The architecture is composed of five stages (conv1 through conv5). After an initial convolution and max pooling layer, the core of the network consists of four stages (conv2 to conv5) made of bottleneck residual blocks. A single bottleneck block consists of three convolutional layers:

1. A 1x1 convolution to reduce the number of channels.
2. A 3x3 convolution that performs the main spatial feature extraction.
3. A 1x1 convolution to restore the number of channels.

Let's perform a sample calculation for a single bottleneck block in the conv3 stage, as shown in the "ResNet-50 vs ResNeXt-50 Architecture" table. This stage processes a feature map with an input shape of $28\times 28$. The blocks in this stage transform $512$ input channels to $512$ output channels.

The block structure is [1x1, 128] -> [3x3, 128] -> [1x1, 512]. Let's apply the formulas from our reading.

1. First 1x1 Convolution (Reduction):

• Input: $H=28,W=28,C_{in}=512$
• Layer: $K=1,D=128$ (output channels)
• Output feature map size remains $28\times 28$.
• Parameters: $D\cdot K\cdot K\cdot C_{in}=128\cdot 1\cdot 1\cdot 512=65,536$
• FLOPs: $2\cdot K^2\cdot C_{in}\cdot D\cdot H^{\prime }\cdot W^{\prime }=2\cdot 1^2\cdot 512\cdot 128\cdot 28\cdot 28\approx 102.8\text{ MFLOPs}$

2. Main 3x3 Convolution:

• Input: $H=28,W=28,C_{in}=128$
• Layer: $K=3,D=128$
• Output feature map size remains $28\times 28$ (due to padding).
• Parameters: $D\cdot K\cdot K\cdot C_{in}=128\cdot 3\cdot 3\cdot 128=147,456$
• FLOPs: $2\cdot K^2\cdot C_{in}\cdot D\cdot H^{\prime }\cdot W^{\prime }=2\cdot 3^2\cdot 128\cdot 128\cdot 28\cdot 28\approx 231.2\text{ MFLOPs}$

3. Second 1x1 Convolution (Expansion):

• Input: $H=28,W=28,C_{in}=128$
• Layer: $K=1,D=512$
• Output feature map size remains $28\times 28$.
• Parameters: $D\cdot K\cdot K\cdot C_{in}=512\cdot 1\cdot 1\cdot 128=65,536$
• FLOPs: $2\cdot K^2\cdot C_{in}\cdot D\cdot H^{\prime }\cdot W^{\prime }=2\cdot 1^2\cdot 128\cdot 512\cdot 28\cdot 28\approx 102.8\text{ MFLOPs}$

Summing these up for a single block in conv3:

• Total Parameters: $65.5\text{k}+147.5\text{k}+65.5\text{k}\approx 278.5\text{k}$
• Total FLOPs: $102.8+231.2+102.8\approx 436.8\text{ MFLOPs}$

This process would be repeated for all blocks in all stages and summed. As you can imagine, this is tedious, but it's exactly what profiling tools do automatically. The "ResNet-50 vs ResNeXt-50 Architecture" table confirms the final result: a complete forward pass for a standard $224\times 224$ input image requires approximately 4.1 billion FLOPs and the model has 25.5 million parameters.

Case Study 2: Deconstructing GPT-2 Small

Now let's turn to our other case study, the GPT-2 small Transformer. Transformers are built differently from CNNs. Instead of convolutions, their primary components are Multi-Head Self-Attention (MHA) and position-wise Feed-Forward Networks (FFN), which are essentially large fully-connected layers.

The key hyperparameters for GPT-2 small are:

• Hidden dimension (n_embd): 768
• Number of layers (n_layer): 12
• Number of attention heads (n_head): 12
• Vocabulary size (vocab_size): 50257
• Context length (n_ctx): 1024

Calculating the exact FLOPs for a Transformer can be complex due to the attention mechanism's quadratic dependence on sequence length. However, a powerful and widely used rule of thumb simplifies this for autoregressive inference.

As the video explains, the inference cost for a Transformer is dominated by its matrix multiplications. For a single token generation step, the number of FLOPs can be approximated as:

where $N_{\text{params}}$ is the number of non-embedding parameters in the model. Let's verify this for GPT-2 small.

• Total Parameters: ~124 Million
• Embedding Parameters: vocab_size * n_embd = $50257\times 768\approx 38.6\text{ Million}$
• Non-Embedding Parameters: $124\text{M}-38.6\text{M}\approx 85.4\text{ Million}$

Applying the rule of thumb:

• FLOPs per generated token: $2\times 85.4\times 10^6\approx \text{171 MFLOPs}$.

This calculation gives us the cost to generate one new token, assuming the states for previous tokens are already computed and cached (a concept we'll revisit).

To process an entire prompt of sequence length $L$, the cost is roughly $L$ times this amount:

• FLOPs for a full prompt (L=1024): $1024\times 171\times 10^6\approx \text{175 GFLOPs}$.

This is a stark contrast to ResNet-50's ~4 GFLOPs. It highlights the immense computational scale of large language models, even a "small" one like GPT-2. The cost is dominated by two main components within each of the 12 decoder layers:

1. Feed-Forward Network (FFN/MLP): Consists of two large linear layers, expanding the 768-dimension hidden state to $4\times 768=3072$ and then projecting it back down. The parameter count for this block alone is $2\times (768\times 3072)\approx 4.7\text{M}$ parameters, leading to about $9.4\text{ MFLOPs}$ per token, per layer.
2. Multi-Head Attention (MHA): Involves the Q, K, V, and output projection matrices. These are four linear layers, each mapping $768\to 768$, contributing about $4\times (768\times 768)\approx 2.36\text{M}$ parameters and $4.7\text{ MFLOPs}$ per token, per layer.

Summing these gives a rough total of $12\times (9.4+4.7)\approx 169\text{ MFLOPs}$ per token, which aligns remarkably well with our $2\times N_{\text{params}}$ heuristic.

Conclusion

In this lesson, we've transitioned from analyzing hardware capabilities to quantifying model requirements. We've developed a methodology to dissect complex architectures like ResNet-50 and GPT-2 small, estimating their computational demands from first principles and with powerful heuristics.

Key Takeaways:

• A model's computational cost is measured in parameters (memory size) and FLOPs (workload).
• For training, a full step (forward + backward pass) costs roughly 3 times the FLOPs of a single forward pass.
• ResNet-50: A standard forward pass costs ~4.1 GFLOPs, with compute dominated by 3x3 convolutions on high-resolution feature maps. It has ~25.5M parameters.
• GPT-2 small: The inference cost is best approximated as $2\times N_{\text{params}}$ per token, which for GPT-2 small is ~171 MFLOPs. The total cost for a full 1024-token context is a much larger ~175 GFLOPs.
• The compute cost of Transformers is overwhelmingly concentrated in the large matrix multiplications within their FFN and MHA blocks.

We now have the two critical pieces of our puzzle: the hardware's supply of computation (from the last lesson) and the model's demand for it. In our next lesson, we will finally bring these together using the Roofline Model, allowing us to predict whether the key layers in ResNet-50 and GPT-2 will be bottlenecked by memory bandwidth or by raw compute power.