Generic Programming with Templates

1. The Problem of Duplication and the Template Solution

Imagine you need a function to find the maximum of two values. For integers, it's simple. But what about floats, doubles, or custom data types? Without templates, you'd be forced to write a separate, overloaded function for each type.

int max(int a, int b) { return (a > b) ? a : b; }
float max(float a, float b) { return (a > b) ? a : b; }
double max(double a, double b) { return (a > b) ? a : b; }
// ... and so on for every type.

This approach is tedious and error-prone. A bug fix in one function must be manually propagated to all others. C++ templates solve this by allowing you to write a single blueprint for the function, from which the compiler can generate the specific versions it needs on demand.

The following article provides a clear introduction to this core motivation and the basic syntax.

The process described in the article, template instantiation, is the key concept. The template itself is not code that gets compiled into the final binary. It's a set of instructions for the compiler to generate code. When the compiler sees a call like myMax(3, 7), it deduces that T is int and generates a specific int myMax(int, int) function. This happens at compile time, with no runtime overhead.

2. Class Templates: Building Generic Data Structures

The same principle extends from functions to classes, allowing us to create generic containers and data structures. You are already familiar with std::vector<T>, std::unique_ptr<T>, and the SafeArray<T> you built. These are all class templates.

The power of class templates lies in their ability to abstract away the container logic from the type of data being contained. For our goal of building an inference engine, this is paramount. We need a Tensor class that can hold floats for standard models, int8_t for quantized models, or even doubles for high-precision scientific computing, all without changing the Tensor's implementation.

Dan Saks gives an excellent lecture on the fundamentals of both function and class templates. His explanation of class templates using a Rational number example clearly shows how to parameterize a class by type.

To bring this closer to our domain, consider a real-world templated Tensor class. The following article describes the motivation and implementation of a header-only tensor library that uses templates extensively.

Notice the use of template <typename T, std::size_t... DIMS>. This is a more advanced technique called variadic templates. It allows the Tensor class to be parameterized by not just the data type T, but also a variable number of non-type arguments representing the dimensions of the tensor. This creates a highly expressive, compile-time defined shape, e.g., Tensor<float, 64, 64, 3>. We will explore non-type and variadic templates in more detail as we build our own engine.

3. Practicalities: The Separation Model and Performance

A common point of confusion when first using templates is how they interact with C++'s compilation model. Ordinarily, you place function declarations in a header file (.h) and definitions in a source file (.cpp). If you try this with templates, you will typically encounter linker errors.

Why? Because a template is not a function; it's a blueprint. When main.cpp includes a header with only a template declaration, the compiler doesn't have the definition needed to instantiate the code. The source file containing the definition might be compiled separately, but without any calls to the template, it won't instantiate anything either. The result is that the linker can't find the required function.

For this reason, the vast majority of template code is written entirely in header files. The following video explains this problem and demonstrates how explicit template instantiation can be used to manage this, a technique that can be vital for controlling compile times in large projects.

The Performance Payoff

While managing template code can seem complex, the performance benefits are enormous. Because template instantiation happens at compile time, the compiler has full knowledge of the types involved. This enables a level of optimization, particularly inlining, that is impossible with runtime polymorphism (i.e., virtual functions) or function pointers.

Passing a templated function object (like a C++ lambda) can be as fast as writing the code manually inside the loop, whereas passing an std::function incurs significant overhead from type erasure and potential dynamic memory allocation.

The same video provides a fantastic, benchmarked example that demonstrates this performance difference. This is a critical insight for building high-performance systems.

4. Project: Building a Generic, Type-Safe Tensor Stub

Let's put these concepts into practice. You will create the first version of our project's Tensor class. This exercise will combine your knowledge of class design and memory management from previous lessons with today's topic of templates.

Your Task:

1. Create a new header file, Tensor.h.
2. Inside, define a class template named Tensor. It should be templated on the data type it will hold.

   template <typename T>
   class Tensor {
       // ...
   };
   

3. Member Variables: The class should store:
   • The shape of the tensor. A std::vector<size_t> m_shape; is a good choice for this.
   • The tensor's data. To build on our last lesson, use a std::unique_ptr<T[]> m_data; to manage the dynamically allocated memory. This automatically gives your class Rule of Zero compliance.
4. Constructor: Implement a constructor that accepts a std::vector<size_t> for the shape.
   • Inside the constructor, copy the passed shape to m_shape.
   • Calculate the total number of elements by multiplying the dimensions in the shape.
   • Allocate the memory for m_data using std::make_unique<T[]>(total_elements).
5. Accessor Method: Implement a method to access data. For now, a simple data() method that returns the raw pointer is sufficient.

   T* data() { return m_data.get(); }
   const T* data() const { return m_data.get(); }
   

6. Test in main.cpp:
   • Instantiate your Tensor for two different types, for example, float and int8_t.
   • Create a Tensor<float> with a shape like {1, 3, 224, 224} (a typical image tensor shape).
   • Create a Tensor<int8_t> with a different shape, e.g., {1, 1000}.
   • Verify that your code compiles and runs without memory leaks. You can fill the tensors with some data and print it out to be sure.

This project will serve as the foundation of the tensor library we will build throughout this course. By making it a class template from the very beginning, you ensure that all future components we build upon it will be generic and reusable.

Conclusion

In this lesson, you've explored C++ templates, a powerful feature for writing generic and high-performance code. You now understand how they solve the problem of code duplication and enable the creation of type-agnostic data structures and algorithms.

Key Takeaways:

• Templates are Blueprints: They are not compiled directly but are used by the compiler to instantiate type-specific code at compile time.
• Generic Programming: Templates allow you to write functions (template <typename T> void func(T arg)) and classes (template <typename T> class MyClass) that work with any data type that supports the required operations.
• Compile-Time Mechanism: Instantiation is a compile-time process, meaning templates introduce no runtime overhead. In fact, they often improve performance by enabling aggressive compiler optimizations like inlining.
• The Separation Model: Template definitions are typically placed in header files to be available for instantiation in any translation unit that needs them.
• Relevance to AI/ML: Templates are essential for building the core components of an inference engine, particularly for creating a generic Tensor class that can handle various data types (float, int8_t, etc.).

In our next lesson, we will complete our tour of modern C++ fundamentals by examining how data is passed to functions. We'll cover const-references, rvalue references, and move semantics, which, when combined with templates, allow us to write generic functions that are exceptionally efficient, avoiding unnecessary copies of large data structures.