Derivatives of Common Functions

Hello! Welcome to the first lesson in our module on calculus.

In the previous module, we concluded our study of Linear Algebra by learning how to implement key operations like finding determinants, inverses, eigenvalues, and SVD using Python's NumPy library. You've built a solid foundation for handling the vectors and matrices that are central to data science.

Now, we pivot to another essential mathematical pillar: Calculus. This module, "Calculus for Optimization," is crucial for understanding how machine learning models learn. At the heart of most ML algorithms is a process of minimizing an error or "cost" function, and calculus provides the tools to do this efficiently.

Today's lesson starts with the most fundamental concept in calculus: the derivative. Our goal is to compute derivatives of polynomial, exponential, logarithmic, and trigonometric functions. We will focus on learning the standard rules that allow us to find these derivatives quickly and accurately.

1. What is a Derivative?

Before we start memorizing rules, it's vital to have an intuitive grasp of what a derivative is. In simple terms, the derivative of a function at a specific point tells us two things:

The instantaneous rate of change of the function at that point.
The slope of the line tangent to the function's graph at that point.

A positive derivative means the function is increasing, a negative derivative means it's decreasing, and a zero derivative often signals a peak, a valley, or a flat spot. This is the key idea behind optimization in machine learning: we want to find the point where the "error" function's derivative is zero to locate its minimum.

To build this intuition, let's watch a short video that explains the concept from the ground up.

What is a Derivative? Deriving the Power Rule

This video from the Professor Dave Explains channel provides a clear, graphical introduction to the derivative, defining it as the slope of a tangent line and deriving its formal definition using limits.

Watch the first two segments of the video (from 00:00 to 05:01). Focus on understanding the connection between the slope of a tangent line, the rate of change, and the formal limit definition of the derivative.

2. Derivatives of Polynomial Functions

Polynomials are functions like $f (x) = 3 x^{2} + 2 x - 5$ . Differentiating them is one of the first skills you learn in calculus, and it relies on a few straightforward rules.

The Core Rules for Polynomials

The Constant Rule: The derivative of any constant is zero. This makes sense because a constant function (e.g., $f (x) = 5$ ) is a horizontal line, and its slope is always 0.
$\frac{d}{d x} (c) = 0$
The Power Rule: This is the most important rule for polynomials. To find the derivative of $x$ raised to a power, you bring the exponent down as a multiplier and then subtract one from the exponent.
$\frac{d}{d x} (x^{n}) = n x^{n - 1}$
For example, the derivative of $x^{3}$ is $3 x^{3 - 1} = 3 x^{2}$ .
The Constant Multiple Rule: The derivative of a constant multiplied by a function is the constant times the derivative of the function.
$\frac{d}{d x} (c \cdot f (x)) = c \cdot f^{'} (x)$
The Sum/Difference Rule: The derivative of a sum or difference of functions is the sum or difference of their derivatives.
$\frac{d}{d x} (f (x) \pm g (x)) = f^{'} (x) \pm g^{'} (x)$

The video you just started watching also gives a great derivation of the power rule. Let's continue with it.

What is a Derivative? Deriving the Power Rule

Let's continue with the same video to see how the power rule is discovered and why it's so useful.

Watch the rest of the video (from 05:01 to 09:37). See how the limit definition is used to find the derivatives of x, x², and x³, which reveals the pattern of the power rule.

Now, let's solidify this by reading a resource that lays out these rules with clear examples.

Derivatives of Polynomial Functions

This resource from Math LibreTexts provides a formal summary of the basic differentiation rules for polynomials and includes several worked-out examples.

Read the sections 'The Basic Rules' and 'The Sum, Difference, and Constant Multiple Rules'. Pay close attention to the theorem boxes that formally state the rules and work through the examples to see how they are applied.

Let's apply these rules to an example. To find the derivative of $f (x) = 4 x^{3} - 7 x^{2} + x + 9$ , we differentiate term by term:

$\frac{d}{d x} (4 x^{3}) = 4 \cdot (3 x^{3 - 1}) = 12 x^{2}$
$\frac{d}{d x} (- 7 x^{2}) = - 7 \cdot (2 x^{2 - 1}) = - 14 x$
$\frac{d}{d x} (x) = \frac{d}{d x} (x^{1}) = 1 \cdot x^{1 - 1} = 1 \cdot x^{0} = 1$
$\frac{d}{d x} (9) = 0$

Putting it all together using the Sum/Difference Rule:
$f^{'} (x) = 12 x^{2} - 14 x + 1$

Test your understanding!

Find the derivative of the function $g (x) = - 2 x^{4} + 5 x^{3} - x^{2} + 6 x - 11$ .

Show answer

We differentiate term by term:
$g^{'} (x) = \frac{d}{d x} (- 2 x^{4}) + \frac{d}{d x} (5 x^{3}) - \frac{d}{d x} (x^{2}) + \frac{d}{d x} (6 x) - \frac{d}{d x} (11)$
$g^{'} (x) = (- 2 \cdot 4 x^{3}) + (5 \cdot 3 x^{2}) - (2 x) + (6) - (0)$
$g^{'} (x) = - 8 x^{3} + 15 x^{2} - 2 x + 6$

3. Derivatives of Exponential and Logarithmic Functions

Exponential and logarithmic functions are fundamental in many areas of science and engineering, including modeling growth and decay processes in data science. Their derivatives are surprisingly simple.

Key Rules for Exponential and Logarithmic Functions

The Natural Exponential Function ( $e^{x}$ ): The function $f (x) = e^{x}$ is its own derivative. This is a unique and remarkable property.
$\frac{d}{d x} (e^{x}) = e^{x}$
General Exponential Function ( $a^{x}$ ): For an exponential function with a base $a$ other than $e$ , the derivative is the function itself multiplied by the natural logarithm of the base.
$\frac{d}{d x} (a^{x}) = a^{x} ln (a)$
The Natural Logarithm ( $ln (x)$ ): The derivative of $ln (x)$ is simply $1/ x$ .
$\frac{d}{d x} (ln (x)) = \frac{1}{x}$
General Logarithmic Function ( $lo g_{b} (x)$ ): For a logarithm with a base $b$ other than $e$ , the derivative is:
$\frac{d}{d x} (lo g_{b} (x)) = \frac{1}{x l n ( b )}$

Let's watch a video that introduces these rules and shows them in action.

Derivatives of Logarithmic and Exponential Functions

This Professor Dave Explains video clearly explains the rules for differentiating logarithmic and exponential functions, including the special case of e^x.

Watch the video from 00:44 to 07:34. The first part covers logarithmic functions, and the second covers exponential functions. Focus on memorizing the four main rules presented.

The image below provides a great summary of the rules for exponential functions.

Derivatives of Exponential Functions — This image summarizes the derivative rules for the natural exponential function $e^x$ and the general exponential function $a^x$, including their variations using the chain rule which we will cover in the next lesson.

For a more detailed explanation and further examples, you can refer to the following text.

Derivatives of Exponential and Logarithmic Functions

This page from Math LibreTexts offers a more formal derivation and discussion of the derivatives for both exponential and logarithmic functions.

Read the sections 'Derivative of the Exponential Function,' 'Derivative of the Logarithmic Function,' and 'Derivatives of General Exponential and Logarithmic Functions.' Focus on the main derivative formulas and how they are derived from the properties of e and logarithms.

Test your understanding!

Find the derivative of the function $h (x) = 3 e^{x} + 1 0^{x} - 7 ln (x)$ .

Show answer

Using the sum/difference rule and the rules for exponential and logarithmic functions:
$h^{'} (x) = \frac{d}{d x} (3 e^{x}) + \frac{d}{d x} (1 0^{x}) - \frac{d}{d x} (7 ln (x))$
$h^{'} (x) = 3 \cdot \frac{d}{d x} (e^{x}) + \frac{d}{d x} (1 0^{x}) - 7 \cdot \frac{d}{d x} (ln (x))$
$h^{'} (x) = 3 e^{x} + 1 0^{x} ln (10) - \frac{7}{x}$

4. Derivatives of Trigonometric Functions

Finally, we need to know the derivatives of the basic trigonometric functions. These appear frequently in engineering and physics, and sometimes in advanced data science models.

The three most important rules are:

Sine:
$\frac{d}{d x} (sin (x)) = cos (x)$
Cosine:
$\frac{d}{d x} (cos (x)) = - sin (x)$
Tangent:
$\frac{d}{d x} (tan (x)) = sec^{2} (x)$

Let's try a simple problem combining all the function types we've learned today.
Find the derivative of $f (x) = x^{5} + cos (x) - 2 e^{x}$ .

Applying our rules term-by-term:
$f^{'} (x) = \frac{d}{d x} (x^{5}) + \frac{d}{d x} (cos (x)) - \frac{d}{d x} (2 e^{x})$
$f^{'} (x) = 5 x^{4} - sin (x) - 2 e^{x}$

Conclusion

Great job! You've taken your first and most important step into calculus. You are now equipped with the fundamental rules for finding derivatives, a skill that forms the bedrock of optimization algorithms in machine learning.

Key Takeaways:

The derivative represents the instantaneous rate of change or the slope of the tangent line.
Power Rule for polynomials: $\frac{d}{d x} (x^{n}) = n x^{n - 1}$ .
Exponential Rules: $\frac{d}{d x} (e^{x}) = e^{x}$ and $\frac{d}{d x} (a^{x}) = a^{x} ln (a)$ .
Logarithmic Rules: $\frac{d}{d x} (ln (x)) = \frac{1}{x}$ and $\frac{d}{d x} (lo g_{b} (x)) = \frac{1}{x l n ( b )}$ .
Trigonometric Rules: $\frac{d}{d x} (sin (x)) = cos (x)$ and $\frac{d}{d x} (cos (x)) = - sin (x)$ .
The Sum, Difference, and Constant Multiple rules allow us to combine these basic rules to differentiate more complex functions.

Preview of the Next Lesson:

In this lesson, we learned the derivatives of individual functions. But what about functions that are multiplied, divided, or nested inside one another (like $sin (x^{2})$ or $e^{3 x}$ )? In the next lesson, we will learn the three essential rules for handling these cases: the product rule, the quotient rule, and the chain rule. These will dramatically expand your ability to differentiate almost any function you might encounter.

Recommended resources