Differentiation Basics: Polynomial, Exponential, and Logarithmic Functions

Hello and welcome to the course! I'm excited to guide you on your journey to build Physics-Informed Neural Networks for radiotherapy dose prediction.

This first lesson marks our starting point in Module 1, "Calculus Essentials for Deep Learning." We will begin by exploring the fundamental concept of the derivative and learn how to compute it for three key types of functions: polynomials, exponentials, and logarithms. Understanding derivatives is the first crucial step in our journey. In the context of deep learning, derivatives allow us to quantify how a small change in a model's parameter affects its performance. This measurement of change is what drives the learning process, enabling a neural network to systematically improve its predictions. Let's begin.

The Idea of a Derivative: Measuring Change

At its core, differential calculus is the study of how things change. A derivative is a precise mathematical tool that measures the instantaneous rate of change of a function. As a radiation oncologist, you frequently encounter rates of change: the rate of dose delivery from a linear accelerator, or the rate of decay of a radioactive isotope. The derivative gives us a way to describe these rates not as averages over time, but at a specific moment.

Graphically, the derivative of a function at a particular point is the slope of the line tangent to the function's graph at that point. A steep slope means a rapid change, while a shallow slope means a slow change.

To build a solid foundation, let's start with the formal definition of a derivative and see how it relates to this idea of approximating a function with a straight line in a very small region.

22.3. Single Variable Calculus

This reading from the "Dive into Deep Learning" book introduces the derivative from the perspective of machine learning and provides the formal limit-based definition.

Please read the section "Differential Calculus". Focus on how the authors motivate the need for calculus by considering how to update a model's weights. Pay attention to the idea that a complex function looks like a straight line when you zoom in far enough. Read from the beginning of the section down to the formal definition of the derivative, which is presented as: \frac{df}{dx}(x) = \lim_{\epsilon \rightarrow 0}\frac{f(x+\epsilon) - f(x)}{\epsilon}. You can stop reading just before the subsection "Rules of Calculus".

While this limit definition is the theoretical bedrock, we rarely use it directly for computations. Instead, we rely on a set of powerful rules that have been derived from it. Let's explore these rules for the function types most relevant to us.

Rule 1: Derivatives of Polynomial Functions

Polynomials are functions built from variables raised to non-negative integer powers, like $f (x) = 4 x^{3} - x^{2} + 7$ . Differentiating them relies on a few straightforward rules.

The most important is the Power Rule:
For any real number $n$ , the derivative of $f (x) = x^{n}$ is $f^{'} (x) = n x^{n - 1}$ .

To handle more complex polynomials, we combine this with two other rules:

Constant Multiple Rule: The derivative of a constant times a function is the constant times the derivative of the function. $\frac{d}{d x} [c \cdot f (x)] = c \cdot \frac{d}{d x} [f (x)]$
Sum/Difference Rule: The derivative of a sum or difference of functions is the sum or difference of their derivatives. $\frac{d}{d x} [f (x) \pm g (x)] = \frac{d}{d x} [f (x)] \pm \frac{d}{d x} [g (x)]$

Let's see this in action. Suppose we want to find the derivative of $f (x) = 2 x^{3} + 5 x - 8$ .

We apply the Sum/Difference Rule to differentiate term by term: $\frac{d}{d x} (2 x^{3}) + \frac{d}{d x} (5 x) - \frac{d}{d x} (8)$
For the first term, we use the Constant Multiple and Power Rules: $\frac{d}{d x} (2 x^{3}) = 2 \cdot \frac{d}{d x} (x^{3}) = 2 \cdot (3 x^{3 - 1}) = 6 x^{2}$
For the second term, note that $x$ is $x^{1}$ : $\frac{d}{d x} (5 x) = 5 \cdot \frac{d}{d x} (x^{1}) = 5 \cdot (1 x^{1 - 1}) = 5 \cdot (1 x^{0}) = 5 \cdot 1 = 5$
For the last term, note that a constant $c$ can be written as $c x^{0}$ . Its derivative is 0. So, $\frac{d}{d x} (8) = 0$ .

Putting it all together, the derivative of $f (x) = 2 x^{3} + 5 x - 8$ is $f^{'} (x) = 6 x^{2} + 5$ .

The following video provides a quick review of the power rule with several clear examples.

Derivatives of Exponential Functions & Logarithmic Differentiation Calculus lnx, e^2x, x^x, x^sinx

This video by The Organic Chemistry Tutor demonstrates the power rule in practice.

Watch the segment from this timestamp for a clear, step-by-step walkthrough of applying the power rule to differentiate polynomials.

Rule 2: Derivatives of Exponential Functions

Exponential functions are central to many physical processes, including those in your field. For example, radioactive decay is described by the function $N (t) = N_{0} e^{- λ t}$ , where $N (t)$ is the number of radioactive nuclei at time $t$ , $N_{0}$ is the initial number, and $λ$ is the decay constant. The rate of decay is simply the derivative, $\frac{d N}{d t}$ .

This graph illustrates exponential decay, a process whose rate of change is described by a derivative. The function shown is $y = 0.5e^{-1.5t}$, representing a quantity decreasing over time.

Exponential functions possess a remarkable property: their rate of change is proportional to their current value. This means the more you have of something, the faster it grows (or decays). Let's build the intuition for this before stating the rule.

What's so special about Euler's number e? | Chapter 5, Essence of calculus

This video from the "Essence of Calculus" series by 3Blue1Brown provides a beautiful, intuitive explanation of why the derivative of an exponential function is what it is.

Please watch the following segments: An intuitive approach: This part uses an analogy to show that the derivative of 2^t is proportional to 2^t. Introducing the number e: Here, the video introduces Euler's number, e, as the unique base for which the proportionality constant is exactly 1. The general rule: This segment explains how this relates back to any base, revealing the origin of the mystery constant.

As the video explains, the number $e \approx 2.718$ is the unique base where the derivative of the exponential function is exactly equal to the function itself.

The Rule for $e^{x}$ :

\frac{d}{d x} e^{x} = e^{x}

This is a profound and elegant result. The slope of the graph of $f (x) = e^{x}$ at any point $x$ is equal to the value of the function $e^{x}$ at that same point.

The graph of $f(x)=e^x$. Notice that at $x=0$, the value is $f(0)=e^0=1$ and the slope of the tangent line is also 1. At $x=1$, the value is $f(1) \approx 2.72$ and the slope is also $\approx 2.72$.

The Rule for a general base $a^{x}$ :
For any positive base $a$ , the derivative is proportional to the original function:

\frac{d}{d x} a^{x} = (ln a) \cdot a^{x}

Here, $ln a$ is the natural logarithm of the base $a$ , which is the "proportionality constant" we saw in the video.

Now, let's watch some practical examples of how to apply these rules.

Derivatives of Exponential Functions & Logarithmic Differentiation Calculus lnx, e^2x, x^x, x^sinx

This video provides clear, worked-out examples for differentiating various exponential functions.

Watch the segment on the derivative of e^u from this timestamp. Notice how it handles functions like e^{2x}. This involves a concept called the "chain rule," which we will study in detail in our next lesson. For now, just observe the pattern: differentiate the "outer" function (e^u) and then multiply by the derivative of the "inner" function (u). Next, watch the segment on the derivative of a^u from this timestamp. This covers the general case for other bases, like 2^x or 7^{4x-x^2}.

Rule 3: Derivatives of Logarithmic Functions

Logarithmic functions are the inverses of exponential functions. The most important one for calculus and deep learning is the natural logarithm, denoted $ln (x)$ , which has base $e$ .

The Rule for $ln (x)$ :
The derivative of the natural logarithm is surprisingly simple:

\frac{d}{d x} ln (x) = \frac{1}{x}

The Rule for a general base $lo g_{a} (x)$ :
Similar to exponentials, there's a slightly more general rule for logarithms with a different base $a$ :

\frac{d}{d x} lo g_{a} (x) = \frac{1}{x ln a}

Let's see these rules applied in practice.

Derivatives of Exponential Functions & Logarithmic Differentiation Calculus lnx, e^2x, x^x, x^sinx

The same video also provides excellent examples for logarithmic functions.

Watch the segment covering the derivative of \ln(u) from this timestamp. Again, this uses the chain rule pattern for expressions like \ln(x^2+1). The rule is \frac{u'}{u}, where u is the inner function. Finally, watch the short segment on the derivative of \log_a(u) from this timestamp to see the general case.

Summary and Next Steps

In this lesson, we've laid the first stone in our mathematical foundation for deep learning. We established the derivative as a measure of instantaneous change and learned the fundamental rules for computing it.

Here are the key takeaways:

The derivative of a function at a point gives the slope of the tangent line, representing the function's instantaneous rate of change.
We use a set of rules to compute derivatives, avoiding the limit definition in practice.
Power Rule for polynomials: $\frac{d}{d x} x^{n} = n x^{n - 1}$ .
Exponential Rule: The derivative of $e^{x}$ is itself, $e^{x}$ . For a general base $a$ , the derivative of $a^{x}$ is $(ln a) a^{x}$ . This property makes exponentials ubiquitous in modeling processes where the rate of change is proportional to the current amount.
Logarithm Rule: The derivative of $ln (x)$ is $\frac{1}{x}$ .

In our next lesson, we will formally introduce and master the Chain Rule. You've already had a preview of it in today's examples (e.g., differentiating $e^{2 x}$ ). The chain rule is the cornerstone of calculus for deep learning, as it is the mechanism that allows us to calculate derivatives through the nested layers of a complex neural network—a process you'll later come to know as backpropagation.

Recommended resources

The Idea of a Derivative: Measuring Change

Rule 1: Derivatives of Polynomial Functions

Rule 2: Derivatives of Exponential Functions

Rule 3: Derivatives of Logarithmic Functions

Summary and Next Steps