Canonical vs. Non-Canonical Link Functions: Trade-offs in Selection

Canonical vs. Non-Canonical Links

As a quick recap, a link function $g(\mu )$ must be a monotonic and differentiable function. The canonical link is special because it makes the linear predictor equal to the natural parameter of the response distribution in its exponential family form ($\eta =\theta$).

Any other valid link function is called a non-canonical link.

So, why would we ever deviate from the canonical choice? While canonical links have desirable statistical properties that can simplify estimation, there is no guarantee that they provide the best description of the relationship in your specific dataset. The primary goal of modeling is to find a function that accurately reflects the underlying data-generating process.

To see this distinction in action, let's watch a brief segment that introduces the idea of non-canonical links.

The key takeaway is that we can, and sometimes should, test different links to see which provides a better fit for our data.

A Case Study: Link Functions for Binary Outcomes

The choice between link functions is most prominent when modeling binary (0/1) or proportional outcomes with the binomial distribution. As you know, the goal is to model the probability of success, $\pi$, which is bounded between 0 and 1. The link function's job is to map this (0, 1) interval onto the entire real number line (-$\infty$, +$\infty$), which is the range of the linear predictor $\eta$.

Any function that produces an "S"-shaped (sigmoidal) curve can accomplish this. The canonical link for the binomial distribution is the logit link, but several non-canonical alternatives exist.

The two most widely used links for binary data are the logit and probit links. Let's explore them in detail.

Deep Dive: The Logit vs. Probit Links

Understanding the difference between logit and probit models is crucial in applied data science. The following video provides an excellent and intuitive introduction to both.

To summarize and formalize the points from the video:

The Logit Link (Canonical)

• Form: $g(\pi )=\log \left(\frac{\pi }{1-\pi }\right)=\eta$
• Underlying Distribution: It is derived from the CDF of the logistic distribution.
• Interpretation: This is its main advantage. Since $\frac{\pi }{1-\pi }$ is the odds of success, the logit link models the log-odds of the outcome as a linear function of the predictors. This means model coefficients can be exponentiated to get odds ratios, which are highly interpretable.

The Probit Link (Non-Canonical)

• Form: $g(\pi )={\Phi }^{-1}(\pi )=\eta$, where ${\Phi }^{-1}$ is the inverse CDF of the standard normal distribution.
• Underlying Distribution: It is derived from the CDF of the standard normal distribution.
• Interpretation: A probit model assumes that there is an unobserved (latent) continuous variable that follows a normal distribution. The binary outcome we observe depends on whether this latent variable crosses a certain threshold. The coefficients are on the scale of this latent variable (in standard deviations), which is less intuitive than odds ratios.

Other Notable Non-Canonical Links

While logit and probit are the most common, other links are useful in specific scenarios, particularly when the relationship is asymmetric.

• Complementary Log-Log (cloglog):

  • Form: $g(\pi )=\log (-\log (1-\pi ))$
  • Use Case: This link is asymmetric. It's appropriate when the probability of success approaches 1 quickly but approaches 0 slowly. This is common in survival analysis where the probability of an event (like equipment failure) is low for a long time and then rises sharply.

• Log-Log:

  • Form: $g(\pi )=-\log (-\log (\pi ))$
  • Use Case: This link has the opposite asymmetry of the cloglog link. The probability approaches 0 quickly but approaches 1 slowly.

The Trade-Offs: How to Choose Your Link

You now know what the main canonical and non-canonical links are. So, how do you choose? The decision involves balancing interpretability, theoretical assumptions, and model fit.

Here is a summary of the key trade-offs:

For more on the theoretical advantages of canonical links, the following resource provides a key insight.

Conclusion

In this lesson, we moved beyond the default choice of canonical links and explored the world of non-canonical alternatives. You learned that while canonical links offer mathematical convenience and desirable properties, non-canonical links provide the flexibility to better capture the true nature of your data.

Key Takeaways:

• A non-canonical link is any valid link function that is not the canonical one for a given distribution.
• For binary data, the main choice is between the canonical logit link and the non-canonical probit link.
• The logit link is generally preferred due to the high interpretability of its coefficients as odds ratios.
• The probit link assumes an underlying normally distributed latent variable and is common in fields like econometrics, but its coefficients are difficult to interpret.
• In practice, logit and probit models often yield very similar results. The choice is often driven by the need for interpretability or conventions within a specific field.
• Other non-canonical links, like complementary log-log, are useful for modeling asymmetric relationships.

Preview of the Next Lesson:

We have now covered all three components of a GLM: the Random component (distribution), the Systematic component (linear predictor), and the Link function. Now we can start using them! In the next lesson, "Inference and Model Assessment for GLMs," we will begin to explore how to evaluate how well a GLM fits the data. We'll start by defining the concept of deviance, which is the core metric for measuring goodness of fit in the GLM framework.