Outlier Detection with IQR and Boxplots

The IQR Method: A Rule for Identifying Outliers

In the previous lesson, we learned that a modified boxplot displays outliers as individual points beyond the "whiskers." The rule that determines what constitutes an outlier is based on the Interquartile Range (IQR), which measures the spread of the middle 50% of your data.

An outlier is formally defined as any data point that falls outside of a specific range, often called the "fences." Here is the procedure:

1. Calculate Quartiles and IQR: First, determine the first quartile (Q1), the third quartile (Q3), and the interquartile range ($IQR=Q3-Q1$).
2. Establish the Fences: Calculate the lower and upper fences which serve as the boundaries for our data.
   • Lower Fence = $Q1-1.5\times IQR$
   • Upper Fence = $Q3+1.5\times IQR$
3. Identify Outliers: Any data point that is less than the lower fence or greater than the upper fence is flagged as a potential outlier.

The diagram below provides a clear visual representation of this rule in the context of a boxplot.

A Practical Example

Let's walk through the process with a simple dataset. Imagine we have the following data points:
[6, 2, 1, 5, 4, 3, 50]

It's immediately obvious that 50 is an unusual value compared to the others, but let's see if our formal rule confirms this intuition.

1. Order the data: [1, 2, 3, 4, 5, 6, 50]
2. Calculate Quartiles and IQR:
   • The median (Q2) is the 4th value: 4.
   • Q1 is the median of the lower half [1, 2, 3], which is 2.
   • Q3 is the median of the upper half [5, 6, 50], which is 6.
   • The IQR is $Q3-Q1=6-2=4$.
3. Establish the Fences:
   • Lower Fence = $Q1-1.5\times IQR=2-1.5\times 4=2-6=-4$.
   • Upper Fence = $Q3+1.5\times IQR=6+1.5\times 4=6+6=12$.
4. Identify Outliers:
   • Our range for "normal" data is [-4, 12].
   • Comparing our data points to this range, we find that 50 is greater than the upper fence of 12. Therefore, 50 is identified as an outlier.

To see this process in action, the following video provides a concise demonstration of calculating the fences and identifying an outlier in a modified boxplot.

Your background in computer science makes it natural to think of this as an algorithm. To see how this procedure translates directly into code, you can review the following article from GeeksforGeeks. It walks through the same steps using Python's NumPy library.

The Rationale: Why 1.5?

Now for the question that stimulates a deeper understanding: why the number 1.5? Why not 1 or 2? The choice of 1.5 is a convention established by John Tukey, the creator of the boxplot, but it is a well-reasoned one. It strikes a balance between being too sensitive and not sensitive enough, and its justification is rooted in the properties of the normal distribution.

For data that is approximately normally distributed (i.e., follows a "bell curve"), we know that:

• Approximately 99.7% of the data falls within 3 standard deviations ($\sigma$) of the mean ($\mu$). Data points beyond $\pm 3\sigma$ are extremely rare and are often considered outliers.
• For a perfect normal distribution, the quartiles have a fixed relationship with the standard deviation:
  • Q1 ≈ $\mu -0.675\sigma$
  • Q3 ≈ $\mu +0.675\sigma$
• This means the Interquartile Range is $IQR=Q3-Q1\approx (\mu +0.675\sigma )-(\mu -0.675\sigma )=1.35\sigma$.

Now, let's substitute this relationship into our fence formulas:

Upper Fence:
$Q3+1.5\times IQR\approx (\mu +0.675\sigma )+1.5\times (1.35\sigma )=\mu +0.675\sigma +2.025\sigma =\mu +2.7\sigma$

Lower Fence:
$Q1-1.5\times IQR\approx (\mu -0.675\sigma )-1.5\times (1.35\sigma )=\mu -0.675\sigma -2.025\sigma =\mu -2.7\sigma$

So, the $1.5\times IQR$ rule creates fences at approximately $\pm 2.7\sigma$ from the mean for normally distributed data. This is quite close to the $\pm 3\sigma$ threshold often used to identify extreme values. The value 1.5 serves as a robust and practical heuristic that effectively flags points that are "unlikely" under the assumption of normality.

The following article provides a superb walk-through of this mathematical derivation, exploring what would happen if we used different multipliers.

Conclusion

In this lesson, we moved from constructing boxplots to using them as a powerful analytical tool for outlier detection. You now have both the practical procedure and the theoretical justification for one of statistics' most common heuristics.

Key Takeaways:

• The 1.5 x IQR rule provides a standardized method for flagging potential outliers in a dataset.
• The procedure involves calculating lower and upper fences ($Q1-1.5\times IQR$ and $Q3+1.5\times IQR$) and identifying any points that lie beyond them.
• The choice of 1.5 as a multiplier is a reasoned convention based on the properties of the normal distribution, creating outlier boundaries at approximately $\pm 2.7$ standard deviations from the mean.
• Boxplots visually implement this rule by representing outliers as individual points, making them easy to spot.

In our journey through exploratory data analysis, we have now examined frequency distributions with histograms and five-number summaries with boxplots. In our next lesson, we will explore a more fluid way to visualize distributions by learning about Kernel Density Estimation (KDE), which can be thought of as a smoothed, continuous version of a histogram.