Detecting Outliers with IQR and Z-score

What are Outliers and Why Do They Matter?

Before we dive into detection methods, it's important to understand what an outlier is and the problems it can cause. An outlier is an observation that lies an abnormal distance from other values in a random sample from a population.

As you just read, outliers can pull the mean in their direction and inflate the variance and standard deviation. This makes methods that rely on these measures (like linear regression) highly sensitive to them. The two detection techniques we will cover approach this problem from different statistical assumptions.

Let's begin with the method tied to the normal distribution: the Z-score.

1. Z-Score Method: A Parametric Approach

The Z-score method measures how many standard deviations a data point is from the mean. It's a parametric method because it assumes the data is approximately normally distributed.

The formula for the Z-score is:

where $X$ is the data point, $\mu$ is the mean of the distribution, and $\sigma$ is the standard deviation.

For a normal distribution, we know that:

• ~68% of the data falls within 1 standard deviation of the mean.
• ~95% of the data falls within 2 standard deviations.
• ~99.7% of the data falls within 3 standard deviations.

A common rule of thumb is to consider any data point with a Z-score greater than 3 or less than -3 as an outlier. These points are rare and lie far from the bulk of the data.

The codebasics channel has a very clear tutorial on implementing this with Python and pandas.

Limitations of the Z-Score Method

The Z-score method has one significant drawback: the mean and standard deviation are themselves sensitive to outliers. A very extreme outlier can inflate the standard deviation and pull the mean towards it, making it harder to detect other, less extreme outliers. For this reason, it's most effective when you have a dataset that is already close to a normal distribution and you don't expect many extreme outliers.

2. Interquartile Range (IQR) Method: A Robust Approach

The IQR method is a more robust alternative that does not assume a specific data distribution. It defines outliers based on the data's quartiles.

• First Quartile (Q1): The 25th percentile of the data.
• Third Quartile (Q3): The 75th percentile of the data.
• Interquartile Range (IQR): The distance between Q3 and Q1, i.e., $IQR=Q3-Q1$. This range contains the middle 50% of the data.

Outliers are identified using "fences" calculated from the IQR.

• Lower Fence: $Q1-1.5\times IQR$
• Upper Fence: $Q3+1.5\times IQR$

Any data point that falls below the lower fence or above the upper fence is considered an outlier.

Because this method relies on quartiles and the median (which is Q2), it is not influenced by extreme values, making it "robust."

Now, let's see how to implement this in Python. Krish Naik's tutorial provides a concise walkthrough.

Z-Score vs. IQR: A Quick Comparison

Practical Application

Let's apply both methods to a sample dataset to see them in action.

import pandas as pd
import numpy as np




# Sample data with some clear outliers (1, 150)
data = [25, 28, 29, 30, 32, 33, 33, 34, 35, 36, 38, 40, 1, 150]
df = pd.DataFrame(data, columns=['values'])




# --- Z-Score Method ---
mean = df['values'].mean()
std = df['values'].std()
z_score_threshold = 3

df['z_score'] = (df['values'] - mean) / std
z_score_outliers = df[np.abs(df['z_score']) > z_score_threshold]

print("--- Z-Score Method ---")
print(f"Mean: {mean:.2f}, Std Dev: {std:.2f}")
print("Outliers detected:")
print(z_score_outliers)
print("-" * 25)




# --- IQR Method ---
Q1 = df['values'].quantile(0.25)
Q3 = df['values'].quantile(0.75)
IQR = Q3 - Q1
lower_fence = Q1 - 1.5 * IQR
upper_fence = Q3 + 1.5 * IQR

iqr_outliers = df[(df['values'] < lower_fence) | (df['values'] > upper_fence)]

print("\n--- IQR Method ---")
print(f"Q1: {Q1}, Q3: {Q3}, IQR: {IQR}")
print(f"Lower Fence: {lower_fence}, Upper Fence: {upper_fence}")
print("Outliers detected:")
print(iqr_outliers)
print("-" * 25)

Running this code, you'll notice the Z-score method only flags 150 as an outlier. The extreme value of 150 inflates the standard deviation so much that the other outlier, 1, now falls within the 3-sigma limit. In contrast, the IQR method, being robust, successfully identifies both 1 and 150 as outliers. This is a powerful demonstration of why the IQR method is often preferred in practice.

Conclusion

You have now learned how to detect univariate outliers using two foundational statistical techniques. This is a vital skill in preparing data for machine learning.

Here are the key takeaways from this lesson:

• Outliers are data points that significantly differ from other observations and can distort statistical measures and model performance.
• The Z-Score method is a simple, parametric approach that identifies outliers based on their distance from the mean in units of standard deviation. It works best for normally distributed data and is sensitive to the very outliers it aims to find.
• The IQR method is a non-parametric, robust technique that identifies outliers based on their position relative to the central 50% of the data. It is generally the safer and more reliable choice, especially for skewed data.

Now that we can confidently detect outliers, the next step is to decide what to do with them. In our next lesson, we will explore common strategies for handling outliers, including removal, transformation, and capping.