Hierarchical Clustering and Dendrogram Visualization

The Toolkit: scipy and scikit-learn

While scikit-learn provides an AgglomerativeClustering class, the scipy library contains a robust and highly-regarded implementation of hierarchical clustering functions, especially for generating the linkage data and plotting dendrograms. A common and effective workflow is to use scipy for the exploratory step of creating and analyzing the dendrogram, and then use scikit-learn to apply the clustering with a chosen number of clusters.

Step 1: Data Preparation and Scaling

As with any distance-based algorithm like K-Means or HAC, the scale of your features is critical. Features with larger variances can dominate the distance calculations, unduly influencing the clustering outcome. Therefore, a standard preprocessing step is to scale the data so that each feature contributes more equally.

The scipy library offers a convenient function for this called whiten, which scales each feature to have a unit variance. The "Statistics Ninja" video provides a practical demonstration of data preparation, including the use of whiten.

Step 2: Generating the Linkage Matrix and Dendrogram

The core of scipy's hierarchical clustering functionality lies in two functions: linkage and dendrogram.

1. linkage: This function takes your dataset and performs the agglomerative clustering algorithm. It returns a "linkage matrix" that encodes the entire history of cluster merges. Its most important parameters are method (the linkage criterion, e.g., 'ward', 'complete', 'average') and metric (the distance metric, e.g., 'euclidean').
2. dendrogram: This function takes the linkage matrix and visualizes it as a dendrogram.

Let's walk through the process with a dataset containing shopping mall customer data, focusing on their annual income and spending score.

The "Definitive Guide to Hierarchical Clustering" article provides an excellent walkthrough of creating a dendrogram using a customer dataset.

Your resulting plot should look something like this, a classic dendrogram showing the full hierarchy of cluster merges.

Step 3: Interpreting the Dendrogram to Choose K

Now comes the crucial analytical step. As we discussed in the last lesson, you determine the number of clusters by finding the largest vertical distance that doesn’t cross any horizontal merge lines and then drawing a horizontal "cut" through it. The number of vertical lines this cut intersects is your optimal number of clusters, K.

This image provides a perfect illustration of this technique.

The article from stackabuse.com also has a great section explaining how to draw a line on the dendrogram to formalize this choice.

Step 4: Applying AgglomerativeClustering from scikit-learn

Once you've determined your optimal K from the dendrogram (in our case, 5), you can use scikit-learn's AgglomerativeClustering class to assign cluster labels to each data point. This is very similar to how you used the KMeans class.

The key parameters for AgglomerativeClustering are:

• n_clusters: The number of clusters you decided on.
• affinity: The distance metric (e.g., 'euclidean'). This is equivalent to metric in scipy's linkage function.
• linkage: The linkage criterion (e.g., 'ward'). This is equivalent to method in scipy.

The following guide from QuantInsti demonstrates this final step clearly using the Iris dataset. The logic is identical for our customer dataset.

Following this process for our customer dataset, we would use the following code to fit the model and then visualize the results.

Fitting the Model:

from sklearn.cluster import AgglomerativeClustering




# We chose K=5 from the dendrogram
cluster_model = AgglomerativeClustering(n_clusters=5, affinity='euclidean', linkage='ward')
cluster_labels = cluster_model.fit_predict(data_scaled) 

Visualizing the Clusters:

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 7))
plt.scatter(data_scaled[:,0], data_scaled[:,1], c=cluster_labels, cmap='rainbow')
plt.title('Clusters of customers')
plt.xlabel('Annual Income (scaled)')
plt.ylabel('Spending Score (scaled)')
plt.show()

The output is a scatter plot showing five distinct groups of customers, visually confirming the result of our hierarchical clustering analysis.

Conclusion

In this lesson, we bridged the gap between the theory of hierarchical clustering and its practical application. You've learned a standard, powerful workflow that combines the exploratory power of scipy's dendrograms with the convenience of scikit-learn's clustering models.

Key Takeaways:

• The practical workflow for HAC often involves both scipy and scikit-learn.
• scipy.cluster.hierarchy is used to generate the linkage matrix (linkage function) and visualize the cluster hierarchy (dendrogram function).
• The dendrogram is analyzed to determine the optimal number of clusters, K, typically by finding the cut that crosses the longest unbroken vertical line.
• sklearn.cluster.AgglomerativeClustering is then used to fit the model with the chosen n_clusters and retrieve the cluster assignments for each data point.
• As a distance-based method, HAC is sensitive to feature scaling, making it a crucial preprocessing step.

What's Next:

Both K-Means and Hierarchical Clustering work well for identifying relatively spherical or convex-shaped clusters. But what if your data contains clusters of arbitrary shapes, like crescents or circles within circles? In our next lesson, we will explore DBSCAN, a density-based algorithm that excels at finding such clusters and has the added benefit of not requiring you to specify the number of clusters beforehand.