Dendrograms: Defining Style Segments

From Hierarchy to Partitions: Cutting the Dendrogram

The dendrogram is not the final result, but rather a map of all possible results. To get a specific number of clusters, we need to "cut" this map. Imagine drawing a horizontal line across the dendrogram. The number of clusters is simply the number of vertical lines your horizontal line intersects.

• A cut at a low distance (close to the bottom) will intersect many vertical lines, creating a large number of small, highly specific clusters.
• A cut at a high distance (close to the top) will intersect few vertical lines, yielding a small number of large, more general clusters.

The image below illustrates this principle. A horizontal cut at a distance of 1.5 creates four distinct clusters, while a higher cut at 2.5 results in only two.

The key question, then, is not how to cut, but where to cut.

Method 1: The "Largest Jump" Heuristic

The most common method for choosing a cut point is based on analyzing the vertical lines of the dendrogram. The y-axis represents the distance (or dissimilarity) at which clusters were merged. A long vertical line indicates that two very dissimilar clusters were combined. This is often a "forced" or unnatural merge.

The core idea is to find the largest jump in distance between successive merges. You should cut the dendrogram just below this largest jump. This implies you are stopping the clustering process right before it's forced to merge two groups that are not very similar to each other.

The following video provides an excellent walkthrough of this concept.

To formalize this, the following reading explains the same principle with a clear diagram.

Method 2: Stability and Interpretability

While the "largest jump" heuristic is a strong starting point, a more robust justification considers two additional factors: stability and interpretability.

Stability: A good clustering solution should be stable. This means that a small change in the cutting point shouldn't drastically alter the outcome. If moving your cut-off line slightly creates a cascade of new, unstable clusters, that's a sign you are in a messy part of the dendrogram. You want to cut in a region where the cluster solution remains consistent.

The following video discusses this idea of finding a stable, robust solution.

Interpretability: This is where your domain expertise as a lingerie designer becomes paramount. The statistical algorithm provides a suggestion, but the ultimate test is whether the resulting clusters are meaningful to you.

Do the clusters represent coherent style families? For example, does one cluster group together all your 'delicate lace bralettes' and 'silk chemises', while another groups the 'padded push-ups' and 'structured corsets'? If a cluster contains a seemingly random mix of items (e.g., a sporty bikini top, a vintage garter belt, and a minimalist bralette), it's not a useful segment, even if the algorithm suggests it.

This process of naming and validating clusters is a crucial part of the analysis. A research paper on fashion trends highlights this exact step.

A Lingerie Styling Example

Let's return to the first dendrogram and imagine it represents a collection of your lingerie pieces.

Suppose after inspecting the items, you find the clusters correspond to:

• Orange Cluster: 'Minimalist & Everyday' styles (e.g., simple cotton bralettes, seamless briefs).
• Green Cluster: 'Structured Boudoir' styles (e.g., underwire balconettes, corsets).
• Red Cluster: 'Delicate & Romantic' styles (e.g., lace chemises, silk robes).

Now, let's justify a number of clusters:

1. Analyze the Structure: The 'Structured Boudoir' (green) and 'Delicate & Romantic' (red) clusters merge at a distance of about 1.2. The 'Minimalist' (orange) cluster joins this larger group at a much higher distance of about 2.2.
2. Apply the Largest Jump Rule: The vertical distance between the merge at ~1.2 and the final merge at ~2.2 is quite large. This is the "largest jump" in the dendrogram. Cutting just below this jump (e.g., at a distance of 2.0) leaves us with two clusters: {Minimalist} and {Boudoir & Romantic}. This is a very strong candidate for the optimal number of clusters.
3. Consider Other Options: What if we cut lower, at a distance of 1.5 (the blue line)? This gives us four clusters. Looking at the dendrogram, this cut splits both the green and red clusters into two sub-clusters each. This might be useful. For example, the 'Delicate & Romantic' cluster might split into 'Sheer Lace' and 'Opaque Silk' sub-groups.
4. Justify the Final Choice: Your decision depends on your goal.
   • If you need broad categories for initial mood boarding, a 2-cluster solution ('Minimalist' vs. 'Ornate') might be perfect.
   • If you are curating specific looks for the photoshoot and need more granular distinctions, the 4-cluster solution ('Minimalist', 'Structured-A', 'Structured-B', 'Romantic-A', 'Romantic-B') might be more practical, allowing you to select one piece from each distinct sub-family.

Your justification would be: "Based on the largest jump in the dendrogram, a 2-cluster solution is mathematically optimal. However, for the purpose of creating diverse photoshoot looks, a 4-cluster solution provides more granular and actionable style families, separating key sub-themes within the 'Structured' and 'Romantic' categories."

Numerical Exercise

You have performed hierarchical clustering on 10 lingerie pieces. The algorithm produces a series of merges, with the distance (height) of each merge recorded. The final five merge heights are as follows:

1. Calculate the jump in height for the last three merges (i.e., the increase in height from Merge 7 to 8, and from Merge 8 to 9).
2. Based on the "largest jump" heuristic, what is the most likely optimal number of clusters?
3. Provide a brief justification for your choice.

Conclusion

In this lesson, you have learned how to move from a complex dendrogram to a practical, justified set of style segments. This is a critical step where statistical output meets expert judgment.

Key Takeaways:

• A specific number of clusters is obtained by "cutting" the dendrogram with a horizontal line.
• The primary heuristic for choosing where to cut is the "largest jump" method, which suggests cutting just before two very dissimilar clusters are merged.
• A strong justification also considers the stability of the cluster solution and, most importantly, the interpretability of the resulting clusters from a domain perspective.
• The final choice is a strategic decision that should be tailored to the specific goal of your analysis—in this case, creating meaningful style families for a lingerie photoshoot.

With this lesson, we conclude our module on unsupervised learning. You have successfully used PCA to identify underlying style dimensions and hierarchical clustering to group your lingerie pieces into meaningful families.

In our next module, we will pivot from discovering structure to making predictions. We will enter the world of supervised learning, starting with one of the most fundamental models: Multiple Linear Regression. You will learn how to build a model that predicts a lingerie piece's appeal score based on its specific design attributes, moving from the question "What groups exist?" to "How do features like lace percentage or strap width influence appeal?"