Building and Interpreting CART Decision Trees for Lingerie Selection

From Probabilities to Rules: The Decision Tree Concept

While Naive Bayes calculates probabilities, a decision tree provides a set of 'if-then' rules. Imagine a flowchart designed to classify a lingerie piece. Each question in the flowchart is a node, each answer a branch, and the final conclusion is a leaf.

This diagram illustrates a simple tree structure. Each 'Yes/No' question splits the data, leading down a path that ends in a classification. We can read this as a series of if-then rules.

For example, a path in a lingerie selection tree might look like this:

1. Is the fabric Lace? -> Yes.
2. Is the support level Low? -> Yes.
3. Is the color Red or Black? -> Yes.
4. Conclusion: Predict 'Photoshoot-Worthy'.

This structure is highly interpretable, making it a powerful tool not just for prediction, but for understanding the drivers of that prediction. The CART algorithm is one of the most common methods for automatically creating these trees from data.

The Heart of CART: Gini Impurity

To build an effective tree, the algorithm must decide on the best question to ask at each step. "Best" means the question that does the best job of separating the data into purer groups—that is, groups that are less mixed in terms of their outcome. CART uses a metric called the Gini Impurity (or Gini Index) to measure this.

Gini Impurity quantifies how "mixed" a group of items is.

• A Gini Impurity of 0 means a perfectly pure node (all items belong to a single class, e.g., all are 'Photoshoot-Worthy').
• For a two-class problem, a Gini Impurity of 0.5 is the maximum impurity (the items are a 50/50 mix of the two classes).

The goal of the CART algorithm is to find splits that reduce impurity as much as possible. The following resource provides the mathematical definition.

As you read, you saw that the algorithm works by calculating a weighted Gini score for every potential split. It then chooses the split that results in the lowest weighted Gini score, as this corresponds to the largest "Gini Gain" or reduction in impurity.

Building a Tree, Step by Step

The process of building a tree is known as recursive partitioning. It's a "greedy" algorithm because it makes the best possible split at the current step, without looking ahead to see if a different split might be better in the long run.

The process is as follows:

1. Start with the entire dataset (the root node).
2. For each feature, test all possible binary splits. For a categorical feature like Fabric = {Lace, Satin, Cotton}, this means testing splits like Fabric = Lace vs. Not Lace, Fabric = Satin vs. Not Satin, etc.
3. Calculate the weighted Gini impurity for each of these potential splits.
4. Select the single feature and split that has the lowest weighted Gini impurity. This becomes the first decision node of your tree.
5. The data is now split into two new datasets (the child nodes).
6. Repeat steps 2-5 for each child node, recursively, until a stopping condition is met (e.g., the node is pure, or the number of samples in it is too small).

The following blog post provides an excellent, detailed walkthrough of this process with a simple dataset.

Your Turn: Building a Lingerie Decision Tree

Now, let's apply this method to our lingerie selection problem. Given your statistical background, the calculations should be straightforward, allowing you to focus on the logic of the algorithm.

Scenario:
You have a dataset of 8 lingerie styles, characterized by binary features, and you want to find the best first question to ask to determine if a style is 'Photoshoot-Worthy'.

Your Task:

1. Calculate the Gini impurity for the root node (the entire dataset).
2. For each of the three features (Has Lace?, Support, Color Tone), calculate the weighted Gini impurity of a split.
3. Identify which feature provides the best split (i.e., the lowest weighted Gini impurity).
4. Draw the top level of the decision tree and interpret the rule it represents.

          [Root]
      {Yes: 5, No: 3}
        Gini = 0.469
             |
   Is Color Tone 'Dark'?
    /                  \
'Yes'                  'No' (i.e., 'Light')
  |                      |
[Node 1]               [Node 2]
{Yes: 5, No: 1}        {Yes: 0, No: 2}
Gini = 0.278           Gini = 0.0 (Pure)

Conclusion

In this lesson, you have learned to build and interpret a CART decision tree. We've seen how it moves beyond the probabilistic approach of Naive Bayes to create explicit, hierarchical rules that can capture complex feature interactions. By mastering the Gini impurity calculation, you can now manually trace the logic of the algorithm and understand exactly why it chooses one feature over another to split the data.

Key Takeaways:

• Decision trees create a flowchart of if-then rules that are highly interpretable.
• The CART algorithm builds binary trees by recursively partitioning the data.
• The Gini Impurity is a measure of node purity, and CART seeks to find splits that maximize the reduction in impurity (Gini Gain).
• The process is "greedy," making the locally optimal choice at each step to construct the tree.

So far, we've discussed Naive Bayes and CART (using the Gini index). However, Gini is not the only metric for building trees. In our next lesson, we will compare CART to another famous algorithm, C4.5, which uses a different splitting criterion called Information Gain (based on entropy). We will explore how this alternative can sometimes lead to different tree structures and decision rules from the same data.