Evaluating Lingerie Screening Performance with Imbalanced Data

From Bayes' Theorem to a Classification Framework

While Bayes' theorem provides the engine for updating beliefs, a more operational view comes from structuring the problem as a classification task. Every time you evaluate a lingerie piece, you are classifying it as either "photoshoot-worthy" (Positive) or "average" (Negative).

The results of this classification can be organized into a confusion matrix, a tool you are undoubtedly familiar with. This matrix provides the raw counts needed to calculate performance metrics.

Let's define these terms for your specific context:

• True Positive (TP): A "star" style (e.g., a unique lace bralette) that your screening process correctly identifies as photoshoot-worthy.
• False Positive (FP): An "average" style that your process incorrectly flags as photoshoot-worthy. This is a Type I error.
• True Negative (TN): An "average" style that your process correctly dismisses.
• False Negative (FN): A "star" style that your process incorrectly dismisses. This is a Type II error.

From these four outcomes, we can calculate the key rates that measure the performance of your screening process.

The Two Most Important Metrics for a Rare-Event Search

When "star" styles are rare, overall accuracy is a misleading metric because a model that rejects everything can achieve very high accuracy. Instead, we focus on more telling indicators.

1. Positive Predictive Value (PPV) or Precision:

   $$\text{PPV}=\frac{\text{TP}}{\text{TP}+\text{FP}}$$

   This answers the question: "Of all the lingerie pieces I've selected for the next round, what proportion are actually stars?" As the diagram above notes, this is your "Trust Metric." A high PPV means your selected items are very likely to be winners.

2. False Positive Rate (FPR):

   $$\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}}$$

   This answers the question: "Of all the average, non-star styles, what proportion did I mistakenly let through my filter?" This is the rate of false alarms. Note that FPR is the complement of Specificity ($\text{Specificity}=1-\text{FPR}$).

An associated "waste metric" is the False Discovery Rate (FDR), which is simply $1-\text{PPV}$. It tells you what percentage of effort in the next stage (e.g., test shoots) will be spent on styles that were incorrectly selected.

To solidify the connection between these metrics and our previous discussion on Bayes' theorem, the following resource is excellent.

As the article confirms:

• PPV is precisely the posterior probability we've been calculating: $P(\text{Star}|\text{Selected})$.
• FPR is the conditional probability of a false alarm: $P(\text{Selected}|\text{Average})$.

The Devastating Effect of Low Prevalence

The reason we focus on FPR is that when the number of actual negatives (average styles) is vast, even a tiny FPR can generate a large number of false positives, which in turn swamps the true positives and crushes your PPV.

To see this in action, I recommend watching this excellent video from Dr. Trefor Bazett. It uses a medical analogy, but the logic is identical to finding a rare "star" lingerie piece.

The core takeaway is that the number of false positives ($FP$) is a product of two things: the FPR and the total number of "average" styles.

When the pool of average styles is enormous compared to the pool of star styles, this number can easily dominate your TP count.

The relationship between prevalence, the test's error rate, and the final PPV is elegantly captured in the graph below.

Numerical Exercise: Evaluating a Stylist's First Pass

Let's apply this to a concrete scenario. You're preparing for the photoshoot and have a collection of 500 potential lingerie pieces. From experience, you estimate that only about 4% of these are true "star" material. You ask a junior stylist to do a first-pass screening.

You've evaluated this stylist's past performance and found:

• Sensitivity (True Positive Rate): They correctly identify 80% of star styles. $P(\text{Selected}|\text{Star})=0.80$.
• Specificity (True Negative Rate): They correctly reject 90% of average styles. $P(\text{Rejected}|\text{Average})=0.90$.

Your Task:

1. Based on the 500 pieces and 4% prevalence, calculate the number of "Star" and "Average" styles.
2. Using the stylist's performance metrics, calculate the expected number of True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Fill out a confusion matrix.
3. Calculate the False Positive Rate (FPR) for the stylist's screening.
4. Calculate the Positive Predictive Value (PPV) of the styles they selected.
5. What is the False Discovery Rate (FDR)? Interpret this number in terms of your project's next steps (e.g., organizing test shoots).

Solution

1. Initial Counts:

   • Total styles = 500
   • Star styles = 4% of 500 = 20
   • Average styles = 96% of 500 = 480

2. Confusion Matrix Calculations:

   • TP: The stylist identifies 80% of the 20 star styles. $0.80\times 20=\text{16}$.
   • FN: The stylist misses the other 20% of star styles. $0.20\times 20=\text{4}$.
   • Specificity is 90%, so the FPR is $1-0.90=0.10$.
   • FP: The stylist incorrectly selects 10% of the 480 average styles. $0.10\times 480=\text{48}$.
   • TN: The stylist correctly rejects the other 90% of average styles. $0.90\times 480=\text{432}$.

   Confusion Matrix:
   | | Predicted: Star | Predicted: Average | Total |
   | :--- | :---: | :---: | :---: |
   | Actual: Star | 16 (TP) | 4 (FN) | 20 |
   | Actual: Average | 48 (FP) | 432 (TN) | 480 |
   | Total | 64 | 436 | 500 |

3. False Positive Rate (FPR):

   \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} = \frac{48}{48 + 432} = \frac{48}{480} = \textbf{0.10 or 10%}

   This was given by $1-\text{Specificity}$, but we've confirmed it from the counts.

4. Positive Predictive Value (PPV):

   \text{PPV} = \frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{16}{16 + 48} = \frac{16}{64} = \textbf{0.25 or 25%}

   This means that of the 64 pieces the stylist recommended, you can only trust that 25% of them are actually star quality.

5. False Discovery Rate (FDR):

   \text{FDR} = 1 - \text{PPV} = 1 - 0.25 = \textbf{0.75 or 75%}

   Interpretation: This is a crucial business metric for you. It means that 75% of the resources (time, model fees, photographer's attention) you allocate to evaluating the stylist's selected pool of 64 items will be wasted on average styles. Despite the stylist's seemingly good 80% sensitivity and 90% specificity, the low 4% prevalence of star styles results in a selection pool where 3 out of every 4 items are false positives.

Conclusion

In this lesson, we transitioned from the theoretical application of Bayes' theorem to the operational framework of a classification matrix. This allowed us to calculate and interpret the two most critical metrics for your lingerie screening process in a low-prevalence environment.

Key Takeaways:

• The Confusion Matrix (TP, FP, TN, FN) is the foundation for calculating performance in a classification context.
• Positive Predictive Value (PPV), or Precision, measures the quality of your selected candidates. It answers: "How many of the styles I picked are actually good?"
• False Positive Rate (FPR) measures the leakiness of your filter for average items. It answers: "How often do I make a false alarm on an average style?"
• In imbalanced datasets, where "star" styles are rare, even a low FPR can lead to a high number of false positives, which drastically reduces your PPV and increases your False Discovery Rate (FDR), or wasted effort.

You are now equipped to evaluate any single stage of your curation process. In our next lesson, we will explore how to make this process dynamic by performing iterative Bayesian updating. You will learn to use the posterior probability from one tryout as the prior for the next, allowing you to continually refine your belief about the best style as you gather more evidence.