Missing Data Imputation for Lingerie Datasets

The Three Mechanisms of Missing Data

In any dataset, values can be missing for various reasons. The reason why a value is missing determines the nature of the problem and its solution. Statisticians classify these reasons into three categories. Given your background, you are likely familiar with these concepts, but a focused review in the context of our project will set the stage for our practical work.

Please read the following sections from the online guide "A Guide on Data Analysis" to formally define these categories.

To make these abstract definitions concrete, let's consider your lingerie photoshoot context. Imagine you have a dataset with ratings for different lingerie pieces on a scale of 1-10, provided by a model, a photographer, and a stylist.

• Missing Completely at Random (MCAR): A rating is missing for a reason that is entirely unrelated to the lingerie itself, the person rating it, or the rating it would have received.

  • Example: While transcribing handwritten rating sheets into a spreadsheet, you accidentally skip a cell. The missing data point is a random error. Its absence tells you nothing about the lingerie's appeal.

• Missing at Random (MAR): A rating is missing for a reason that is related to other information you have recorded in your dataset, but not the missing rating itself.

  • Example: You notice that one of your stylists, who specializes in high-fashion, avant-garde looks, frequently "forgets" to rate simple, classic cotton bralettes. However, she always rates the more elaborate lace corsets. The probability of a rating being missing depends on an observed variable—the "Style Complexity" or "Fabric" of the lingerie piece, and perhaps the "Stylist ID." If you account for these variables, the missingness is random.

• Missing Not at Random (MNAR): A rating is missing precisely because of the value it would have had. The reason for the absence is directly linked to the unobserved rating.

  • Example: A model tries on a balconette bra with underwire that she finds extremely uncomfortable. When asked to rate it, she declines, saying "I'd rather not say." The missing rating is a direct result of the low comfort score she would have given. The value of the missing data is the cause of its missingness.

Distinguishing between these types is crucial. Treating an MNAR case as if it were MCAR could lead you to, for example, overestimate the average appeal of underwire bras by ignoring the systematically missing low scores.

Diagnosing the Mechanism and Choosing a Strategy

Your task, as the analyst, is to play detective. By combining statistical tests with your knowledge of the data collection process, you can make an educated assessment of the missingness mechanism. This diagnosis then points directly to a set of appropriate strategies.

The following flowchart provides an excellent mental model for this decision-making process.

Let's break down the path from diagnosis to strategy.

1. Is the missingness related to other observed variables?

You can investigate this by creating a "missingness indicator" variable (1 if the rating is missing, 0 if present). Then, you can compare the distributions of your other variables across these two groups. For instance:

• Does the average "Lace Percentage" differ for items with missing ratings versus those with complete ratings?
• Is the proportion of missing ratings significantly higher for one photographer than for others?

A formal test for this is Little's MCAR test, which evaluates the null hypothesis that the data are MCAR. A significant p-value suggests the data are likely MAR or MNAR. While this test can help rule out MCAR, it cannot prove it, nor can it distinguish between MAR and MNAR. Your domain knowledge is indispensable here.

If the answer is No, you are likely in the MCAR world.

• Strategies:
  • Listwise Deletion: If only a very small fraction of your data is missing (e.g., <5%), the simplest approach is to remove the observations with missing values. This is statistically "safe" under MCAR, though it reduces your sample size and statistical power.
  • Mean/Median/Mode Imputation: Replacing the missing value with the average rating. This is generally a poor choice as it artificially reduces the variance of your data and distorts relationships between variables. It should only be considered as a last resort for tiny amounts of MCAR data.

If the answer is Yes, you move to the next question.

2. Is the missingness related to the missing value itself?

This is the hardest question to answer statistically because it involves an unobserved value. There is no direct test for MNAR. Your judgment, based on the context of data collection, is paramount. Did a model express discomfort? Did a stylist mention a piece was "uninspiring"? These qualitative notes are clues that point towards MNAR.

If the answer is No, you are in the MAR world. The missingness is explainable by other variables you have.

• Strategies: This is where more sophisticated methods are required to avoid bias.
  • Multiple Imputation by Chained Equations (MICE): This is a very powerful and robust technique. Instead of filling in one "best guess" for a missing value, MICE creates several plausible completed datasets. It uses the other variables (e.g., style, fabric, color, other experts' ratings) to predict the missing rating, adding a degree of randomness to reflect the uncertainty. You then perform your analysis (e.g., calculate the average rating for each style) on all the imputed datasets and then pool the results. This ensures your final standard errors and p-values correctly account for the uncertainty of the imputation.
  • Machine Learning-Based Imputation: Methods like k-Nearest Neighbors (k-NN) or Random Forests can also be used. For k-NN, you could find the k lingerie pieces that are most similar to the one with a missing rating (based on features like cut, fabric, etc.) and use the average of their ratings as the imputed value.

If the answer is Yes, you are in the difficult MNAR world.

• Strategies: Simple or multiple imputation will likely produce biased results because they cannot account for the fact that the missing values are systematically different.
  • Model the Missingness: This is an advanced approach that your background in econometrics and causal analysis prepares you for. It involves building a statistical model (like a Heckman selection model) that jointly estimates the value of the rating and the probability of it being missing.
  • Sensitivity Analysis: Since you have to make an assumption about the missing values, you can test a range of plausible scenarios. For example, what if all missing ratings are 1s? What if they are 3s? If your overall conclusion (e.g., "The plunge bra is our top style") remains the same across these scenarios, your result is robust.
  • Use Domain Expertise: Often the most practical approach. If a model declines to rate a piece due to discomfort, you could work with her to agree on a specific low score (e.g., a '2' for 'unwearable'). This injects external knowledge to solve the problem.

For a broader look at the various imputation methods available, from simple to complex, the following reading provides a helpful survey.

Numerical Exercise

Let's apply these concepts. Below is a small snippet of your lingerie rating data. Three experts (a Stylist, a Photographer, and a Model) are rating pieces on a scale of 1-10.

Your Task:

1. Classify the missing value for item B02 (the Photographer's rating). Is it MCAR, MAR, or MNAR? Justify your reasoning based on the information provided.
2. Classify the missing value for item C03 (the Model's rating). Is it MCAR, MAR, or MNAR? Justify your reasoning.
3. For each case, propose a primary strategy to handle the missing value and explain why it is appropriate.

Self-Correction and Reflection:

• For the photographer's missing rating (B02), is simple mean imputation a good idea? What are the immediate drawbacks even in this seemingly simple case?
• For the model's missing rating (C03), what would be the consequence of imputing the mean of the other two experts' ratings (6.5)? How would that bias your assessment of the "Wireless Bralette" style?

Conclusion

In this lesson, we established the critical first step in handling real-world data: diagnosing and treating missing values. You learned to differentiate between MCAR, MAR, and MNAR mechanisms and saw how these classifications apply directly to a lingerie rating dataset.

Key Takeaways:

• The reason data is missing is more important than the fact that it is missing.
• The three mechanisms of missingness—MCAR, MAR, and MNAR—form a diagnostic framework.
• Your strategy must match the diagnosis: simple deletion for some MCAR cases, sophisticated multiple imputation for MAR, and explicit modeling or domain expertise for the challenging MNAR cases.
• Choosing the wrong strategy can introduce significant bias, leading you to misjudge the appeal of your lingerie styles.

In our next lesson, we will continue preparing our dataset for analysis by tackling another common data quality issue: outliers. We will explore robust statistical methods to identify and handle extreme ratings that could otherwise skew your evaluation of different lingerie designs.