Simulating App Reliability & Complaint Volume

The Dynamics of Predictive Value

Before we build a full simulation, let's run a simple "what-if" analysis to solidify the principle. In our investigation, we found a 15% prevalence of compatibility among 200 couples. But what if the dating pool were different? How would the app's PPV and NPV change if genuine compatibility were much more common, or much rarer?

The most intuitive way to see this is to adjust the counts in our 2x2 confusion matrix. Let's start with a hypothetical scenario where compatibility is common—a 50% prevalence.

The resource below walks through a very similar exercise, demonstrating how a tenfold change in the number of healthy subjects (i.e., a change in prevalence) dramatically alters the PPV and NPV of a medical test, even while sensitivity and specificity remain constant.

Let's apply this logic to our case. Recall the app's observed performance characteristics from your 200-couple investigation:

• Sensitivity: 90%
• Specificity: 80%

Now, let's run two scenarios using a hypothetical cohort of 1000 couples.

Scenario 1: High Prevalence (50%)
If prevalence is 50%, we have 500 compatible couples and 500 incompatible couples.

• True Positives (TP): $500\times 0.90=450$
• False Negatives (FN): $500\times (1-0.90)=50$
• True Negatives (TN): $500\times 0.80=400$
• False Positives (FP): $500\times (1-0.80)=100$

The Positive Predictive Value would be:

In this high-prevalence world, the app would be genuinely useful.

Scenario 2: Low Prevalence (5%)
Now, consider a tougher dating market with only 5% prevalence. We have 50 compatible couples and 950 incompatible ones.

• True Positives (TP): $50\times 0.90=45$
• False Negatives (FN): $50\times (1-0.90)=5$
• True Negatives (TN): $950\times 0.80=760$
• False Positives (FP): $950\times (1-0.80)=190$

The PPV plummets:

Here, the app is worse than useless; it's actively misleading. This hands-on example confirms the core principle: for a given test, prevalence is the master variable controlling its predictive power.

The Performance Profile: A Full Simulation

Running discrete scenarios is insightful, but to build a truly damning case, we need to see the full picture. We'll now construct a continuous profile of the app's reliability across the entire spectrum of prevalence, from 0% to 100%.

To do this, we use the formulas for PPV and NPV expressed in terms of probabilities, which you'll recognize as variations of Bayes' theorem. This allows us to plot them as continuous functions of prevalence. Given your background, the connection between these formulas and the concepts of statistical power ($1-\beta$) and significance ($\alpha$) will be clear. In our diagnostic context:

• Sensitivity = Power = $1-\beta$
• Specificity = $1-\alpha$
• False Positive Rate = $\alpha$

The resource below formalizes these relationships.

By applying these formulas for every prevalence value from 0 to 100, we can generate a complete performance graph. Imagine this as the output from a specialized piece of analysis software.

The following short video provides an excellent summary of this exact process, showing how PPV changes dramatically when prevalence shifts from 5% to 20%.

Exhibit A: The ErosEncore Reliability Profile

Running this simulation for ErosEncore's observed performance (90% sensitivity, 80% specificity) produces our key piece of evidence.

Let's analyze this exhibit to close our case:

1. The Reality (Green Line): At the observed prevalence of 15% (approximated by the 6% line's low value), the PPV curve is far below 50%. A "Compatible" result is overwhelmingly likely to be a false positive, perfectly explaining the client complaints. Our observed PPV of 44.3% sits squarely on this curve.
2. The Crossover Point: The PPV and NPV curves intersect at a prevalence of approximately 31%. This is the point where a positive test and a negative test have equal predictive power.
3. The Zone of Reliability: For the PPV to be greater than the NPV (and for a positive result to be more likely true than false, i.e., PPV > 50%), the prevalence of genuine compatibility must be greater than 20%. For the app to be truly reliable (e.g., PPV > 80%), prevalence would need to exceed 47%.
4. The Verdict: The app is only reliable in a dating market where genuine compatibility is exceptionally common. In the real-world scenario you investigated, with a modest 15% base rate, its design guarantees a poor user experience driven by a high volume of false positives.

The simulation reveals the full extent of the scandal. The app's failure is not an anomaly; it is an inherent mathematical property of its design operating in a world of realistic base rates.

Conclusion: Case Closed

Our investigation is complete. We began with a flood of complaints and a suspicious marketing campaign. We proceeded to quantify the app's true performance, revealing a shocking 44.3% PPV. We then used Bayes' theorem to formally identify the low base rate of compatibility as the primary culprit. Finally, with today's simulation, we have created a comprehensive performance profile that leaves no room for doubt. We have proven that the ErosEncore app is fundamentally flawed, designed in a way that makes it unreliable in any realistic scenario.

Key Takeaways:

• The reliability of a diagnostic or screening tool (its PPV and NPV) is critically dependent on the prevalence of the condition in the population being tested.
• By simulating performance across a range of prevalence values, we can create a "reliability profile" that shows under which conditions a test is useful.
• The ErosEncore app, with 90% sensitivity and 80% specificity, only achieves a PPV over 50% when the prevalence of genuine compatibility exceeds 20%.
• The high volume of client complaints is the direct, predictable result of the app's low PPV at the observed 15% prevalence rate. The case is closed.

Preview of the Next Case: The Studio Paradox

With the ErosEncore case filed, a new dossier lands on your desk. It concerns two rival boudoir photography studios, "Satin & Shadow" and "Velvet & Vellum." Both studios are running aggressive marketing campaigns, and strangely, both present data that seems to prove they have higher average client satisfaction scores than their competitor. It appears to be a statistical impossibility.

Your next assignment: unravel this paradox. Is one of them lying, or is there a more subtle statistical illusion at play? This will take us into the realm of confounding variables, paradoxes, and the foundations of causal inference.