Mitigating Bias in Outcome Reporting

Introduction

In our previous lesson, we established a robust method for creating a single, composite Triadic_Satisfaction index using a second-order confirmatory factor model. This gives us a powerful, theoretically-grounded primary outcome for our experiment. However, the integrity of this index—and indeed our entire study—hinges on the quality of the raw data we collect. If participants' reports of their satisfaction are colored by a desire to present themselves or their relationship in a positive light, our sophisticated models will simply be analyzing a distorted reality.

This brings us to this lesson's critical objective: to design protocols to mitigate social desirability and observer biases during outcome reporting and assessment. We will tackle this from two angles. First, we will focus on minimizing social desirability bias in the self-report data collected from the triad members, which forms the core of our measurement strategy. Second, we will consider how to manage observer bias if we were to supplement our self-reports with assessments from trained, external raters for validation purposes. Your background in research methodology will be valuable here, as we explore both psychological and statistical techniques for ensuring data fidelity.

1. Mitigating Social desirability Bias with Ecological Momentary Assessment (EMA)

The most pervasive threat to self-reported relationship data is social desirability bias: the tendency for respondents to answer questions in a manner that will be viewed favorably by others. In the context of our vee triad, this could manifest as underreporting feelings of jealousy or insecurity, and overreporting feelings of happiness or compersion, to align with an idealized version of a successful polyamorous relationship.

Our primary strategy to combat this is not to rely on single, retrospective questionnaires, but to use Ecological Momentary Assessment (EMA). EMA involves collecting data in real-time, in the participants' natural environment, through repeated assessments. This approach fundamentally disrupts the cognitive processes that facilitate curated, socially desirable responses.

Ecological Momentary Assessment to Measure Emotion, Cognition, and Behavior in Daily Life | DIGIT-MI

This presentation by Dr. Amy Cochran provides an excellent overview of EMA, its core features, and its advantages over traditional retrospective reports.

Please watch the following segments: First, watch the introduction from the opening quote to understand the fundamental limitations of global, retrospective reports that EMA is designed to overcome. Next, focus on the section from "key features of EMA". The presenter outlines the four defining tenants of EMA, which are central to its ability to reduce bias.

As the video explains, EMA's strength comes from capturing experiences "as life is lived." By assessing current states ("right now") rather than retrospective summaries ("over the past week"), we reduce recall bias and limit the opportunity for participants to construct a narrative. The high frequency of measurement also means that each individual report feels less consequential, potentially encouraging more honest answers than a single, high-stakes survey.

Protocol Design for Bias Reduction

Based on the principles of EMA, here is a specific protocol for our triadic experiment designed to maximize data quality:

Sampling Strategy: Semi-Random and Event-Contingent
- We will primarily use a semi-random sampling schedule. The day will be divided into several blocks (e.g., four 3-hour blocks from 10 am to 10 pm). A survey prompt will be sent at one random time within each block. This ensures coverage across the day without being predictable, preventing participants from preparing answers.
- We can supplement this with event-contingent sampling. Participants will be instructed to initiate a brief, specific survey immediately following a significant dyadic interaction (e.g., a shared meal, a conflict, or an intimate moment). This captures data in the immediate aftermath of key events before emotional responses are rationalized.
Item Selection and Phrasing
- Items should be specific, behaviorally-focused, and neutrally worded. Instead of "How happy are you with your partner?" (vague and leading), we might ask, "In the past hour, how supported have you felt by Partner A?"
- We will use validated items where possible. Resources like the ESM Item Repository are invaluable for this.
Building Trust and Ensuring Privacy
- The initial training session is crucial. We must explicitly state that the goal is to understand the full range of experiences, both positive and negative, and that there are no "right" or "wrong" answers.
- Assure participants of data confidentiality and frame the study as a tool for their own understanding, rather than as an evaluation.

This EMA protocol forms our first line of defense, creating an environment where honest, in-the-moment reporting is encouraged and facilitated.

2. Advanced Protection: The Randomized Response Technique (RRT)

Even with EMA, some questions may remain too sensitive for direct answers. Consider a question like, "Have you wished for more time alone, away from your partners, in the last hour?" Answering "yes" might feel like a betrayal, even in a private survey. For these highly sensitive binary items, we can employ a statistical method to ensure plausible deniability: the Randomized Response Technique (RRT).

The core idea is to introduce a randomizing device (like a virtual coin flip) that separates a participant's answer from their true state. This makes it impossible for the researcher to know if an individual's response is genuine, thereby protecting the respondent and encouraging honesty.

Week 8 : RANDOMIZED RESPONSE

The video "RANDOMIZED RESPONSE" from the Data Learner channel gives a wonderfully clear and concise explanation of the RRT mechanism and its logic.

Please watch from the explanation of the device to the derivation of the estimation formula. This covers the setup, the statistical logic, and the core equation for recovering the true prevalence. The example of its successful use in a survey about marijuana use at the end is a compelling demonstration of its power.

Protocol for Integrating RRT

We can embed an RRT question within our EMA surveys. For example:

The instruction: "Please think of the last two digits of your phone number. If the number is between 00 and 49 (50% chance), please answer 'Yes' to the following question, regardless of the truth. If the number is 50 or greater, please answer the question truthfully. Have you felt resentful toward either of your partners in the last hour?"

Because you, the researcher, cannot know the participant's phone number, a "Yes" response has plausible deniability. You cannot conclude that this specific person felt resentful.

However, as a statistician, you know that at the aggregate level, you can recover the true proportion. If $π$ is the true proportion of "yes" answers in the population, and $P (Y)$ is the proportion of "yes" responses you observe, the formula is:

π = \frac{P ( Y ) - p}{1 - p}

where $p$ is the probability of being instructed to answer "yes" automatically (in our example, $p = 0.5$ ). This technique trades statistical efficiency (the variance of our estimate of $π$ will be higher) for a reduction in measurement bias.

For a deeper dive into the statistical underpinnings, the foundational paper by Imai provides a comprehensive treatment.

[PDF] Sensitive Survey Questions with Auxiliary Information - Kosuke Imai

This paper by Kosuke Imai discusses several indirect questioning techniques, including the Randomized Response Technique (RRT). It formalizes the method within a robust statistical framework.

Read the introduction to the three main techniques. Then, focus on the section titled "Randomized Response Technique" (p. 424). This section provides the formal model and likelihood function, which will resonate with your statistical background. It treats RRT not just as a clever trick, but as a formal estimation problem.

By strategically using RRT for a few key sensitive items, we add a powerful layer of protection against social desirability bias, ensuring we capture even the most difficult aspects of the triad's experience.

3. Mitigating Observer Bias in External Assessments

While self-report EMA is our primary data source, we might want to validate our findings using data from external, trained observers who code video-recorded interactions (e.g., a structured conflict-resolution task). Here, the threat is not social desirability from the participant, but rater bias from the observer.

Rater bias is systematic error introduced by the rater's own psychology. Key forms include:

Leniency/Severity Bias: A rater's general tendency to rate everyone higher or lower than the average rater.
Halo Effect: A rater's global positive or negative impression of a person influences their ratings on specific, unrelated attributes.

This model shows how a person's report of another is a mix of different signals. An external observer's report would be similar: a combination of the target's actual behavior (accuracy) and the observer's own perceptual lens (bias).

This diagram illustrates how a report (e.g., husband's report of wife's conflict style) is influenced not just by the target's actual style ("wife accuracy") but also by the reporter's own tendencies ("husband bias"). Our goal with external observers is to maximize accuracy and minimize bias.

Generalizability theory provides a framework for quantifying these biases. Research using this theory has shown that rater bias can be a significant source of measurement error, attenuating correlations and reducing statistical power.

[PDF] Rater bias in observational data: A generalizability analysis

This thesis by Mark Becker provides a thorough review of rater bias and methods to assess and mitigate it.

Please read the following sections to understand the nature of the problem and the solutions: Start with the section "Bias in Observer Ratings" (p. 9) to get a clear definition of leniency and halo effects. Continue to "Impact of Rater Bias" (p. 11), which explains precisely how this bias attenuates correlations and reduces statistical power—a crucial point for experimental design. Finally, review Appendix B, "DESCRIPTION OF RATER TRAINING" (p. 71), which outlines a concrete, high-quality training protocol.

Protocol for Mitigating Rater Bias

Drawing from the principles in the resource, here is a protocol for a hypothetical observational component of our study:

Extensive Rater Training: As detailed in Appendix B of the Becker thesis, raters must undergo a rigorous training program. This involves studying a detailed, behaviorally-anchored coding manual, rating standardized training videos, passing proficiency tests (e.g., >80% of ratings within 1 point of a gold-standard score), and participating in ongoing calibration meetings to prevent "rater drift."
Use of Multiple Raters: This is the single most effective strategy. Every interaction should be coded independently by at least two, preferably three, raters. Idiosyncratic biases (the unique rater-target interaction) tend to cancel out when ratings are averaged. From a generalizability theory perspective, this dramatically increases the proportion of variance attributable to the "true" target score and improves the generalizability coefficient of your measure.
Blinding: Where possible, raters should be blind to the experimental condition and the study's primary hypotheses to prevent their expectations from influencing their scores.

By implementing this protocol, we can have confidence that any supplemental observational data is as objective as possible, providing a valuable check on our primary self-report measures.

Conclusion

In this lesson, we have constructed a comprehensive, multi-layered defense against the biases that threaten the validity of our outcome measures. By understanding the nature of these threats, we can proactively design protocols to minimize their impact.

Key Takeaways:

Ecological Momentary Assessment (EMA) is our primary strategy against social desirability bias. Its focus on in-the-moment reporting in the natural environment reduces recall bias and the cognitive space for constructing curated responses.
For highly sensitive topics, the Randomized Response Technique (RRT) provides statistical plausible deniability, encouraging honest answers by making it impossible to attribute a sensitive response to any single individual.
If using external observers for validation, rater bias is the main threat. This is best mitigated through a combination of rigorous training, using multiple independent raters whose scores are averaged, and blinding them to experimental conditions.

We now have robust plans for what to measure (the composite index) and how to measure it with high fidelity. However, collecting clean data is only half the battle. In our next module, we will confront the "replication crisis" in science and learn how to design our analysis plan to be as rigorous as our measurement plan, starting with identifying and pre-registering our "researcher degrees of freedom."

Recommended resources