Introduction
Welcome to the third lesson in our module on validating claims. In our previous lesson on Logic, we established that for a claim to be credible, the argument supporting it must be structurally valid. We learned that in a valid argument, if the premises are true, the conclusion is guaranteed. This led us to the concept of a sound argument: one that is both valid and has all true premises.
This raises a crucial question: how do we assess if the premises—the evidence and facts our argument is built on—are actually true? A perfectly logical argument can be used to "prove" a false conclusion if it's built on faulty or manipulated evidence. This brings us to the "O" in our FLOATER framework: Objectivity.
The learning outcome for this lesson is to assess the objectivity of a claim by identifying potential sources of bias, such as brand-sponsorship or self-selection of participants. We will define bias as a systematic deviation from the truth and explore the common ways it can infiltrate the data and evidence used to support a claim.
1. What is Bias?
As a statistician, you are intimately familiar with the concept of error in data. We often think of random error, the natural variation that can be accounted for with probabilistic models. Bias is different. It is a systematic error, a thumb on the scale that pushes results in a specific direction, leading to a distorted view of reality.
To get a formal definition and sense of its impact, let's start with a short video segment.
How Medical Research Gets It Wrong (Medical Bias Part 1)
The video 'How Medical Research Gets It Wrong' from Medlife Crisis provides a clear and concise definition of bias as a systematic error and explains why it's a pervasive issue in research.
Please watch from 00:51 to 01:43. Focus on the definition of bias and the distinction between research that tries to minimize it and research that doesn't.
As the video states, while bias is impossible to avoid completely, the goal of rigorous analysis is to identify, understand, and minimize its effects. When we assess a claim for objectivity, we are essentially acting as detectives, looking for clues of systematic error. Let's examine some of the most common culprits.
2. The Influence of the Sponsor: Funding and Career Bias
One of the most powerful sources of bias comes from those who have a vested interest in a particular outcome. This often involves financial sponsorship but can also include non-financial interests like career advancement or protecting a reputation.
The Cycle of Funding Bias
A sponsor's influence can be subtle and can appear at every stage of the research process. This is sometimes called the "cycle of bias."
Chapter: 2 Do Sponsoring Organizations Influence Research?
The article 'Do Sponsoring Organizations Influence Research?' from the National Academies Press offers a detailed look at how sponsors can introduce bias. This is a foundational concept for assessing objectivity.
Please read from the beginning of the chapter to the end of the 'Publication' subsection. Focus on the definition of 'funding bias' and the four points in the 'cycle of bias': agenda setting, research methods, conduct of research, and publication.
Let's apply this cycle to a claim from your area of interest in bra design:
Claim: "A study shows that the new 'ComfortCurve' bra, made by Brand X, is preferred by 8 out of 10 women over traditional designs."
To assess objectivity, we use the cycle of bias as a checklist:
- Agenda Setting: Who funded the study? If it's Brand X, their agenda is commercial, not purely scientific. They are less likely to fund a study asking, "What is the most comfortable bra design on the market?" and more likely to fund one asking, "Is our new design comfortable?" This frames the evidence from the start.
- Research Methods: Did Brand X have a role in setting the methods? Perhaps "traditional designs" were represented by poorly-fitting, low-quality bras. The standard of comparison can be chosen to make the sponsor's product look better.
- Conduct of Research: The reading points out that research is often part of a marketing strategy. Were the researchers—even if nominally independent—pressured to analyze the data in a certain way? Did the sponsor have final say on the report?
- Publication: This is publication bias, a critical concept. What if 5 studies were conducted, and 4 of them showed no significant preference for the 'ComfortCurve' bra? If Brand X only publishes the one favorable study, the public evidence is completely skewed. The claim that "a study shows" this result might be true, but it's profoundly misleading because it hides the full body of evidence.
Non-Financial Interests
Bias doesn't only stem from direct financial payments. The pressure to publish, secure grants, and build a career can also act as a powerful biasing force.
Money Isn’t the Only Thing That Can Bias Research
The video 'Money Isn’t the Only Thing That Can Bias Research' from Healthcare Triage expands on the idea of bias, showing how academic pressures and reputational concerns can be just as influential as direct funding.
Please watch from 00:17 to 02:58. Pay attention to how the desire for grant funding, career advancement, and protecting an organization's reputation can influence which studies are pursued and which results are published.
This "publish or perish" culture can create a bias toward novel, positive, and headline-grabbing results, while null findings (e.g., "This new bra strap material showed no improvement in comfort") are often seen as less valuable and may not be published.
3. The Problem of the Sample: Selection Bias
A claim can only be as objective as the data it's based on. If the group of people or data points selected for a study is not representative of the larger population of interest, we have selection bias.
The video we watched earlier has a great segment on this.
How Medical Research Gets It Wrong (Medical Bias Part 1)
Let's return to the Medlife Crisis video, which gives an excellent overview of selection bias and its various forms, including how subject drop-outs can affect results.
Please watch from 02:26 to 06:12. Notice the different examples given, from phone books to trial participants dropping out.
There are many specific types of selection bias. This image provides a useful summary of the most common ones.
Let's consider these in the context of your interests:
- Self-Selection Bias (Polyamory): Imagine a researcher posts a survey on a social media group for people in polyamorous relationships, asking about relationship satisfaction. The people who choose to respond are likely those who are most engaged with the community and potentially most satisfied with their relationship style. Those who had negative experiences may have left the group. This creates a sample that is not representative of all polyamorous experiences.
- Attrition Bias (Bra Design): A six-month study aims to test the durability and long-term comfort of a new bra design. Over time, participants who find the bra very uncomfortable or whose straps break are more likely to drop out of the study. The final data will be from the participants for whom the bra worked well, creating an overly positive assessment of its durability and comfort.
- Survivorship Bias (Bra Design): You decide to analyze the most popular vintage lingerie designs from the 1950s to find "timeless" design principles. You are only studying the designs that survived—the ones that were popular and are remembered today. You are missing all the failed designs. Concluding that "all 1950s designs used Feature Y" would be incorrect; it's the surviving ones that used Feature Y.
4. The Problem of the Response: Response Bias
Even with a perfect sample, bias can be introduced by the way participants provide information. Response bias occurs when people's answers don't reflect their true beliefs or experiences.
Social Desirability Bias
This is the tendency for people to answer questions in a way they believe is more socially acceptable or will be viewed favorably by others, rather than answering truthfully. This is a huge issue in social research, especially on sensitive topics.
Your interest in non-monogamy provides a classic context for this bias.
Updated Estimate of Number of Non-Monogamous People in U.S.
This article, 'Updated Estimate of Number of Non-Monogamous People in U.S.' from Psychology Today, briefly but directly explains how social desirability bias leads to under-reporting of non-monogamous behaviors.
Please read the short paragraph under the heading 'Probable Underestimate.' Note how the stigma around a topic can cause people to hide their true behavior, even in anonymous surveys.
Let's apply this to a claim:
Claim: "A survey of women in single-female-multiple-male polyamorous relationships found that 95% experience no jealousy."
The objectivity is highly suspect due to social desirability bias. Within a subculture that values compersion (finding joy in a partner's joy with another), admitting to jealousy might feel like a personal failing or a violation of the community's ideals. Participants might under-report or deny these feelings to align with perceived social norms.
Recall Bias
This bias occurs when participants' recollection of past events is inaccurate or incomplete. The Medlife Crisis video also touched on this (section 4, from 09:44). If a study asks, "How often did your bra feel uncomfortable last month?" the answers will be based on fallible memory, not precise data.
5. The Problem of the Researcher: Cognitive Biases
Finally, researchers themselves are not immune to bias. Their own beliefs, expectations, and cognitive shortcuts can systematically influence how they collect and interpret data.
- Confirmation Bias: The tendency to search for, interpret, favor, and recall information that confirms one's pre-existing beliefs.
- Bra Design Example: A designer believes firmly that flexible, non-metal underwires are the future of lingerie. During prototype testing, they might unconsciously interpret a participant's neutral comment like "it feels different" as positive confirmation ("see, she notices the innovation!"), while dismissing a negative comment like "it digs in here" as a fit issue unrelated to the core design concept.
- Observer/Interviewer Bias: This occurs when a researcher's expectations influence their interaction with participants or their interpretation of subjective data.
- Polyamory Example: A researcher studying relationship stability in a polyamorous triad holds a personal belief that such structures are inherently unstable. When interviewing the members, they might ask leading questions like, "Tell me about the challenges you've faced with scheduling," or interpret a simple disagreement over household chores as a sign of deep-seated structural conflict. The researcher's expectations color the data they collect.
The Medlife Crisis video has excellent short explanations of these two related concepts.
How Medical Research Gets It Wrong (Medical Bias Part 1)
Let's look at two final clips from the Medlife Crisis video that explain biases originating from the researcher's own mind.
Please watch two short segments: Confirmation Bias: from 01:43 to 02:26. Observer/Interviewer Bias: from 13:09 to 13:55. \nFocus on how the researcher's own mind can shape the evidence.
Conclusion
In this lesson, we have moved from the structure of an argument to the objectivity of the evidence it relies on. A claim cannot be trusted if the data supporting it is systematically skewed. Being an effective data analyst requires a healthy skepticism and the ability to spot these potential distortions.
Key Takeaways:
- Objectivity is the assessment of whether evidence is free from bias, which is a systematic error that pushes results in a particular direction.
- Sponsor Influence (Funding Bias) can distort research by setting a biased agenda, manipulating methods, influencing conduct, and selectively publishing favorable results (publication bias). Non-financial interests like career advancement also play a role.
- Selection Bias occurs when the study sample is not representative of the target population. Key types include self-selection, attrition, and survivorship bias.
- Response Bias happens when participants provide inaccurate data, often due to social desirability bias (giving socially acceptable answers) or recall bias (faulty memory).
- Researcher Bias involves the researcher's own mind skewing the results, through confirmation bias (favoring supporting evidence) and observer bias (letting expectations influence interpretation).
Preview of the Next Lesson:
We've now established that a claim must be falsifiable, logically structured, and based on objective evidence. But is the evidence sufficient? In our next lesson, we will address the "E" in FLOATER: Evidence. We will move beyond just identifying bias to evaluating whether the claim is supported by sufficient and appropriate evidence from the underlying data.