Introduction
In our previous lesson, we explored the 'A' in our FLOATER framework: Alternative Explanations. We established that for any observed finding, we must act as our own skeptics, systematically generating and evaluating rival hypotheses—from confounding variables to statistical artifacts like survivorship bias. This process ensures we don't prematurely accept a simple causal story.
Having rigorously challenged the why behind a claim, we now turn to the final step before assessing reproducibility: deciding on the strength of our conclusion. This brings us to the 'T' in FLOATER: Tentative Conclusions.
The learning outcome for this lesson is to distinguish between definitive conclusions and tentative conclusions that are open to revision with new data. We will explore the idea that science is a process of reducing uncertainty, not establishing absolute proof. Your background in probability theory, which deals with degrees of belief and likelihoods rather than absolute certainties, provides an excellent foundation for this concept.
1. Science Reduces Uncertainty; It Does Not Provide "Proof"
A common misconception, especially among the public, is that a single scientific study "proves" a fact. As a statistician, you know that the reality is more nuanced. We deal in evidence and probabilities, not absolutes. Empirical research, particularly in the social sciences and applied fields, is an inductive process. We observe specific instances and attempt to infer general principles. These inferences are, by their very nature, provisional.
The goal of research is not to arrive at an unshakable, definitive truth, but to methodically reduce our uncertainty about a phenomenon. Every study contributes a piece to the puzzle, and with it, our confidence in a conclusion may grow, but it should never become absolute.
The temptation to present findings as more certain than they are is strong. Researchers are often rewarded for making strong claims. This can lead to what are called "breakdowns" in scientific practice, where the certainty of a conclusion is overstated. Let's look at a resource that explains the mechanics behind this.
Breakdowns in Scientific Practices: How and Why Some Accepted ...
The paper 'Breakdowns in Scientific Practices' explains why researchers might overstate their conclusions. It introduces the critical distinction between inductive and deductive reasoning and how applying a deductive mindset to inherently inductive problems leads to overconfidence.
Please read the subsection 'The Deductive-Style Reasoning Condition'. Focus on how the text contrasts inductive reasoning (tentatively suggesting a broad conclusion from specific examples) with deductive-style reasoning (where the conclusion is guaranteed by the premises). Pay attention to the argument that researchers often prefer the false certainty offered by deductive-style logic.
This desire for certainty can lead researchers to misinterpret statistical tools. For instance, as the paper notes, a low p-value from a Null Hypothesis Significance Test (NHST) is sometimes treated as deductive "proof" that an effect is real, rather than what it is: a piece of inductive evidence against the null hypothesis. This mindset is a primary driver for presenting tentative findings as if they were definitive.
2. Groundbreaking vs. Definitive: A Crucial Distinction
To better manage the level of certainty we assign to a claim, it's useful to differentiate between two types of findings: groundbreaking and definitive. A groundbreaking finding is novel and exciting, but it is also the first step on a long road. A definitive conclusion is one that has withstood repeated testing by independent researchers.
The following blog post from a social psychologist explains this distinction beautifully.
Groundbreaking or Definitive? Journals Need to Pick One - SPSP
In 'Groundbreaking or Definitive? Journals Need to Pick One', the author argues that these two qualities are often incompatible within a single study. This piece will help us formalize the journey a claim takes from initial discovery to established fact.
Please read the first three paragraphs, which define 'groundbreaking' and 'definitive'. Then, read the section that begins 'How do we get from groundbreaking to definitive?'. Focus on the key phrase the author introduces: 'independent replication across multiple methods'.
As the author states, a single study cannot be both the start and the finish of an idea. Let's apply this framework to our domains of interest.
Bra Design Example:
Imagine you conduct a small-scale wear test (n=30) for a new bra prototype. Your data suggests a novel underwire material reduces poke-through and discomfort by 40% compared to standard steel wires.
- This is a groundbreaking finding. It's exciting and suggests a new direction for design.
- It is, however, highly tentative. Could the effect be due to the specific body types in your sample? The particular garment construction you used? A placebo effect from participants knowing it's a "new" material?
- To become definitive, this finding would need to be confirmed through:
- Independent replication: Other lingerie companies and academic labs would need to run their own studies.
- Multiple methods: The finding would need to hold up in larger longitudinal studies (tracking discomfort over months), in lab tests using force sensors, and with various bra styles and fabric combinations.
Only after this extensive body of evidence accumulates can you state with high confidence that the new material is definitively superior.
Polyamorous Relationships Example:
You conduct a survey and find that in single-female-multiple-male triads, groups that report using a shared calendar for scheduling one-on-one time also report higher levels of relationship satisfaction.
- This is a groundbreaking, interesting correlation.
- It is deeply tentative. As we discussed in the last lesson, this could be a case of a third variable (e.g., more conscientious people are both more likely to use calendars and more likely to work on their relationships) or even reverse causality (higher satisfaction leads people to invest more in tools like shared calendars).
- To become definitive, this claim would need support from a much wider range of evidence, such as:
- Longitudinal studies: Following new triads over time to see if adopting a calendar precedes an increase in satisfaction.
- Qualitative interviews: In-depth case studies to understand how and why a calendar might contribute to satisfaction, or if it's merely a proxy for other communication habits.
- Experimental designs: An intervention study where some new triads are randomly assigned to use a calendar and a control group is not.
3. The Consequences of Overstatement
When we fail to distinguish between tentative and definitive conclusions, we risk what the first resource called a "breakdown." This happens when we act with a level of confidence that the evidence simply does not support.
Breakdowns in Scientific Practices: How and Why Some Accepted ...
Let's return to the 'Breakdowns' paper to see how the problematic application of statistics contributes to this issue. This section is particularly relevant given your statistical background.
Please read the subsection 'NHST fulfills the deductive-style reasoning condition' and the section 'The “replication crisis”...'. Observe how the authors argue that misinterpreting NHST creates undue certainty in single studies, which in turn devalues replication and perpetuates weak claims.
Presenting a tentative conclusion as definitive is not just a philosophical error; it has practical consequences.
- In bra design: If a designer rushes a "groundbreaking" feature into mass production based on a single, small study, the company could face millions in losses if the finding doesn't hold up in the broader market.
- In relationship research: If a therapist or writer presents a tentative finding from one survey as "the key to successful polyamory," they could give damaging advice. People might adopt practices that are a poor fit for their relationships, leading to conflict and distress, all based on a conclusion that was never more than provisional.
4. Communicating with Intellectual Humility
The final, crucial skill is to communicate your findings in a way that reflects the appropriate level of certainty. Your students must learn to frame their conclusions not as final verdicts, but as contributions to an ongoing conversation.
Groundbreaking or Definitive? Journals Need to Pick One - SPSP
Let's read the final important takeaway from 'Groundbreaking or Definitive?'. It offers direct advice on the mindset and language we should adopt.
Please read the last two paragraphs of the article. Focus on the author's call for skepticism and the specific phrase they suggest as the standard, enthusiastic response to a new finding.
The author's proposed response—"wow, that'll be really interesting if it replicates"—perfectly encapsulates the principle of the tentative conclusion. It acknowledges the finding's interest (groundbreaking) while immediately pointing to the need for confirmation (the path to definitive).
Here is how you can teach your students to operationalize this:
| Avoid (Definitive Language) | Prefer (Tentative Language) |
|---|---|
| "This study proves that..." | "This study's findings suggest that..." |
| "The results show that..." | "The evidence is consistent with the hypothesis that..." |
| "We have demonstrated a causal link." | "An association was observed between X and Y." |
| "This is the optimal design." | "This design shows promise and warrants further investigation." |
| "The conclusion is clear." | "While the results are intriguing, they must be considered provisional." |
By adopting this language, we behave more ethically as researchers and communicate more accurately, building trust and fostering a healthier scientific ecosystem.
Conclusion
Today we have integrated the principle of Tentative Conclusions into our FLOATER framework. We have moved from evaluating the evidence and its alternatives to deciding how strongly to state our claim.
Key Takeaways:
- Scientific knowledge is inherently provisional. Research aims to reduce uncertainty, not to achieve absolute proof.
- Distinguish between a groundbreaking finding (novel, interesting, but new) and a definitive one (supported by a large body of replicated evidence).
- Overstating certainty, often by applying deductive-style reasoning to inductive problems, leads to scientific "breakdowns" and poor real-world decisions.
- Communicate your findings with intellectual humility, using language that reflects the true level of evidence. A good finding is interesting, but it's only as good as its ability to be replicated.
Preview of the Next Lesson:
Our discussion today has repeatedly highlighted one critical concept: replication. The journey from a tentative, groundbreaking finding to a near-definitive conclusion is paved with successful replications. This brings us to the final letter in our FLOATER mnemonic: 'R' for Reproducibility. In our next lesson, we will define what it means for a study to be reproducible and discuss why it serves as the ultimate arbiter of a claim's validity.