Planning to Avoid HARKing

1. HARKing: The Texas Sharpshooter Fallacy

HARKing is the practice of presenting a hypothesis that was formulated after data analysis as if it were an a priori hypothesis. It's a natural human tendency to want to tell a clean, compelling story, and results often suggest stories we hadn't anticipated. The error is not in finding something unexpected, but in presenting that discovery as a successful prediction.

A common analogy for this is the "Texas Sharpshooter."

This act of drawing the target after the shot is the essence of HARKing. It turns a random hit into a bullseye, creating a misleading impression of precision and foresight. In research, this means a p-value from a post-hoc analysis is presented as if it were from a pre-specified confirmatory test, rendering the statistical inference invalid.

The cartoon below illustrates this concept on a spectrum. On the far left, you see the sharpshooter fallacy, representing purely exploratory work disguised as confirmatory—a recipe for "wonky stats." On the right, sound statistics are associated with true confirmatory research, where the target is defined before the shot is taken.

2. The Two Mindsets: Confirmatory vs. Exploratory Analysis

The antidote to HARKing is not to stop exploring data, but to be rigorously honest about when you are exploring and when you are confirming. These two activities are both vital to science, but they have different goals and different rules. Your advanced training in statistics provides the tools for both, but the credibility of your work depends on keeping them separate.

Let's formalize this distinction:

• Confirmatory Research:

  • Goal: Rigorous hypothesis testing.
  • Error to Minimize: Type I Error (false positives). We want to be very sure that a declared effect is real.
  • Method: A small number of pre-specified tests of a priori hypotheses.
  • P-value Interpretation: Statistically valid and interpretable as the probability of observing the data (or more extreme) if the null hypothesis is true.

• Exploratory Research:

  • Goal: Hypothesis generation and discovery of unexpected patterns.
  • Error to Minimize: Type II Error (false negatives). We don't want to miss a potentially important new lead.
  • Method: Can involve many tests, data visualization, and flexible modeling.
  • P-value Interpretation: Not statistically valid for inference. A low p-value here is merely a "suggestion" or a "lead" that warrants a new, confirmatory study.

HARKing occurs when a result from the exploratory column is moved into the confirmatory column within the same study. Pre-registration prevents this by forcing you to commit to your confirmatory plan ahead of time.

3. Structuring the Analysis Plan to Prevent HARKing

Now, let's translate this philosophy into the structure of the analysis plan for our vee-relationship experiment. In the last lesson, we defined the content of the plan; now we define its structure for reporting. The plan, and the eventual manuscript, must have two distinct sections.

3.1. The Confirmatory Analysis Section

This section is sacred and immutable. It contains only the single primary analysis we pre-specified.

• Hypothesis 1 (Confirmatory): The "Nesting" living arrangement will result in a significantly higher mean Triadic_Satisfaction score compared to the average of the two "Separate" living arrangements.
• Analysis Plan for H1:
  • Model: A linear mixed-effects model Triadic_Satisfaction ~ Living_Arrangement + Period + (1 | Participant_ID).
  • Test: A single, planned contrast of (Nesting) vs. (average of Separate A, Separate B).
  • Inference Criterion: The result will be judged against an alpha level of $\alpha =0.05$. The result of this one test will be the sole basis for our confirmatory conclusion.

Anything not in this section is, by definition, not confirmatory.

3.2. The Exploratory Analysis Section

This is the sandbox. Here, we can look for other patterns, but we must label our findings appropriately. This section itself can be subdivided.

• Pre-specified Exploratory Analyses: These are analyses we anticipate doing but which are not our primary focus. We pre-register our intent to conduct them to increase transparency.
  • Hypothesis E1 (Exploratory): Does Living_Arrangement affect the individual satisfaction scores of Person A, Person B, and Person C, or the dyadic satisfaction scores of AB and AC?
  • Analysis Plan for HE1: We will run five separate LMMs, one for each of these secondary outcomes. We will apply the Benjamini-Hochberg procedure to the resulting p-values to control the False Discovery Rate at q = 0.05.
• Unplanned Exploratory Analyses: Our plan must also have a clause for true discoveries—things we didn't even think to look for.
  • Clause: "Any analyses conducted that were not specified in the Confirmatory or Pre-specified Exploratory sections will be reported in a dedicated 'Unplanned Exploratory' subsection. For example, if we observe a strong time-of-week effect, we may retrospectively add it as a covariate to a model. However, the results of such analyses will be clearly labeled as post-hoc and hypothesis-generating, requiring independent replication."

This partitioned structure makes HARKing nearly impossible. If a fascinating result emerges from the exploratory analyses, you cannot pretend it was your primary hypothesis. You must report it as an exciting, but tentative, exploratory finding.

4. What Constitutes a "Good" Plan?

It's tempting to think that a "good" pre-registration must be exhaustively detailed, anticipating every possible contingency. However, this can be counterproductive. An overly verbose plan can obscure the primary, confirmatory hypothesis. The true goal is clarity and demarcation.

The key takeaway from this resource is that specificity, not verbosity, is what matters. Our plan for the single confirmatory test is highly specific, even if it's short. That specificity is what gives it power. By stipulating the DV (Triadic_Satisfaction), the IV (Living_Arrangement), the model (LMM), the test (planned contrast), and the inference criterion ($\alpha =0.05$), we have eliminated the degrees of freedom that would allow for HARKing around our primary claim.

Conclusion

Today, we've formalized the critical distinction between confirmatory and exploratory analysis, the conceptual backbone of credible research. By designing our analysis plan with explicit, separate sections for each, we move beyond simply avoiding p-hacking and actively prevent HARKing.

Key Takeaways:

• HARKing (Hypothesizing After the Results are Known) is the fallacy of presenting a post-hoc finding as an a priori prediction.
• The antidote to HARKing is not to forbid exploration, but to transparently distinguish confirmatory (hypothesis-testing) from exploratory (hypothesis-generating) analyses.
• A robust analysis plan should be physically structured into separate sections for confirmatory and exploratory analyses.
• Confirmatory plans are defined by their specificity on key points (hypothesis, variables, model, test), not by their length.
• Any deviation from the pre-specified confirmatory plan, or any unplanned analysis, must be labeled as exploratory, signaling that the findings are tentative and require new data for confirmation.

With this structure in place, our pre-registration becomes more than just a procedural hurdle; it becomes a clear statement of intent that protects our work from bias and shores up the credibility of our findings.

In our next lesson, we will examine another tool for ensuring our results are credible: the Test for Excess Success. This will allow us to evaluate pilot data or even our final results to see if they are, perhaps, "too good to be true," which can be a symptom of publication bias or other subtle problems.