Reproducibility Assessment for Data Analysis

1. Defining Reproducibility vs. Replicability

The terms "reproducibility" and "replicability" are often used interchangeably, but in the context of data analysis, they have precise and distinct meanings. Understanding this distinction is the first step in assessing a claim.

Let's begin with a clear definition from a guide on the topic.

As the guide clarifies:

• Reproducibility is computational. It means taking the original author's data and code and running it to obtain the exact same results (e.g., tables, figures, statistical estimates). It answers the question: "Does this analysis actually produce the results the authors claim?" It is, as the resource notes, a "minimum necessary condition for a finding to be believable."

• Replicability is scientific. It means conducting a new study with new participants and data, following the original study's methods, to see if the same underlying finding emerges. It answers the question: "Is this finding robust and generalizable, or was it a fluke of the original sample or context?"

In our FLOATER framework for validating a specific claim from a given analysis, our immediate concern is reproducibility. Before we can even think about funding a new replication study, we must first be able to verify that the original analysis is computationally sound.

2. Why Reproducibility is a Non-Negotiable Standard

Reproducibility isn't just a "nice-to-have"; it's fundamental to the scientific process. It provides transparency and allows for independent verification, which protects against both honest error and intentional misrepresentation.

The following article explains why this concept is so critical for the scientific community.

As the article argues, science advances through gradual progress backed by independent verification. If an analysis cannot be reproduced, its findings are effectively a dead end. We cannot learn from it, build on it, or trust it. It remains an isolated assertion, not a piece of scientific evidence.

3. How to Assess Reproducibility: The Analyst's Checklist

To assess whether an analysis is reproducible, you must check for the presence and quality of three key components:

1. The Data: The exact, raw (or minimally processed) data used for the analysis.
2. The Code: The scripts (e.g., in R, Python, Stata) used to clean the data, perform the analysis, and generate figures and tables.
3. The Documentation: Instructions explaining the computational environment (software versions, libraries), the steps to execute the code, and a data dictionary explaining the variables (tying back to our SAD metadata lesson).

Many journals and organizations have developed checklists to formalize this assessment. While they may seem geared toward clinical trials, their principles are universal.

Let's apply this checklist mentality to our core examples.

Bra Design Example

• Claim: "A new strap attachment point, Angle X, reduces shoulder pressure by an average of 15% compared to the standard Angle Y, based on sensor data from 50 wear-testers."
• Reproducibility Checklist:
  • Data: Is the raw sensor pressure data for all 50 participants, for both Angle X and Angle Y conditions, provided in a machine-readable format (e.g., .csv)?
  • Code: Is the script provided that reads the raw data, filters artifacts, calculates the average pressure for each condition, and runs the statistical test (e.g., a paired t-test) comparing the two?
  • Documentation: Is there a README.md file specifying that the analysis was run using R version 4.2.1 with the tidyverse package? Does it explain what the columns in the data file mean?

If the authors only provide a paper with a summary table, the claim is not reproducible. You cannot verify their calculations or check for potential errors in their analysis pipeline.

Polyamorous Relationships Example

• Claim: "A thematic analysis of 25 interviews reveals that 'explicit scheduling of one-on-one time' is a primary strategy for maintaining relationship satisfaction in single-female-multiple-male triads."
• Reproducibility Checklist:
  • Data: This is trickier due to high sensitivity. Reproducibility doesn't always require public data. It requires verifiable data. Are the anonymized interview transcripts available in a secure data repository for credentialed researchers?
  • Code/Method: Qualitative analysis is different from quantitative. Is the detailed "codebook" used for thematic analysis provided? This would define what counts as "explicit scheduling" and other themes. Are sample passages provided showing how the codes were applied? This is the "code" of qualitative work.
  • Documentation: Is the protocol for selecting and interviewing participants clear? Are the researchers' positionality statements (potential biases) included?

If the paper only presents exemplar quotes without access to the broader dataset or the coding framework, another researcher cannot verify that the theme was indeed primary and not just a product of selection bias. The claim is not reproducible.

4. Barriers to Reproducibility

If reproducibility is so crucial, why is it often not achieved? There are innocent reasons (e.g., messy code, poor documentation habits) and less innocent ones. Certain analytical practices make a study's findings fragile and unlikely to be reproducible. A researcher who engages in these practices is often reluctant to share the data and code that would expose them.

Your background in statistics will make you familiar with these issues, often called "questionable research practices."

When you assess a claim and find that the data or code is unavailable, you should be skeptical. It may be that the researchers are simply disorganized. But it may also be that the analysis is not reproducible because the results were generated through p-hacking (trying many analyses until one is significant) or HARKing (Hypothesizing After the Results are Known). Sharing the full analytical script would make these practices obvious, providing a strong disincentive for transparency.

Therefore, an absence of reproducible materials is not a neutral signal; it's a red flag that lowers the credibility of a claim.

Conclusion

This lesson completes our journey through the FLOATER framework for validating data-driven claims. Reproducibility is the final checkpoint, ensuring that an analysis is technically sound and its results are verifiable.

Key Takeaways:

• Reproducibility means getting the same results using the same data and code. It is a minimum requirement for a claim to be believable.
• Replicability means getting similar results in a new study with new data. It speaks to the generalizability of a finding.
• To assess reproducibility, you must check for the availability and quality of the data, the code, and the documentation.
• The absence of these materials is a red flag, as it can hide questionable research practices like p-hacking and HARKing that make findings fragile.

With the FLOATER mnemonic—Falsifiability, Logic, Objectivity, Alternative Explanations, Tentative Conclusions, Evidence, and Reproducibility—you now have a comprehensive toolkit to critically evaluate any claim that is presented to your students.

Preview of the Next Module:
Having mastered the art of validating a claim, we will now move on to validating a solution. In the next module, we will introduce the PULPED framework. This will guide us in assessing a proposed course of action or implementation that is based on data analysis. We will shift from asking "Is this claim true?" to asking "Is this proposed solution practical, ethical, and sound?"