In the last lesson, we focused on dissecting the results of a clinical trial, learning how to interpret efficacy measures like the hazard ratio and evaluate the safety profile. You learned to answer the question, "What did the study find?" Now, we take the crucial next step in critical appraisal. It's not enough to know what the results are; we must also question their credibility and applicability.
This lesson addresses two fundamental questions that underpin all evidence-based practice: "Can I trust these results?" and "Do these results apply to a broader population?" To answer them, we will explore the concepts of internal and external validity. By the end of our session, you will be able to evaluate the internal and external validity of a clinical trial, identifying the design features that make a study's conclusions believable and the factors that limit their generalizability. This skill is the cornerstone of moving from simply reading research to critically appraising it.
The Two Pillars of a Trial's Credibility
When you evaluate a study, you are essentially acting as a detective, looking for clues about the quality of the evidence. A leading researcher in oncology, Dr. Alberto Sobrero, suggests a structured approach to this "investigation."
Clinical interpretation and validity of clinical trials: When do they become practice changing?
This video from HMP Education introduces a powerful framework for interpreting trial results. We'll focus on the first two core questions it poses.
Watch the segment from the structured way to the end of the section. The speaker outlines three sequential questions. The first two—"Can I trust these results?" and "How relevant are these results?"—are the focus of today's lesson.
These two questions correspond directly to the core concepts of internal and external validity.
Internal vs. External Validity | Understanding Differences & Threats
This article from Scribbr provides concise, clear definitions of our two key concepts.
Read the introductory paragraphs, starting from the definitions, to firmly establish the distinction between internal and external validity.
To put it simply:
- Internal Validity is about the integrity of the study's findings for the specific people who participated. It answers: Are the conclusions drawn about the treatment's effect on this sample of patients correct and free from bias?
- External Validity is about the generalizability of those findings to the outside world. It answers: Can I expect the same results if I apply this treatment to different patients in different settings?
A study must have strong internal validity to be credible. If the results are not even true for the participants in the study, they are certainly not useful for anyone else.
Internal Validity: Did the Treatment Really Cause the Effect?
Internal validity is a measure of how confident we can be that the observed outcome was caused by the intervention (the treatment) and not by other factors, often called confounding variables or biases. In the context of your background, think of this as a controlled unit test for a piece of code. The goal is to isolate the function (the treatment) and test its logic with carefully controlled inputs (the patient groups) to ensure the output (the health outcome) is a direct result of that function's execution, not some unexpected system state or bug elsewhere.
The design features we've already discussed—randomization, blinding, and the use of a control group—are the primary tools for ensuring high internal validity.
- Randomization minimizes selection bias by creating comparable groups from the outset.
- Blinding prevents performance and detection bias by ensuring that knowledge of the treatment assignment doesn't influence the behavior of patients or investigators.
However, even in a well-designed trial, things can go wrong. These potential problems are known as "threats to internal validity."
Internal vs. External Validity | Understanding Differences & Threats
We will revisit the Scribbr article to explore common threats that can undermine a trial's internal validity.
Focus on the table under the heading "Threats to internal validity". While the example is about job satisfaction, the principles apply directly to clinical trials. Pay special attention to the threats of Attrition, History, and Social Interaction.
Let's translate the most relevant threats into a clinical trial context:
- Attrition Bias: This is a major concern. It occurs when participants who drop out of the study are systematically different from those who remain. For example, if patients in the treatment arm drop out at a high rate due to severe side effects, analyzing only the remaining "per-protocol" patients (who could tolerate the drug) would create an overly optimistic view of the treatment's efficacy and safety. This is precisely why the intention-to-treat (ITT) principle, which we discussed in the previous lesson, is so critical for preserving internal validity.
- History: An external event occurs during the trial that affects the outcome. For instance, if a new, effective supportive medication becomes widely available during a long cancer trial, it might improve outcomes in both arms, making it harder to discern the true effect of the investigational drug.
- Instrumentation: The way the outcome is measured changes over time. Imagine a trial measuring tumor size where the hospital acquires a new, higher-resolution MRI scanner halfway through. If measurements from the new scanner are not calibrated with the old one, it could introduce a systematic error.
To see how these elements are assessed in practice, let's watch a short video segment.
How to EVALUATE A CLINICAL TRIAL (journal club/clinical practice)
This video from The Pharmacist Academy walks through the practical steps of evaluating a trial's methods.
First, watch the discussion on study arms and design. Notice how the speaker emphasizes randomization, the type of control group, and blinding as key indicators of a strong trial. Then, jump to the section explaining different analysis populations, from intent-to-treat versus per-protocol. This reinforces why the ITT population is the standard for assessing efficacy because it reflects real-world challenges like non-compliance and dropouts, thus protecting internal validity against attrition bias.
External Validity: Do the Results Matter in the Real World?
Even if a trial has perfect internal validity—meaning its conclusions are perfectly true for its participants—it might have limited value if the results cannot be generalized. This is the question of external validity.
Continuing our software analogy, this is like deploying your perfectly unit-tested function into a complex production environment. It may fail because real-world user inputs are messy and unpredictable, or it may conflict with other parts of the system—factors you controlled for and eliminated during testing.
The main factors that limit a trial's external validity are related to its participants, setting, and protocol.
Internal vs. External Validity | Understanding Differences & Threats
Let's return one last time to the Scribbr article to examine the threats to external validity.
Read the table under the heading "Threats to external validity". Pay close attention to Sampling bias and the Hawthorne effect.
In clinical trials, the most significant threat is sampling bias, which often manifests through overly strict inclusion and exclusion criteria.
- Strict Eligibility: To ensure a clean "signal" and enhance internal validity, researchers often exclude patients who are older, have multiple other diseases (comorbidities), or are taking other medications.
- The Problem: The resulting study population may be much younger and healthier than the typical patient who would receive the treatment in a real-world clinic. As the video LINK highlighted, if a trial for a new breast cancer drug only includes patients with Stage IV disease but excludes those with common comorbidities like diabetes or heart disease, can you confidently apply its findings to a Stage IV patient who does have diabetes? The answer is often "no," or at least, "with great caution."
Another issue is the trial setting. A study conducted at a world-class academic hospital with top specialists and high-tech equipment might produce results that can't be replicated in a standard community hospital with fewer resources.
The Inevitable Trade-Off: Explanatory vs. Pragmatic Trials
You might have noticed a tension here. The steps taken to increase internal validity (like strict patient selection and controlled settings) often decrease external validity. Conversely, a study conducted in a messy, real-world environment (high external validity) might be so full of confounding factors that it's hard to be sure the treatment caused the effect (low internal validity).
This is a fundamental trade-off in research design.
This trade-off has led to a useful distinction between two types of trials: explanatory and pragmatic.
[PDF] Promoting Both Internal and External Validity: Designing the Trial to ...
This document from the NIH Collaboratory provides an expert perspective on the distinction between explanatory and pragmatic trials.
First, read the introductory section that defines the two validities and introduces the idea of a trial as a test of a causal mechanism. Next, read the following section, which formally introduces the concepts of explanatory and pragmatic trials. Finally, read the section that describes how this choice affects design, focusing on the bulleted list of characteristics of traditional trials.
As the document explains:
- Explanatory Trials are designed to determine if an intervention can work under ideal, or "laboratory-like," conditions. They prioritize internal validity. The goal is to understand a biological mechanism. Question: Does this drug work in principle?
- Pragmatic Trials are designed to determine if an intervention works under usual, real-world conditions. They prioritize external validity. The goal is to inform clinical, patient, and policy decisions. Question: Does this drug work in practice?
Most Phase III trials you read about will be somewhere on a spectrum between purely explanatory and purely pragmatic. Understanding a trial's primary intention helps you judge its results appropriately. If a highly explanatory trial shows a benefit, your reaction might be, "Interesting, it seems the drug has a biological effect." If a highly pragmatic trial shows a benefit, your reaction is more likely to be, "This is relevant, and I should consider this for my patients."
Conclusion
Evaluating a clinical trial goes far beyond checking a p-value. It requires a critical assessment of the study's design and context. Internal validity tells you if the findings are trustworthy for the sample studied, while external validity tells you if they are meaningful for the wider world.
Here are your key takeaways:
- Internal validity is the degree of confidence that the observed effect was caused by the intervention. It is enhanced by randomization, blinding, and ITT analysis, and threatened by biases like attrition.
- External validity is the degree to which study results can be generalized to other populations, settings, and times. It is often limited by strict eligibility criteria and idealized trial settings.
- There is an inherent trade-off between internal and external validity.
- This trade-off is framed by the explanatory-pragmatic spectrum. Explanatory trials prioritize internal validity to test a mechanism, while pragmatic trials prioritize external validity to inform real-world practice.
This lesson concludes our module on the design and interpretation of clinical trials. You now have a comprehensive toolkit to read a trial paper, understand its results, and critically appraise its strengths and limitations.
In our next module, we will pivot to a new topic that is central to both epidemiology and clinical trials: Survival Analysis. We'll begin by defining time-to-event data and the fundamental concepts of censoring, which are essential for analyzing data where the primary outcome is not just if an event occurs, but when.