Understanding Bias in Study Design

From Random Error to Systematic Bias

In any study, the results we observe will almost never be exactly the true value in the population. This deviation can be due to two types of error: random error and systematic error (bias).

• Random Error (Chance): This is the variability that arises from chance alone. If you flip a fair coin 10 times, you might get 6 heads just by chance. If you flip it 1000 times, you're much more likely to get close to 50% heads. Random error is non-directional and can be reduced by increasing the study's sample size.
• Systematic Error (Bias): This is a flaw in the study's design or conduct that causes the results to be consistently distorted in a particular direction. Unlike random error, bias cannot be reduced by increasing the sample size. If your measuring tape is miscalibrated and is one inch too short, measuring more people won't fix the error; it will just make you more precisely wrong.

The following video provides a concise overview of the three major threats to a study's validity: confounding, chance, and bias.

Epidemiological studies are ultimately concerned with getting as close to the "truth" as possible. Bias is a major obstacle that can mislead us. Broadly, these systematic errors fall into two main categories, which can be thought of as errors in the inputs of a study.

As the diagram illustrates, we'll focus on the two main types of bias that corrupt a study's inputs: selection bias and information bias.

Selection Bias: Are We Studying the Right People?

Selection bias occurs when the procedures used to select study participants—or retain them in the study—lead to a study population that is systematically different from the target population you want to learn about. This bias happens at the recruitment or retention stages.

From your CIS perspective, this is analogous to training a machine learning model on a flawed or non-representative dataset. If you build a spam filter trained only on emails from a decade ago, it will perform poorly on modern spam because the "selection" of training data was biased.

The video below explains this concept using the Framingham Heart Study data we've discussed.

Selection bias can manifest in several ways depending on the study design. The following reading provides detailed examples of its various forms.

Here are some of the most common and important types of selection bias:

• Inappropriate Control Selection (Case-Control Studies): This is a critical vulnerability of case-control studies. Controls should represent the exposure distribution in the source population that produced the cases. A classic example is Berkson's bias, where using hospitalized controls can be problematic because they may have different exposure patterns than the general population. The coffee and pancreatic cancer study, where controls were advised not to drink coffee for other medical reasons, is another prime example of this.
• Loss to Follow-up (Cohort Studies): If people who drop out of a study are systematically different from those who remain, the results can be biased. This is particularly damaging if the dropout rate differs by exposure status (e.g., more unexposed people drop out than exposed people).
• Healthy User/Worker Effect (Cohort Studies): People who are healthy enough to work (or who choose to take a preventative medication like Hormone Replacement Therapy in the famous example) are often systematically healthier than the general population. Comparing them to the general population can create a spurious association between the "exposure" (working, taking the drug) and better health outcomes.

Selection bias is extremely difficult, and often impossible, to correct in the analysis phase. The primary defense is careful study design.

Information Bias: Is the Data We're Collecting Accurate?

Information bias, also called measurement bias or misclassification, occurs when there is a systematic error in the way exposure, outcome, or other variables are measured or classified. This bias happens during the data collection stage. It means the data itself is flawed.

To continue the computer science analogy, this is like using a faulty sensor that consistently reports incorrect temperature readings, or having a data-entry script that always mistakes the number '1' for the letter 'l'. The information itself is corrupted.

The following video section explains this concept.

As with selection bias, there are several important sub-types of information bias.

Two of the most common types are:

• Recall Bias: Occurs when there are systematic differences in how accurately participants recall past events. This is a major threat in case-control studies. For example, a mother of a child with a rare birth defect (a case) is likely to think much harder about and recall potential exposures during pregnancy than a mother of a healthy child (a control).
• Observer/Interviewer Bias: Occurs when the person recording the data has knowledge of the participant's exposure or disease status, and that knowledge influences their measurements or questions. For instance, an interviewer who knows a participant has lung cancer might probe more deeply for a history of smoking than they would with a healthy control. Blinding is the primary defense against this.

Differential vs. Non-differential Misclassification

A crucial distinction in information bias is whether the measurement error is the same for all groups being compared.

• Non-differential misclassification: The measurement error is the same in all groups. For example, an imprecise scale is used to weigh both the exposed and unexposed groups. This type of bias is problematic, but it usually biases the measure of association towards the null (i.e., it makes the groups look more similar than they really are).
• Differential misclassification: The measurement error differs between groups. For example, cases are asked about exposures more carefully than controls. This is far more dangerous because it can bias the association in any direction—either away from or towards the null—and the direction can be unpredictable.

Practice Identifying Bias

Let's test your ability to identify these biases. The following scenarios are adapted from the "Observational Studies and Bias in Epidemiology" document. For each one, try to identify the most likely type of bias.

Scenario 1: A case-control study is conducted to assess the association between Alzheimer's disease and exposure to aluminum-containing homecare products within the last five years. Information is gathered via an in-home survey of the participants.

Scenario 2: A case-control study of fatal car crashes compares crash victims (cases) with living drivers (controls). Blood samples are taken from all fatalities to chemically measure blood-alcohol content. For the living controls, data on alcohol use is gathered via a self-report interview.

Conclusion

In this lesson, you have learned to distinguish systematic bias from random error and to identify the two major categories of bias that threaten the validity of a study's findings.

Here are the key takeaways:

• Bias is a systematic error in study design or execution that distorts results in a consistent direction. It cannot be fixed by increasing sample size.
• Selection Bias concerns who gets into the study and who stays in it. It occurs when the study population is not representative of the target population in a way that relates to the exposure and outcome.
• Information Bias concerns how data are collected. It occurs when measurements or classifications of exposure or outcome are systematically inaccurate.
• The primary strategy for dealing with bias is prevention through careful study design, such as using robust sampling methods, blinding, and standardized data collection protocols.

In this module, we have now covered the major observational study designs and the biases that can affect them. We have briefly mentioned a third threat to validity: confounding. This occurs when the effect of the exposure is mixed up with the effect of another variable. Our next module will be dedicated entirely to defining, identifying, and controlling for confounding.