In our last lesson, we explored the cognitive costs of task-switching, noting the measurable performance drops seen in experiments. We touched on concepts like p-values and effect sizes () when reviewing those findings. Today, we will build directly upon your statistical foundation to equip you with the tools to critically evaluate the very research this course is built on.
This lesson is designed to move you from being a consumer of scientific claims to being a discerning critic. We will address the learning outcome of interpreting effect sizes and common study designs used in cognitive neuroscience. By understanding the methods, their strengths, and their pitfalls, you will be able to assess the true weight of the evidence behind any proposed "performance-enhancing" habit.
From Statistical Significance to Practical Significance
As a statistician, you are intimately familiar with Null Hypothesis Significance Testing (NHST) and the role of the p-value. In modern research, however, there is a strong movement towards what is called "The New Statistics," which emphasizes estimating the magnitude of an effect over the binary decision of statistical significance.
An effect size is simply the quantitative answer to a research question—"by how much?"—and it should always be accompanied by a confidence interval to express the uncertainty of that estimate. This shifts the focus from statistical significance to practical significance.
This paper, "The New Statistics for Neuroscience Majors," provides an excellent introduction to this mindset.
The New Statistics for Neuroscience Majors: Thinking in Effect Sizes
This paper by Calin-Jageman and Cumming argues for a shift in statistical practice toward reporting and interpreting effect sizes.
Please read the section "What is an effect size?". Pay close attention to the example of Prozac in a mouse model. The finding was statistically significant (p = 0.05), but the 95% CI for the effect size was [-80s, -0.008s]. This highlights a crucial point: a "significant" p-value can mask profound uncertainty about the actual magnitude and practical relevance of the effect. Then, continue reading the box that provides examples of effect sizes for common tests like t-tests and factorial ANOVAs. This connects the abstract concept to specific statistical applications.
The key takeaway is that an effect size and its confidence interval provide a much richer, more transparent picture of a study's finding than a p-value alone.
The Challenge of Power and Publication Bias
To estimate an effect with reasonable precision, a study must be adequately powered. Statistical power, as you know, is the probability of detecting an effect of a certain size, given that it truly exists. A conventional target for power in many fields is 80%.
This short video from the NIH provides a solid primer on the fundamental concepts of power and the factors that influence it.
Sample Size and Power: Fundamentals Part 1
This video, "Sample Size and Power: Fundamentals Part 1," explains the core components of power calculations from a biomedical research perspective.
First, watch the introductory segment which defines power and discusses the ethical implications of conducting underpowered research. Next, watch the portion that details the factors affecting power, such as sample size, variance, significance level, and the magnitude of the effect. Finally, observe the concrete examples showing how sensitive power calculations are to the initial assumptions about standard deviation and the expected effect. This demonstrates the fragility of power estimates in practice.
Unfortunately, many published studies, particularly in psychology and neuroscience, are severely underpowered. This has two pernicious consequences:
- Low Precision: Results from underpowered studies have wide confidence intervals and are highly variable from sample to sample.
- Inflated Effect Sizes: Because only studies that happen to find a "statistically significant" result tend to get published (the "file drawer problem"), the literature becomes skewed. Small studies that, by chance, find a large effect are published, while those that find a small or null effect are not. This creates a systematically inflated view of the true effect sizes.
The image below, from a 2018 study in Frontiers in Psychology, provides a stunning visualization of this problem.
This histogram compares the distribution of reported effect sizes (Pearson's r) for studies with and without pre-registration. The blue bars (without pre-registration) show a larger mean effect size and a suspicious "bump" of results just over the typical threshold for statistical significance. The green bars (with pre-registration) show a much more conservative distribution with a smaller mean effect.
This graph strongly suggests that the published literature you read may be presenting an overly optimistic picture of many effects. The shift towards pre-registration, where researchers commit to their hypothesis and analysis plan before collecting data, is a key institutional reform aimed at combating this bias.
Common Designs and Analytical Approaches in Neuroscience
Now let's turn to the types of experimental designs you will frequently encounter when reading this literature. While you are familiar with the underlying statistical principles, their specific application in cognitive science is worth noting.
Factorial Designs
Researchers often want to understand how two or more independent variables (factors) interact. For this, they use factorial designs. A 2x2 factorial design, for example, involves two factors, each with two levels, resulting in four experimental conditions. This allows for the assessment of not only the main effect of each factor but also their interaction effect.
This video provides an exceptionally clear explanation of this common and powerful design.
How Factorial Design Works | NEJM Evidence
"How Factorial Design Works" from the NEJM Group uses a simple analogy to explain this essential experimental design.
First, watch the explanation of a 2x2 factorial design in a clinical trial context. This defines the core concepts of factors, levels, and conditions. Next, see how this is applied in the cookie-baking analogy to calculate main effects. Finally, focus on the graphical explanation of interaction effects. The key insight is that an interaction is present when the effect of one factor depends on the level of another factor, visualized as non-parallel lines on a graph.
In the last lesson, we saw that the negative memory effect of task-switching was larger for more conflicting stimuli () than for simple stimuli (). This was a finding from a factorial design that manipulated both task-switching (switch vs. repeat) and stimulus conflict (high vs. low). The difference in effect sizes indicates an interaction between these two factors.
Repeated Measures and Linear Mixed-Effects Models
Most cognitive experiments, like the task-switching paradigms we've discussed, are repeated-measures designs. Each participant completes many trials, and their performance is measured under different conditions. This design is powerful because it controls for individual differences between participants.
However, it introduces statistical complexity. You have at least two sources of random variability: variability between participants and variability between the items or stimuli used. The traditional approach was to run two separate ANOVAs: one averaging across items to test the effect by-participant (F1 analysis), and one averaging across participants to test the effect by-item (F2 analysis).
This approach is problematic. As a statistician, you will appreciate that the modern, superior method is to use a Linear Mixed-Effects Model (LMM). An LMM allows you to model both participants and items as random effects simultaneously, providing a single, more accurate estimate of the fixed effect of interest.
The following paper provides a superb (and commendably technical) tutorial on this topic. It reveals a critical statistical artifact that you are perfectly positioned to understand.
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
This tutorial by Brysbaert and Stevens is aimed at cognitive psychologists and explains the move from traditional analyses to mixed-effects models.
First, review the section on Traditional F1 and F2 analysis. Note the problems it raises: differing means, no single significance measure, and—most importantly—differing effect sizes for the F1 and F2 analyses. Next, read about the LMM approach. The R code and explanation of random effects (intercepts and slopes for both participant and item) will be familiar territory. Now, read the most crucial section, which discusses effect size calculation. The key insight is in Table 3: the standardized effect size (Cohen's d) calculated from an F1 or F2 analysis is artificially inflated by the number of observations being averaged over. The d calculated from the variance components of an LMM is a more "pure" and much smaller estimate of the true effect size. This is a vital point for interpreting meta-analyses and published effect sizes. Finally, read the main recommendations in the first two bullet points of the conclusion. They recommend a minimum of 1,600 observations per condition to reliably detect a typical 15 ms effect in reaction time studies. This underscores the challenge of conducting well-powered research in this field.
Conclusion
In this lesson, we have taken a critical look under the hood of cognitive neuroscience research. We've moved beyond simply accepting findings at face value to understanding the statistical machinery that produces them, along with its inherent limitations and biases.
Key Takeaways:
- Focus on Estimation: Prioritize effect sizes and their confidence intervals over p-values to understand the practical magnitude and uncertainty of a finding.
- Maintain Healthy Skepticism: Be aware that the published literature likely contains inflated effect sizes due to a historical prevalence of underpowered studies and publication bias. Pre-registered studies generally offer more reliable estimates.
- Recognize the Designs: Factorial designs are used to test for interactions between variables, while repeated-measures designs are the workhorse for studying within-participant cognitive processes.
- Appreciate Modern Methods: Linear Mixed-Effects Models (LMMs) are the standard for analyzing repeated-measures data, providing more accurate effect size estimates than older F1/F2 analyses.
- Power is Demanding: Detecting the typically small effects in cognitive research requires a large number of observations. A study with 40 participants and 40 trials might seem substantial, but as we saw, it can still be severely underpowered.
You are now in a much stronger position to critically evaluate the evidence we will encounter throughout the rest of this course. In our next lesson, we will conclude this first module by creating the "cheat sheet" you requested, summarizing the core principles of attention we have covered so far.