Deutsch: From Falsifiability to Good Explanations

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to the first lesson in our third module, "The Deutschian Turn: The Power of Good Explanations."

In the first two modules, we explored Karl Popper's critical rationalism, from his solution to the problem of induction to his vision of an "Open Society" built on piecemeal reform. We saw how falsifiability provides a powerful criterion for demarcating science from non-science and how policy itself can be treated as a testable hypothesis.

Today, we take a significant step forward with the physicist and philosopher David Deutsch. While working firmly within the Popperian tradition, Deutsch argues that Popper's focus on falsification, while crucial, doesn't capture the full story of scientific progress. He proposes a subtle but profound shift in emphasis: from mere falsifiability to the search for good explanations.

This lesson will explore that very shift. We will examine why falsifiability alone isn't enough and how Deutsch's focus on the quality of our conjectures provides a more robust account of how we generate knowledge. This will lay the groundwork for understanding the principles that distinguish powerful scientific theories from plausible-sounding myths.

Time to complete: Approximately 50 minutes.

A Quick Recap

In our first module, we established Popper's evolutionary view of knowledge:
$P_{1} \to TT \to EE \to P_{2}$
A problem ( $P_{1}$ ) leads to a Tentative Theory ( $TT$ ), which is then subjected to Error Elimination ( $EE$ )—criticism and testing—leading to a new, often deeper, problem ( $P_{2}$ ). The key to this process being scientific was that the tentative theory had to be falsifiable. But what if a theory is falsifiable, yet fundamentally flawed in its structure? This is the question that motivates Deutsch's contribution.

1. The Limits of Falsifiability

Popper's great insight was that we learn from our mistakes. A theory is scientific if it makes bold claims about reality that could, in principle, be proven wrong by observation. However, is testability the only thing that matters?

David Deutsch argues that it is not. Many theories, including ancient myths and modern pseudo-science, are perfectly testable. If simply being testable were the hallmark of a good theory, we would waste our time on all sorts of dead ends.

To see why, please watch this segment from David Deutsch's TED talk. He uses the ancient Greek myth of Persephone to illustrate how a theory can be testable, and even be falsified, without leading to any real progress.

Watch from 05:58 to 11:04. Focus on his argument about why the Persephone myth is testable but is still a terrible theory.

As Deutsch explains, the myth that seasons are caused by the goddess Demeter's sadness is testable. If her sadness causes winter, winter should occur everywhere on Earth simultaneously. Discovering that Australia has summer while the Northern Hemisphere has winter would falsify this theory.

However, the problem is that the theory is trivially easy to fix. A myth-maker could simply add a new rule: "When Demeter is sad, she banishes heat to the other side of the world." The theory now accounts for the new evidence, but are we any closer to the truth? Not at all.

This reveals the core issue: the details of the myth are arbitrary. They are not constrained by anything other than the storyteller's imagination. This "easy variability" is the signature of a bad explanation.

This should remind you of some concepts from your background in statistics. Think of a model that overfits the data. A polynomial of a very high degree can be made to pass through every single data point in a sample. It is "testable" against the data and performs perfectly. Yet, it's a poor explanation of the underlying process because it has no predictive power for new data. Its coefficients are, in a sense, "easy to vary" to fit any dataset, just as the details of the Persephone myth are easy to vary to fit new observations. The model explains nothing; it merely fits.

2. The Shift: The Search for "Good Explanations"

If testability isn't enough, what is the crucial ingredient for progress? Deutsch's answer is that we must seek good explanations. A good explanation is one that is hard to vary.

Let's continue with Deutsch's TED talk, where he contrasts the "easy-to-vary" myth with our modern, "hard-to-vary" scientific explanation for the seasons.

Now watch from 11:04 to 14:53. Pay close attention to why the axial-tilt theory of seasons is considered a good explanation.

The modern explanation is that the Earth's axis is tilted relative to its orbital plane. This is a good, hard-to-vary explanation because every detail is constrained and plays a functional role:

The tilt itself is a specific assertion.
We know from basic physics that a spinning object in space (like Earth) maintains a stable axis of rotation.
We also know, independently, that surfaces tilted away from a source of radiant heat receive less energy.
This single, simple postulate (the tilt) not only explains why it's colder in winter but also explains why the seasons are out of phase in the two hemispheres and why the sun is lower in the sky during winter.

You can't just change one detail of this explanation without wrecking its ability to account for all these other phenomena. If an observation had contradicted it (e.g., seasons being in-phase globally), the theory would be in deep trouble. There's no easy patch. This is the strength of a good explanation.

The revolutionary shift, then, is this: The engine of scientific progress is not just testing theories, but the creative act of conjecturing hard-to-vary explanations that are then subjected to testing. The quality of the conjecture comes first. It's the difference between guessing "a wizard did it" and proposing a mechanism constrained by the laws of physics.

3. Explanations vs. Predictions

This focus on explanation has profound implications. It sets science apart from mere prediction or curve-fitting. In your work with financial and econometric models, you've likely encountered the distinction between a structural model (which explains why) and a purely statistical or machine learning model (which only predicts what). Deutsch's philosophy places enormous value on the former.

An "explanationless" theory might observe a statistical trend and predict its continuation, but it offers no understanding of the underlying causes. Deutsch is scathing about such approaches.

Please watch the conclusion of his TED talk, followed by a short clip from an interview with Sean Carroll where he expands on this.

First, watch the final part of the TED talk:

Watch from 14:53 to 16:36.

Now, watch this clip from the Mindscape podcast:

Watch from 00:32:51 to 00:35:31.

Deutsch's warning against "explanationless theories" that extrapolate statistical trends is a direct critique of naive empiricism. Simply saying "the economy grew at 2% for five years, so it will grow at 2% next year" is, in his view, no better than saying a wizard is responsible. A good economic theory, by contrast, would propose a hard-to-vary explanation based on capital, labour, technology, and institutions.

Furthermore, as he notes in the Mindscape clip, scientists are not seeking the most probable theory, a point that is crucial for someone with your background in Bayesian methods. A bold, simple, and deeply explanatory theory might, from a Bayesian perspective, have a very low prior probability. A more complex, ad-hoc theory that wiggles to fit all the current data might seem "more probable." Yet, science progresses by preferring the former. Galileo's heliocentric model was a far better explanation than the convoluted Ptolemaic system of epicycles, even if one could argue the latter was better "tuned" to the existing data of the time.

The search is for the best explanation, not the most probable one.

Conclusion

In this lesson, we have explored David Deutsch's crucial refinement of Popperian epistemology. You should now be able to explain the shift he proposes.

Key Takeaways:

Falsifiability is necessary but not sufficient for scientific progress. Many bad ideas, like myths, are easily testable.
The flaw in bad theories is that they are "easy to vary." Their details are arbitrary and can be patched ad-hoc to fit new evidence without yielding real understanding.
Deutsch shifts the focus to the conjecture itself, arguing that progress comes from the search for "good explanations," which are "hard to vary."
In a good explanation, the details are tightly constrained by logic and other established knowledge, playing a functional and non-arbitrary role.
This framework prioritizes deep explanation over mere prediction, correlation, or probabilistic confirmation, challenging both naive empiricism and some interpretations of Bayesian epistemology.

Next Steps

We've established why the search for good explanations is paramount. In our next lesson, we will delve into the mechanics. We will formally define Deutsch's criteria for a good explanation: being "hard to vary" and possessing "reach." This will give us a practical toolkit for evaluating the quality of theories, not just in science, but in all domains of knowledge.