Validating Climate Models

Hello. The previous lesson showed how a climate model combines resolved motion, parameterized processes, and exchanges among the atmosphere, ocean, land and ice. The next question is empirical: how do we find out whether that combination works well enough for a particular purpose?

In this lesson, you will examine three complementary tests: simulations of the observed historical period, simulations of much older climates, and comparisons across independently developed models. The aim is not to award a model a single “validated” stamp. It is to establish what a comparison tests, what it cannot test, and how much confidence its result warrants.

Historical simulations: more than matching a temperature curve

A historical hindcast runs a model over a period for which observations already exist. Modellers supply estimates of influences over that period—such as greenhouse-gas concentrations, volcanic eruptions, solar changes and aerosols—and compare the simulated climate with measured climate. This tests the model’s response to supplied influences. It does not test whether the model could have anticipated those influences unaided.

The World Climate Research Programme’s short introduction explains why the Coupled Model Intercomparison Project (CMIP) is useful here. Participating groups run specified experiments and provide comparable outputs, making systematic evaluation possible.

What should that examination compare? Global mean surface temperature is one target, but a model might reproduce its overall warming trend while getting ocean heat uptake, rainfall patterns or seasonal cycles wrong. A useful evaluation therefore specifies the variable, region, time scale and physical process relevant to the question. For a rainfall projection, for example, reproducing historical rainfall matters; so does representing the circulation that produces it.

Read the following review with that distinction in mind. Its performance-oriented tests ask how closely output matches observations; its discussion of compensating errors explains why a close match is not always sufficient.

Comparability matters as much as the metric. A satellite may observe a quantity related to, but not identical to, a model variable; an instrument record may cover only part of a model grid cell. Comparing them without accounting for those differences can make a model look better or worse for the wrong reason. Observational estimates also carry uncertainty. A discrepancy is evidence to investigate, not automatically a measurement of model error alone.

What a hindcast can and cannot establish

Consider this evaluation plot. Its two panels display the same comparison in different temperature units, not two independent tests.

The rising ensemble mean tracks the broad observed rise, while the observations fluctuate from year to year. This supports a claim about the modelled long-term temperature response over the period shown. It does not establish that the models reproduced the causes or timing of every short-term fluctuation, nor that their cloud and ocean processes are all correct.

Notice also the distinction between a historical fit and a stronger prospective test. The figure is titled “models run in 2004,” but its dashed hindcast–forecast divider sits around 2000. The divider alone does not show which observations were unavailable when the models were constructed or evaluated. To assess a genuinely out-of-sample prediction, you would need to establish what model version and inputs were fixed, when they were fixed, and which subsequent observations were held back.

There is a statistical reason not to demand that an uninitialized climate simulation trace every annual wiggle. Even with the same external influences, different initial ocean and atmospheric states can yield different sequences of internally generated variability. Schematically, for model $m$ and run $r$,

where $F$ denotes the supplied influences, ${\mu }_m$ the model’s response averaged over internal variability, and ${\epsilon }_{m,r}$ one run’s departure from that response. Agreement in a multi-decade trend is consequently a different test from agreement in the exact year of a fluctuation.

Finally, a band labelled “95% CI” should not be read automatically as “a 95% probability that the real world’s temperature lies here.” Its interpretation depends on what generated the band—different initial conditions, different models, another calculation—and whether the resulting uncertainty estimate is calibrated against observations. A plotted range is useful evidence, but its label is not a substitute for its construction method.

Paleoclimate: testing beyond the instrumental record

Historical measurements cover a limited range of climate conditions. Paleoclimate simulations test models under conditions more different from the present. A prominent example is the Last Glacial Maximum, roughly 21,000 years ago, when greenhouse-gas concentrations were lower and large ice sheets existed. Other coordinated experiments examine the mid-Holocene and the last millennium. These are not attempts to reproduce a particular ancient day’s weather: they test climate states, changes and spatial patterns against reconstructions.

The logic resembles an out-of-sample test, with important qualifications. If a model developed principally using modern observations also captures an ancient climate under independently specified past conditions, that adds evidence that its mechanisms work outside the modern range. But the ancient “observations” are usually proxies—such as measurements from ice or sediments—not complete global temperature maps. The model–data comparison must respect what each proxy measures, where it measures it, and its uncertainty.

The distinction between model error and test-data error is especially consequential here. Suppose a simulation and a proxy-based temperature reconstruction disagree. Possible contributors include the model’s climate response, uncertainty in the ancient greenhouse-gas or ice-sheet inputs, a proxy influenced by something besides temperature, or a mismatch between a local proxy and a large model grid cell. Long coupled simulations can also drift if the deep ocean has not been adequately brought into balance before the comparison begins.

These difficulties do not make paleoclimate evidence dispensable. They define the conditions for using it well: compare like with like, report reconstruction and input uncertainties, and seek consistency across multiple quantities and periods. Equally, a good match to an ancient mean temperature does not by itself establish a precise future climate sensitivity. For that stronger inference, the aspect of past performance being measured must have a defensible physical relationship to the future response.

Why compare multiple models?

One model can be rerun from different initial states. That produces an initial-condition ensemble, useful for seeing how much the timing and size of fluctuations vary within that model. CMIP also enables a multi-model ensemble: outputs from models with different numerical choices, representations of processes and development histories. The latter comparison exposes some uncertainty about model construction that repeated runs of a single model cannot reveal.

Read the review’s treatment of both the advantage and the statistical limitation of this approach.

A multi-model mean can be informative: model-specific errors may partly cancel, making a shared large-scale response easier to see. But agreement among models is not independent confirmation if they share assumptions, code or biases. CMIP is an ensemble of opportunity: participating models are not a random, independent draw from a known distribution of all plausible models. Its raw spread is therefore not automatically a probability distribution over future climates, and its mean need not be the best result for every region or process.

A careful ensemble evaluation asks which models reproduce the historical or paleoclimate feature relevant to the question; whether related models are supplying nearly duplicate information; and whether their apparent success survives examination of the underlying processes. It also keeps distinct three sources of uncertainty: internal variability, differences among models, and, for future results, differences among assumed future influences. Combining all three into one unexplained shaded band would conceal what has actually been tested.

Putting the evidence together

Model validation is best understood as a portfolio of conditional tests. Historical hindcasts ask whether simulations driven by estimated past influences reproduce observed trends, patterns and variability. Paleoclimate comparisons ask whether models represent climate under substantially different conditions, subject to proxy and past-input uncertainties. CMIP supplies common experiments and multi-model comparisons that reveal both shared behaviour and disagreements. Process checks guard against a model obtaining the right aggregate result for the wrong reasons.

Each successful test earns a specific conclusion, not a universal one: skill at global mean temperature does not guarantee skill at regional rainfall, and neither guarantees the precise timing of an individual event. Conversely, a model’s failure to match one year’s fluctuation need not invalidate its simulated long-term response.

Next, we will sharpen that last distinction by separating a climate projection, conditional on specified future influences, from a weather forecast, which depends critically on the initial state.