How good is an ensemble an capturing truth? Using bounding boxes for forecast evaluation

Quarterly Journal of the Royal Meteorological Society 133 (2007) 1309-1325

K Judd, LA Smith, A Weisheimer

Ensemble prediction systems aim to account for uncertainties of initial conditions and model error. Ensemble forecasting is sometimes viewed as a method of obtaining (objective) probabilistic forecasts. How is one to judge the quality of an ensemble at forecasting a system? The probability that the bounding box of an ensemble captures some target (such as 'truth' in a perfect model scenario) provides new statistics for quantifying the quality of an ensemble prediction system: information that can provide insight all the way from ensemble system design to user decision support. These simple measures clarify basic questions, such as the minimum size of an ensemble. To illustrate their utility, bounding boxes are used in the imperfect model context to quantify the differences between ensemble forecasting with a stochastic model ensemble prediction system and a deterministic model prediction system. Examining forecasts via their bounding box statistics provides an illustration of how adding stochastic terms to an imperfect model may improve forecasts even when the underlying system is deterministic. Copyright © 2007 Royal Meteorological Society.

Dynamically-based seasonal forecasts of Atlantic tropical storm activity issued in June by EUROSIP

Geophysical Research Letters 34 (2007)

F Vitart, MR Huddleston, M Déqué, D Peake, TN Palmer, TN Stockdale, MK Davey, S Ineson, A Weisheimer

Most seasonal forecasts of Atlantic tropical storm numbers are produced using statistical-empirical models. However, forecasts can also be made using numerical models which encode the laws of physics, here referred to as "dynamical models". Based on 12 years of re-forecasts and 2 years of real-time forecasts, we show that the so-called EUROSIP (EUROpean Seasonal to Inter-annual Prediction) multi-model ensemble of coupled ocean atmosphere models has substantial skill in probabilistic prediction of the number of Atlantic tropical storms. The EUROSIP real-time forecasts correctly distinguished between the exceptional year of 2005 and the average hurricane year of 2006. These results have implications for the reliability of climate change predictions of tropical cyclone activity using similar dynamically-based coupled ocean-atmosphere models.

Daily Forecast Skill of Multi-Center Grand Ensemble

SOLA 3 (2007) 29-32

M Matsueda, M Kyouda, HL Tanaka, T Tsuyuki

Ensemble decadal predictions from analysed initial conditions.

Philos Trans A Math Phys Eng Sci 365 (2007) 2179-2191

A Troccoli, TN Palmer

Sensitivity experiments using a coupled model initialized from analysed atmospheric and oceanic observations are used to investigate the potential for interannual-to-decadal predictability. The potential for extending seasonal predictions to longer time scales is explored using the same coupled model configuration and initialization procedure as used for seasonal prediction. It is found that, despite model drift, climatic signals on interannual-to-decadal time scales appear to be detectable. Two climatic states have been chosen: one starting in 1965, i.e. ahead of a period of global cooling, and the other in 1994, ahead of a period of global warming. The impact of initial conditions and of the different levels of greenhouse gases are isolated in order to gain insights into the source of predictability.

Convective forcing fluctuations in a cloud-resolving model: Relevance to the stochastic parameterization problem

JOURNAL OF CLIMATE 20 (2007) 187-202

GJ Shutts, TN Palmer

Initialisation strategies for decadal hindcasts for the 1960-2005 period within the ENSEMBLES project. ECMWF Tech Memo.

(2007) 521

A Weisheimer, FJ Doblas-Reyes, P Rogel, N Keenlyside, MA Balmaseda, J Murphy, D Smith, M Collins, B Bhaskaran, TN Palmer

Seasonal Forecast Datasets - A resource for Calibrating Regional Climate Change Projections?

CLIVAR Exchanges 43 (2007) 6-7

TN Palmer, FJ Doblas-Reyes, A Weisheimer, M Rodwell

Multi-center grand ensemble using three operational ensemble forecasts

SOLA 2 (2006) 33-36


Changing frequency of occurrence of extreme seasonal temperatures under global warming (vol 32, art no L20721, 2005)


A Weisheimer, TN Palmer

Developments in dynamical seasonal forecasting relevant to agricultural management

CLIMATE RESEARCH 33 (2006) 19-26

FJ Doblas-Reyes, R Hagedorn, TN Palmer

Impact of increasing greenhouse gas concentrations in seasonal ensemble forecasts


FJ Doblas-Reyes, R Hagedorn, TN Palmer, JJ Morcrette

Malaria early warnings based on seasonal climate forecasts from multi-model ensembles

NATURE 439 (2006) 576-579

MC Thomson, FJ Doblas-Reyes, SJ Mason, R Hagedorn, SJ Connor, T Phindela, AP Morse, TN Palmer

Multi-Center Grand Ensemble using Three Operational Ensemble Forecasts

SOLA Meteorological Society of Japan 2 (2006) 33-36

M Matsueda, M Kyouda, HL Tanaka, T Tsuyuki

In this study, we investigate the impact of Multi-Center Grand Ensemble (MCGE) forecasts, consisting of three operational ensemble forecasts by the Japan Meteorological Agency (JMA), the National Centers for Environmental Prediction, and the Canadian Meteorological Center. We verified the skill of MCGE forecasts in comparison with that of JMA ensemble forecast using root mean square error, anomaly correlation, and Brier skill score for 500 hPa geopotential height and 850 hPa temperature in the Northern Hemisphere (20°N-90°N) in September 2005.<br>Our results show that MCGE forecasts are more skillful than single-center ensemble forecast without considering weight among ensemble members and bias corrections. This implies that considering weight or bias corrections may result in further improvement of MCGE forecasts, specifically in probabilistic forecasts.

Erratum: "Changing frequency of occurrence of extreme seasonal temperatures under global warming" (Geophysical Research Letters (2005) vol. 32 10.1029/2005GL023365)

Geophysical Research Letters 33 (2006)

A Weisheimer, TN Palmer

A new view of seasonal forecast skill: Bounding boxes from the DEMETER ensemble forecasts

Tellus, Series A: Dynamic Meteorology and Oceanography 57 (2005) 265-279

A Weisheimer, LA Smith, K Judd

Insight into the likely weather several months in advance would be of great economic and societal value. The DEMETER project has made coordinated multi-model, multi-initial-condition simulations of the global weather as observed over the last 40 years; transforming these model simulations into forecasts is non-trivial. One approach is to extract merely a single forecast (e.g. best-first-guess) designed to minimize some measure of forecast error. A second approach would be to construct a full probability forecast. This paper explores a third option, namely to see how often this collection of simulations can be said to capture the target value, in the sense that the target lies within the bounding box of the forecasts. The DEMETER forecast system is shown to often capture the 2-m temperature target in this sense over continental areas at lead times up to six months. The target is captured over 95% of the time at over a third of the grid points and maintains a bounding box range less than that of the local climatology. Such information is of immediate value from a user's perspective. Implications for the minimum ensemble size as well as open foundational issues in translating a set of multi-model multi-initial-condition simulations into a forecast are discussed; in particular, those involving 'bias correction' are consider. Copyright © Blackwell Munksgaard, 2005.

Influence of a stochastic parameterization on the frequency of occurrence of North Pacific weather regimes in the ECMWF model


T Jung, TN Palmer, GJ Shutts

Representing model uncertainty in weather and climate prediction


TN Palmer, GJ Shutts, R Hagedorn, E Doblas-Reyes, T Jung, M Leutbecher

Recurrent climate winter regimes in reconstructed and modelled 500 hPa geopotential height fields over the North Atlantic/European sector 1659-1990

CLIMATE DYNAMICS 24 (2005) 809-822

C Casty, D Handorf, CC Raible, JF Gonzalez-Rouco, A Weisheimer, E Xoplaki, J Luterbacher, K Dethloff, H Wanner

Probabilistic prediction of climate using multi-model ensembles: from basics to applications.

Philos Trans R Soc Lond B Biol Sci 360 (2005) 1991-1998

TN Palmer, FJ Doblas-Reyes, R Hagedorn, A Weisheimer

The development of multi-model ensembles for reliable predictions of inter-annual climate fluctuations and climate change, and their application to health, agronomy and water management, are discussed.

Changing frequency of occurrence of extreme seasonal temperatures under global warming

Geophysical Research Letters 32 (2005) 1-5

A Weisheimer, TN Palmer

Using a multi-model multi-scenario ensemble of integrations made for the forthcoming fourth assessment report of the Intergovernmental Panel on Climate Change, the frequency of occurrence of extreme seasonal temperatures at the end of the 21st Century is estimated. In this study an extreme temperature is defined as lying above the 95 percentile of the simulated temperature distribution for 20th Century climate. The model probability of extreme warm seasons is heterogeneous over the globe and rises to over 90% in large parts of the tropics. This would correspond to an average return period of such anomalous warm seasons of almost one year. The reliability of these results is assessed using the bounding box technique, previously used to quantify the reliability of seasonal climate forecasts. It is shown that the dramatic increase in extreme warm seasons arises from the combined effect of a shift and a broadening of the temperature distributions. Copyright 2005 by the American Geophysical Union.