Strictly Proper Scoring Rules, Prediction and Estimation
Abstract
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the forecast and on the event or value that materializes. A scoring rule is strictly proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if she issues the probabilistic forecast F, rather than any G not equal F. In prediction problems, strictly proper scoring rules encourage the forecaster to make careful assessments and to be honest. In estimation problems, strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the scientific problem at hand. This paper characterizes strictly proper scoring rules on general probability spaces, and proposes and discusses examples of such. The continuous ranked probability score applies to probabilistic forecasts that take the form of predictive cumulative distribution functions; it generalizes the absolute error and forms a special case of a new and very general type of score, the energy score. Proper scoring rules for quantile and interval forecasts are also discussed. We relate proper scoring rules to Bayes factors and to cross-validation, and show that a particular form of cross-validation, random-fold cross-validated likelihood, corresponds to a proper scoring rule. This also allows us to define proper scoring rules when parameters defining the rule are estimated from the data. A case study on probabilistic weather forecasts in the North American Pacific Northwest illustrates the importance of strict propriety. Optimum score approaches to point estimation are noted, and the intuitively appealing interval score is proposed as a utility function in interval estimation that addresses width as well as coverage.
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 01, 2004
- Accession Number
- ADA459827
Entities
People
- Adrian Raftery
- Tilmann Gneiting
Organizations
- University of Washington