Assessing Probability Assessors: Calibration and Refinement.
Abstract
For well-calibrated multivariate forecasters, we can define the concept of refinement by means of a multivariate stochastic transformation. Moreover, this notion of refinement can again be directly linked to sufficiency in the comparison of experiments with a finite number of outcomes. Finally, the concept of one forecaster being marginally or conditionally more refined than another can be developed. Critical to the multivariate versions of calibration and refinement as proposed in this section is the orientation of the vector of forecasted probabilities x. Each component of x refers to a specific outcome. This methodology should be contrasted with the multivariate approach, described for example by Lichtenstein, Fischhoff, and Phillips (1977), in which the forecaster 'selects the single most likely alternative and states the probability that it is correct.' Kadane and Lichtenstein (1981) show that such a loss of orientation leads to the inability to recalibrate a forecaster's assessments. From the discussion here, it should be clear that a careful description of calibration and refinement in both the binary and multivariate settings requires a well-specified set of outcomes, and probability assessments specifically tied to those outcomes. (Author)
Document Details
- Document Type
- Technical Report
- Publication Date
- Jul 01, 1981
- Accession Number
- ADA104174
Entities
People
- Morris H. Degroot
- Stephen E. Fienberg
Organizations
- Carnegie Mellon University