Exploring Contingency Skill Scores Based on Event Sizes
Abstract
Space weather forecasts are generally made for events with an arbitrary size threshold imposed on an event statistical size distribution which is likely described by a power law. This is the case for solar energetic (E > 10 MeV) particle (SEP) events, which have a differential power law exponent of γ = 1.2. Event forecasts are usually evaluated by skill scores using a contingency table that matches the forecasted events against observed events independently of the event sizes. Each observed event is either a forecasted hit or a miss, and each forecasted event is either an observed hit or a false alarm. However, for SEP events and most other space weather parameters the event size is a critical factor for the user. It is more important that large events be well forecasted than threshold events. In addition, false alarms may be useful when they match observed events just below the forecast threshold. We explore a forecast evaluation scheme to incorporate the event size within the usual format of a binary contingency table to evaluate model performance. The scheme is applied to three different input options of a recently published evaluation of the Proton Prediction System (PPS) for SEP events to show differences between numbers‐based and intensity‐based skill scores of the PPS. We demonstrate how identical skill scores can result from models with extremely different performances of event intensity forecasts. The scheme requires model validation and would benefit from testing with other space weather applications.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- May 01, 2021
- Source ID
- 10.1029/2020sw002604
Entities
People
- H. Darsey
- S. W. Kahler
Organizations
- Air Force Office of Scientific Research
- Air Force Research Laboratory
- University of New Mexico