A Reputation System for Uncertain Assertions

Abstract

We investigate reputation systems that rate the performance of analysts who make uncertain assertions (claims accompanied by estimated probabilities). Accuracy metrics (based on the fraction correct) are fair only if all analysts handle identical or statistically similar cases. Furthermore, accuracy metrics discourage analysts from offering predictions on difficult-to-predict events. Because of these difficulties, we develop a class of performance scoring functions that are maximized when the analyst provides accurate probabilities, especially when these probabilities differ from the norm. Under these metrics, the disincentives to forecast low-probability events is removed and analysts are rated fairly, independent of the base event probabilities of the cases they consider. Reputation systems built around these metrics can support productivity management and increase manipulation resistance when information providers are not trustworthy. An application to citizen event reporting is presented.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Jan 01, 2009
Accession Number
AD1107437

Entities

People

  • Arnon Rosenthal
  • Mark D. Kramer

Organizations

  • MITRE Corporation

Tags

Communities of Interest

  • Biomedical
  • Energy and Power Technologies
  • Human Systems
  • Weapons Technologies

DTIC Thesaurus Topics

  • Accuracy
  • Agent-Based Simulations
  • Asymmetric Warfare
  • Consumers
  • Differential Equations
  • Economic Forecasting
  • Errors
  • Explosive Devices
  • False Alarms
  • First Responders
  • Improvised Explosive Devices
  • Mathematical Analysis
  • Normal Distribution
  • Personnel Management
  • Simulations
  • Standards
  • Workload

Fields of Study

  • Computer science

Readers

  • Computational Modeling and Simulation
  • Strategic Security Studies