Properties of Cross-Entropy Minimization.
Abstract
The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) provides a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values. Previous work has shown that cross-entropy minimization can be characterized axiomatically by a small set of consistency axioms. Our purpose in this paper is to collect in one place and prove various fundamental properties of cross-entropy minimization. We include examples, and we discuss general analytic and computational methods of finding minimum cross-entropy probability densities. An interesting aspect of the results presented in this paper is the interplay between properties of cross-entropy minimization as an inference procedure and properties of cross-entropy as an information measure. Cross-entropy's well-known and unique properties as an information measure in the case of arbitrary probability densities are extended and strengthened when one of the densities involved is the result of cross-entropy minimization. For example, cross-entropy does not in general satisfy a triangle relation involving three arbitrary probability densities. But in certain important cases involving densities that result from cross-entropy minimization, cross-entropy satisfies triangle inequalities and triangle equalities.
Document Details
- Document Type
- Technical Report
- Publication Date
- Apr 01, 1980
- Accession Number
- ADA085471
Entities
People
- J. E. Shore
- Wayne Johnson
Organizations
- United States Naval Research Laboratory