Measures of Information via Representation Learning
Abstract
Abstract - Approved for Public Release Title- Measures of Information via Representation Learning Topic Number Addressed- Topic 14 - Machine Learning, Reasoning, and Intelligence. Program Officer Contacted about Proposed Work- Dr. Behzad Kamgar-Parsi, ONR PI- Dr Luis Gonzalo Sanchez Giraldo IHE- University of Kentucky The use of information theoretic quantities as objective functions in machine learning has been limited by a lack of robust and scalable estimators. Our view is that the fundamental problem is the quantities themselves. Here, we propose novel information theoretic quantities, with similar properties as the traditional quantities, that break through this barrier. This work will develop the theoretical foundations of alternative definitions of entropy and mutual information to address a fundamental problem in Machine Learning- building objective functions that can handle and integrate data from multiple sources with minimal supervision. Traditional paradigms for mutual information and entropy estimation either do not scale well with dimensionality or are difficult to use for optimization. Recently, estimators based on lower bounds of mutual information that use representation learning have been proposed to overcome these limitations. However, it has been shown that even these estimators underperform under high entropy, high mutual information scenarios requiring exponential sample sizes that are impractical. To address this issue we propose a shift from focusing on developing estimators for conventional information theoretic quantities to using quantities that expose similar properties to conventional information theoretic quantities but that lend themselves to reliable estimators. In order to explore this theoretical shift we propose to- 1) formulate the properties of the alternative definitions of entropy and mutual information, and establish similarities and differences with their conventional counterparts, 2) identify what properties are relevant when these quantities are used as objective functions, and 3) provide empirical estimators based on representation learning and study their convergence. While the proposed approach is basic in nature, we expect it to be applicable across disciplines and domains of interest to the Department of Defense. To ensure our study is comprehensive, we will apply this theory to develop a robust and scalable approach to multimodal data integration.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 07, 2023
- Source ID
- FA95502110227
Entities
People
- Luis Sanchez Giraldo
Organizations
- Air Force Office of Scientific Research
- Office of the Secretary of Defense
- University of Kentucky