Learning with Optimal Labels
Abstract
The key achievements of the STR team were: We developed of explicit and useful lower and upper risk bounds for ML models (including complex DNNs developed by TA1 performers) for many supervised ML tasks: Image Classification, Object Detection, and Video Classification. We developed of novel geometric formulation of Transfer Learning and construction of novel upper bounds for excess transfer risk. We developed formalism for understanding precise tradeoff between regularization and inference for ML tasks with label constraints. Our results are generally applicable they are valid for any ML model including complex deep neural networks. We provided convincing proof of this in the three evaluation tasks over the course of the program. While our results provide significant improvements to the state of the art, we would also like to point out some limitations. These limitations apply to all current theoretical approaches to machine learning (including ours), so the questions we lay out are open questions for the whole field. The main limitation is that all current results are based on Statistical Learning Theory, which has intrinsic limitations for out-of-distribution generalization. Our opinion is that one really needs some neuro-symbolic approach to make seminal advances in leveraging prior out-of-distribution (but semantically related) information. Symbolic information is key, it could be logic-based, knowledge-based or physics/dynamics based or a combination of these. Understanding how to combine neural and symbolic approaches will ultimately lead to data/label efficient models (similar to what is done in the human brain).
Document Details
- Document Type
- Technical Report
- Publication Date
- Jul 01, 2023
- Accession Number
- AD1205584
Entities
People
- Piyush Kumar
- Spencer Douglas