Active meta learning
Abstract
The success of machine learning models hinges on whether an appropriate learning algorithm was selected for the task of interest. Meta-learning algorithms asses the complexity of the training data to help select an appropriate learning model for solving the problem at hand. For instance, in the context of classification, the complexity of the problem can be characterized by understanding two complementary meta-features: (a) the amount of overlap between classes, and (b) the geometry of the decision boundary. The practitioner typically uses a large amount of training data to assess these meta-features, and then selects the model class that best matches the data. However, this scenario is arguably more appropriate in settings where one has access to large corpora of carefully curated and labeled training data that reasonably captures the complexity of the target task. On the other hand, active learning has been proposed as a promising approach to dealing with the paucity in curated training data here the learner sequentially and adaptively selects the data points whose labels would be most informative in the learning process. While this family of techniques is promising for defense applications where the labeled data is expensive to obtain, it requires the declaration of a family of models (or hypotheses) before even seeing the data, and therefore runs the risk of not being adaptable to dynamic environments.In this proposal we retain the virtues of meta-learning for algorithm design but extendits applicability to applications where labeled data is expensive but can be actively acquired. We leverage a combination of results from previous work done by the PIs and propose a new framework active meta learning. This new approach provides an active, non-parametric approach to characterizing the amount of overlap between classes and the geometry of the decision boundary in data. With active meta learning we iteratively label points that are maximally informative for characterizing the complexity of the classification task. This enables new practical applications related to model selection, particularly for situations where data labels are expensive. Our preliminary results demonstrate that active meta learning can help recover the geometry of the decision boundary with 50% fewer labeled exemplars. Furthermore models selected using this novel paradigm yield lower error rates and significantly better estimates of classification uncertainty.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jun 09, 2021
- Source ID
- N000142112615
Entities
People
- Visar Berisha
Organizations
- Arizona State University
- Office of Naval Research
- United States Navy