Building Autonomously Improving Foundation Models by Learning from Interaction
Abstract
While advances in foundation models # large generalist models, that can then be fine-tuned to downstream tasks # have pushed the frontiers of machine learning, these models are still brittle: it is easy to trick even the best models into producing unsafe outputs;these models hallucinate even in problems that admit an unambiguous answer (e.g., reasoning problems); and more often than not, they fail to accomplish multi-turn, goal-directed tasks. In theory, the current paradigm of training on a finite-capacity model on a finite amount of static data alone should never be sufficient to ensure correctness on any input, in the worst case. In this proposal,I argue that the only scalable approach to address these challenges is to train foundation models to seek and learn from feedback obtained via interaction. This involves making them capable of proposing hypotheses about their own predictions, autonomously interacting with the external world (e.g., with software tools, books, open web, oreven other AI models) to perform experiments to verify them, and learning from the associated (sub-optimal) outcomes by altering their beliefs and hypotheses over time. For example, a foundation model tasked with proving mathematical results may not already know the proof for a given statement, but it must be able to interact with external tools (e.g., the web) for discovering intermediate results that could be useful. Even if the external input obtained is un-curated or not directly useful, the model should try to utilize it to refine its predictions in the best possible way. In this project, we will develop ML techniques to build foundation models that enjoy the above desiderata. Specifically, we will develop a class of reinforcement learning (RL) methods that train foundation models to be capable of interacting with the external world by executing learned strategies for intelligent data collection, and subsequently self-improve by training on the collected interaction data. This class of methods will extend the paradigm of offline RL from static datasets alone, with the capability of intelligent data collection and self-improvement using interaction data. We will organize the project along three thrusts: (1) we will firstdevelop approaches for learning information-seeking strategies from data (i.e., strategies that attempt to first probe and identifythe characteristics of the external tool / world to seek more information to better solve tasks of interest); (2) we will then develop methods for using this uncurated and sub-optimal interaction data for improving the foundation model policy; (3) we will developmethods that allow us to scale techniques from Thrusts (1) and (2) to very large models, aiming to provide a #workflow# for practitioners, prescribing recipes for addressing optimization and computational challenges when scaling up to large models. Since we wantour approaches to be generally helpful in improving several capabilities, our evaluations willattempt to maximize the breadth of domains, including code, mathematical reasoning, theorem proving, and tool use. We will utilize existing benchmarks for these problems, but will set up new tasks as necessary. We will also study vision-language models (VLMs), in the context of interaction in embodied decision making. Anticipated outcome: This project will develop algorithms to make foundation models generally capable of reasoning and correcting their mistakes, by interacting with the external world, which is critical in a plethora of use cases. We anticipatethat our techniques will lead to a paradigm shift in training foundation models: instead of training them via (un)supervised learning objectives like today, our research would prescribe new algorithms for training, so that these models can interact, improve, and subsequently discover new facts about the world.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 15, 2024
- Source ID
- N000142412206
Entities
People
- Aviral Kumar
Organizations
- Carnegie Mellon University
- Office of Naval Research
- United States Navy