Symbolic Knowledge Distillation of LLMs for All: Diverse Scales, Skills, and Values
Abstract
While frontier large language models (LLMs) such as GPT-4 have unlocked enormous capabilities and potential powered by the equally enormous scale of data and compute, the current status quo of using these LLMs via API calls for downstream applications is far fromideal to serve all real-world and DoD use cases. For example, for DoD applications, it wouldn#t be acceptable to feed DoD#s national security data into e.g., OpenAI#s ChatGPT interface. Moreover, the current scale requirements are too costly for most private and public sector engineers and researchers to build such foundation models from scratch, essentially resulting in major concentration of power with respect to frontier LLMs. In fact, never mind training from scratch#even fine-tuning or aligning such extreme-scale models requires a level of compute that most engineers and researchers outside a handful tech companies cannot afford, which entails that it is practically impossible to customize these models to the particular needs of users, apart from relying on so-called #prompt engineering# in API calls.In this proposal, we lay out an ambitious research plan to address the major limitations and pain points of the current status quo by seeking an alternative learning paradigm that can lead to models that are orders of magnitude smaller than the frontier models while providing (1) competitive task/capability-specific performance, (2) facilitated white-box customization, and (3) interpretable andcontrollable symbolic knowledge as the backbone of such models.In essence, what we propose to build is Symbolic Knowledge Distillation of LLMs for All with three-pronged objectives: supporting# diverse scales (from the micro-scale of less than 1B parameter models tothe medium-scale of 30B - 70B parameter models),# diverse skills (ranging from commonsense tasks to various language tasks)# diverse values (supporting diverse moral, cultural, and social normsbeyond what the frontier LLMs currently support).
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 15, 2024
- Source ID
- N000142412207
Entities
People
- Yejin Choi
Organizations
- Office of Naval Research
- United States Navy
- University of Washington