Sketching for More Efficient Foundation Models
Abstract
Approved for public releaseRecent advances have created an explosion in interest in modern machine learning. For example, recent generative AI products such as ChatGPT, Gemini, DALL-E, and more, have captured the imagination of the general public across the world.The workhorse of such machine learning models is the transformer model, which has been successfully applied to a wide variety of learning tasks in areas such as natural language processing, computer vision, time series forecasting, and more. Despite their success, such models suffer from the curse of dimensionality because exact computation of their attention layers incurs quadratic (in the sequence length) runtime and memory complexities. This presents a challenge for scaling transformer models to longer context lengths.The goal of this project is to develop techniques using the tools of theoretical computer science, especially sketching algorithms,to make transformers more efficient by creating novel dimensionality reduction techniques to address the unique requirements of transfomers and the underlying attention mechanism.Impact on DoD capabilities:The proposed project is foundational research that aims to improve the efficiency of both training and inference with foundation models and LLMs, to allow for efficient training with longercontext length to provide improved accuracy. Thus, there is likely to be positive impact in multiple domains of interest to the navy. For example, transformers are used in state-of-the-art machine learning systems for both video and language understanding as wellas speech recognition, in robotics, ocean modeling, large-scale acoustic signal analysis, and navigation, to name just a few.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 09, 2024
- Source ID
- N000142412647
Entities
People
- David Woodruff
Organizations
- Carnegie Mellon University
- Office of Naval Research
- United States Navy