Back to the Future: Scalable Kernel Machines for Large-Scale Learning
Abstract
PROJECT SUMMARY / ABSTRACT Deep neural networks are having signi cant impact throughout academia and industry, and grand claims are made that no other methods can match their performance. We hypothesize that methods such as kernel machines are overlooked for computational reasons, not for their expressivity or generalizability. Naively scaling kernel machines is cubic in the number of data points, but with some classical and some modern techniques, one can get linear or sublinear scaling. On speech, random features for kernel machines (invented by PI Recht) match the performance of deep learning. On text extraction tasks, low-rank kernel templates (invented by PI R e) achieve dramatically better performance than neural networks and win recent competitions like the Knowledge Base Population challenge. Moreover, there are plausible reasons to believe that GPUs are superior computing platforms to commodity CPUs, but most of the investigations to date have super cial and not at all conclusive. We propose to explore the general theory of scalable kernel machines with the goal of improving multimodal classi cation and information integration tasks on commodity hardware. Our focus will be on new mathematics for improving the performance of randomized feature generators. We will investigate variance reduction techniques for compressing the representations of random features. Using tools from geometric functional analysis, we will study how to design data-adaptive random features. And using low-rank approximation techniques, we will design feature template matching features aimed at discrete models such as language processing. We will also investigate new algorithms for large-scale training of complex learning tasks. This will include studying how to rapidly evaluate and train of convolutional neural nets on commodity hardware and how to integrate these fast-algorithms into end-to-end systems while simplifying model prototyping.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 12, 2016
- Source ID
- N000141512620
Entities
People
- Benjamin Recht
Organizations
- Office of Naval Research
- United States Navy
- University of California Regents