Foundations of Robust Neural Networks
Abstract
This research proposal aims to address critical gaps in our understanding of deep neural networks (DNNs), particularly their robustness and reliability, especially in safety-critical applications. Despite the broad success of DNNs and Deep Learning in various tasks such as image classification, object detection, and reinforcement learning, their brittleness, characterized by poor performance in response to slight changes or variations in the input or operating environment, poses considerable challenges. The susceptibilityto minor perturbations not only hinders the performance of DNNs in their current applications but also raises serious concerns about their deployment in more sensitive, safety-critical domains where the cost of failure could be substantial. This brittleness also acts as a barrier to the full realization of the potential of these advanced technologies, underscoring the urgent need for researchefforts aimed at enhancing their robustness and reliability. More fundamentally, there are significant gaps in our theoretical understanding of the fundamental behavior and robustness of neural net models, particularly in safety critical applications.This research proposal aims to address these challenges and open questions by integrating mathematical concepts and principles from convex optimization. The proposed research focuses on establishing a theoretical foundation of robust neural networks, designing optimal and robust network architectures, ensuring robustness under changing environments, enabling safe and robust learning for control systems, and developing scalable algorithms for training robust networks. The project intends to provide a novel theoretical framework for robust and reliable neural network models, leveraging hidden convex regularization theory and exploring connections to compressed sensing and sparsity.The principal goal of this project is to develop a theoretical framework for robust and reliable neural networks, drawing on a diverse body of mathematical theories. A revolutionary hidden convex regularization theory for neural networks, recently discovered by the principal investigator, constitutes a significant underpinning of this theoretical framework. This theory will demystify current overparameterized neural networks and potentially replace them with robust models that come with theoretical guarantees. The proposal also explores connections to compressed sensing and sparsity, leading to new theoretical and algorithmic developments. The research is expected to result in significantly better-performing, robust, and safe neural models, addressing a multitude ofpractical problems.The proposal is structured around four research thrusts. Thrust 1 focuses on developing a novel convex analysis and perturbation theory for robust neural networks. Thrust 2 applies the theory developed under Thrust 1 to design practical continual learning methods focused on adapting to changing environments. Thrust 3 develops a novel methodology for learning-based optimal control using robust neural networks. Thrust 4, finally, develops novel optimization algorithms for distributed training of robust models.This research has direct relevance to the Navy s mission, as well as many other potential use cases in safety-critical applications. The proposed approach aims to significantly advance the state of the art in the development and application of DNNs, particularly in mission-critical tasks."Approved for Public Release"
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 15, 2024
- Source ID
- N000142412164
Entities
People
- Mert Pilanci
Organizations
- Office of Naval Research
- Stanford University
- United States Navy