The Next Generation of Operator Regression Networks: Theory, Algorithms, Applications
Abstract
We propose to develop generalized neural networks (NNs) that learn nonlinear multi-operators and can solve problems at a higher leve,l of abstraction and 1,000 times faster, hence developing the next generation of neural operators such as DeepOnet, a biologically-i,nspired NN we introduced recently for real-time forecasting of multi-physics systems. The overarching goal is to develop new algorit,hms for biologically-inspired neural operators that can be implemented in the next generation of neuromorphic computers. The propose,d work is to develop technology for both military and civil applications.Building on our preliminary theoretical results on the univ,ersal approximation properties of DeepOnet, we plan to first extend the theory to multi-operator regression architectures for multi-,modal inputs, inspired by dendritic branching of human neurons, and resolve the issue of catastrophic forgetting in order to enable,multi-tasking and continual learning. We will develop corresponding approximation and generalization theory, focusing on the curse-o,f-dimensionality in the multi-modal input space and examining system-stability subject to random and adversarial perturbations. In a,, first studying promising fractional gradient methods but more importantly going beyond back propagation. Specifically, we will int,roduce alternative methods to the current backpropagation such as local methods and weight mirrors, and explore,le models, e.g., Hebbian Learning for forward-only weight updates. Spiking Neural Networks (SNNs) have emerged as biologically-plaus,ible methods with significantly lower computational cost but the lack of smoothness of the spiking signals has led to fundamental di,fficulties. We will investigate the bio-inspired spike time-dependent backpropagation (STDB) incremental training although current r,esults show sub-optimal performance compared to artificial neural networks (ANNs). To this end, we propose to investigate a combinat,ion of methods such as the ANN-SNN conversion for initialization and STDB for final convergence, as well as new batch-normalization-,through-time techniques. We also plan to investigate NeuroEvolution of Augmenting Topologies (NEAT) to learn simultaneously both the, ideal topology and weights in SNNs.On the application front, we will consider real-time forecasting of complex multi-physics system,s (e.g., hypersonics and autonomy), and we will combine deep learning with deep reasoning (e.g., a DeepOnet that includes Relation N,etworks) for interactive design and for predicting human behavior in social dynamics. Developing higher level abstractions for deep,learning, addressing the exponentially rising costs of inefficient training, and incorporating reasoning in DeepOnet will lead to a,paradigm shift in scientific machine learning and more broadly to AI. These fundamental developments will have an immediate technolo,gical impact on autonomy, design, human-robot interactions, which are critical areas to DoD, but also societal impact, e.g., quantif,ying human dynamics. We plan to work with neuroscientists and cognitive scientists, the computer industry, other VBF fellows, and wi,th researchers from ARL, NRL, AFRL, and the DOE labs. The PI will develop new curricula and tutorials for training a new cadre of si,mulation scientists on ?mathematics + machine learning + X?, including under-represented groups. He will organize the 2023 MSML (Mat,hematical and Scientific Machine Learning) conference at Brown University, which will include tutorials both on theory and practical, implementation of machine learning tools.APPROVED FOR PUBLIC RELEASE
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 08, 2022
- Source ID
- N000142212795
Entities
People
- George Karniadakis
Organizations
- Brown University
- Office of Naval Research
- United States Navy