Incorporating Scientific Inductive Biases into Neural Networks
Abstract
For many problems in the natural sciences, we are faced with a scarcity of data combined with lack of methods tailored to the type of inductive biases suitable for these problems. Some example settings include capturing atmospheric interactions to better forecastclimate events; designing proteins/small molecules for therapeutics and environmental applications; and designing materials for carbon capture, waste water treatment, and energy storage. Although in some settings technology may progress to increase the amount of data, in many domains, we expect to always be hampered by lack of data relative to current areas of great machine learning success. To make faster progress in these areas, we require a thoughtful revisiting of the coremodeling assumptions of appropriate inductive biases relevant to the sciences, and how they can be injected, in a principled manner, into state-of-the-art modeling strategies. Herein we propose to do so within the context of neural networks. These inductive biases can be viewed as science-informed constraints, and we consider classes of constraints of direct relevance to anumber of problem areas in the natural sciences.The key unifying technical challenge in our problem is that of training neural networks to impose constraints that are specified in a language that makes the resulting loss function not amenable to straightforward optimization. We will develop two main strategies to solve this technical challenge. The first is grounded in overcoming difficulties in setting up implicit layers in neural networks; these layers can be a natural way to incorporate physics-based solvers into neural networks. A second strategy will be focused on enabling norm-basedsoft constraints defined in #natural" bases, rather than on neural network parameters, to be implemented. In particular, we will develop a new suite of approaches wherein we bypass the need for computing an explicit transformation from neural network parameters to the chosen naturalbasis in which the constraint are specified.In addition to yielding improved modeling capabilities for prediction problems in the natural sciences, we anticipate that our methods will also impact more mainstream areas of machine learning, by unlocking new techniques for incorporating useful inductive biases more broadly. Overall, our work will help the navy have more accurate predictive models for tasks in thephysical world.Approved for public release.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 15, 2023
- Source ID
- N000142312587
Entities
People
- Aditi Krishnapriyan
Organizations
- Office of Naval Research
- United States Navy
- University of California Regents