Landscapes of Large Scale Problems with Applications to Machine Learning

Abstract

Abstract. The proposal was originally centered on landscapes in deep learning, but after our first investigations, we discovered a "hole" in the variational theory of modern machine learning. There was indeed no satisfying mathematical model of autodiff which is massively used by the AI community and even part of the applied mathematics community. The project was then naturally redirected toward the building of a proper theory of nonsmooth automatic differentiation. Great emphasis was put to have a model corresponding to real-world implementation through TensorFlow, PyTorch, or Jax. The first stones of the theory have been set with two papers at MPA and NeurIPS (SpotLight 20), and then we have studied various aspects such as nonsmooth implicit differentiation (with application to DEQs, differentiable programming), the differentiation of algorithms, and various regularity approaches. At the same time, other investigations have been led in ML and optimization, such as convex optimization, global optimization, and SOS methods.

Document Details

• Document Type: Technical Report
• Publication Date: Apr 17, 2023
• Accession Number: AD1211793

Entities

People

• Jerome Bolte

Organizations

• Toulouse School of Economics

Tags

Communities of Interest

• Autonomy
• Energy and Power Technologies

DTIC Thesaurus Topics

• Air Force Research Laboratories
• Algorithms
• Applied Mathematics
• Artificial Intelligence
• Computer Programming
• Data Analysis
• Data Mining
• Deep Learning
• Differential Equations
• Equations
• Information Science
• Machine Learning
• Mathematical Models
• Mathematical Programming
• Models
• Neural Networks
• Numerical Analysis
• Operations Research
• Optimization
• Probability
• Sampling

Readers

• Artificial Intelligence
• Operations Research
• Theoretical Analysis.

Technology Areas

• AI & ML
• AI & ML - Machine Learning Algorithms
• AI & ML - Neural Networks