Thermodynamics of Learning
Abstract
The recent explosion in the number and scope of machine learning applications have given rise to many questions as to the theoretical soundness of these algorithms, as well as to their performance and accuracy in data poor environments. In light of the next generation of applications, such as self-driving cars or diagnostic systems, a priori quantification of errors of a given learning procedure with a given dataset will become critical. Following successful applications of statistical mechanics to learning problems in the 90 s, well summarized in Statistical Mechanics of Learning , by Engel and Van den Broeck, several attempts were made to apply these methods to real-world applications. However, given the technical and computational resources at the time, these methods remained successful only on toy models featuring planted solutions and very basic algorithms such as variations on the perceptron. We proposed to give these methods a second chance given significant advances in three fields. First, nonequilibrium statistical physics, following the development of information thermodynamics and stochastic thermodynamics, has grown into a more mature discipline, intricately linked with information theory. Second, the ease, availability, and engineering know how regarding machine learning algorithms has been significantly increased through open source software libraries such as Scikit-Learn, Tensorflow, or Torch, allowing a vast body of engineering knowledge to be established as to which techniques are the most useful for a particular project. Third, computational resources have become orders of magnitude faster and cheaper, specifically thanks to the rapid development of low-level GPU programming environments and hardware such as CUDA or OpenCL. We therefore believe that, armed with these new theoretical and computational tools, we can successfully apply the ideas of statistical physics to machine learning problems on real data.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 11, 2018
- Source ID
- W911NF1710107
Entities
People
- Henry Hess
Organizations
- Army Contracting Command
- Columbia University
- United States Army