Thermodynamics of Learning
Abstract
The recent exponential increase in the applications of machine learning is based on algorithms that were already well known in the second half of the 20th century. These recent successes became possible due to the increased availability of computing resources, which allowed for a new level of complexity in the algorithms, as well as the increased availability of large datasets, which allowed these algorithms to be fit in very high dimensional parameter spaces without overfitting. While these methods have been very successful, two fundamental challenges remain. The first challenge lies in evaluating how well an algorithm works a priori, and in providing bounds on the predictions emanating from the algorithm. We aim to present research directions that may address these ideas at the algorithmic level (Task 1), then show how information theory can help address this constraint at the abstract learning level, independently of the algorithm (Task 2). The second challenge is to overcome the energetic constraints that are currently the principal limits on the size of the computational tasks required by the training of these algorithms. We will outline how information thermodynamics may help the emerging approximate computation paradigms produce energy efficient frameworks for learning (Task 3).
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 28, 2018
- Accession Number
- AD1078109
Entities
People
- Henry Hess
Organizations
- Columbia University