AdaFrame: Adaptive Frame Selection for Fast Video Recognition
Abstract
We present AdaFrame, a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame contains a Long Short-Term Memory network augmented with a global memory that provides context information for searching which frames to use over time. Trained with policy gradient methods, AdaFrame generates a prediction, determines which frame to observe next, and computes the utility, i.e., expected future rewards, of seeing more frames at each time step. At testing time, AdaFrame exploits predicted utilities to achieve adaptive lookahead inference such that the overall computational costs are reduced without incurring a decrease in accuracy. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet. AdaFrame matches the performance of using all frames with only 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We further qualitatively demonstrate learned frame usage can indicate the difficulty of making classification decisions; easier samples need fewer frames while harder ones require more, both at instance-level within the same class and at class-level among different categories.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jun 16, 2019
- Accession Number
- AD1152386
Entities
People
- Caiming Xiong
- Chih-yao Ma
- Larry S. Davis
- Richard Socher
- Zuxuan Wu
Organizations
- Georgia Tech
- University of Maryland