Towards a Sequential Multi-Modal Subset Selection Framework
Abstract
Finding a small subset of informative items from a large ground set, known as subset selection, has become an indispensable tool for"" reducing redundancy in massive datasets and has found numerous applications, includingsummarization, feature and model selection,"" clustering, product recommendation, network routing and sensor placement. Sequential data, including time-series, such as videos, s""peech, biomedical signals, and ordered data, such as text, form an important large part of modern datasets, requiring effective subs""et selection techniques. Such datasets contain dependencies among data points, imposed by underlying dynamic models of data, that mu""st play a vital role in the selection of representatives. Moreover, data often consist of multiple modalities that should inform eac""h other in the selection of representatives across all modalities. On the other hand, humans perform remarkably well for summarizati""on of sequential data, such as videos, speech and text, motivating the development of methods that can learn from humans to summariz""e sequential data. In spite of rich literature on subset selection, most of the existing work ignore sequential dependencies among d""ata points, treat different modalities independently and are focused on unsupervised settings. In this project, we develop a unified"" mathematical framework for subset selection in multimodal sequential data that incorporates dependencies among data, effectively de""als with multiple modalities and takes advantage of the human high-level reasoning for summarization. More specifically, we will dev"elop and analyze a sequential subset selection framework that selects representatives capturing the data distribution and with a high global compatibility according to underlying dynamics of data. We will generalize our framework to deal with multiple data modalit"ies by taking advantage of inter and intra modality information. We will propose a supervised sequential subset selection method, by"learning representations of data so that the input of transformed data toour framework will lead to ground-truth summaries. Finall"y, we will apply our methods to address several problems in computer vision.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 20, 2018
- Source ID
- N000141812132
Entities
People
- Ehsan Elhamifar
Organizations
- Northeastern University
- Office of Naval Research
- United States Navy