Data-driven 3D Event Browsing from Multiple Mobile Cameras
Abstract
1. Summary Cameras on mobile platforms, such helmet-mounted cameras on soldiers, cameras sensors fixed to robots, or cameras mounted on vehicles, are increasingly used in military security and reconnaissance missions. In all these cases, an event is recorded from multiple sources and vantage points and sent to a central location where analysts sift through the data manually and prepare information digests for higher-level authorities to make decisions. Similarly, in the civilian domain, citizen journalists capture large amounts of visual data, using their smart phones, which then reach media and law enforcement. Automatically organizing such massive amounts of data is a challenging task since images are acquired from different locations, angles, zoom, focus, with different types of cameras (video, still, high quality, low quality), and at different times. Piecing together all of this data is like solving a dynamic jigsaw puzzle but without the advantage of seeing the geometry or 3D shape of the wooden pieces that fit together. In this proposal, we seek to organize the disparate visual data streaming in from a possibly live event into a georeferenced dynamic 3D environment. Analysts will have a bird s eye view of the entire event and will be able to browse through the event in 3D. Such a visual organization will facilitate efficient and accurate decision-making. To build such a system, we propose a 3-year research program that achieves several fundamental advances in computer vision techniques. The main technological barriers to such a system are: (1) reconstructing dynamic 3D activity from moving cameras is a hard under-constrained problem that often presents with missing data, and (2) establishing correspondence between visual data that is multi-modal (photos, videos of different qualities, point clouds). Our approaches to overcome these barriers will be based on three principles: (a) Principled combination of data-driven and model-driven approaches for image understanding, (b) Developing data-driven optimization methods for corresponding visual data with provable guarantees for accuracy and (c) Building large scale data-rich priors for human activity that better constrain the dynamic reconstruction problem. The key differentiator of the proposed research is the use of data-driven priors to address every part of the reconstruction pipeline, and the production of worst-case guarantees for our algorithms. We will develop a rigorous test-bed to conduct quantitative evaluation in several scenarios including meetings and social events, citizen journalism events, simulated adversarial behavior, such as paintball competitions, and live-traffic intersections. The quantitative metrics include numerical reconstruction errors, prediction of missing data, and user studies to measure usability of the interface we develop. Data collection for the test-bed will occur in our Panoptic Studio, a CMU facility with 500 cameras, to produce 3D shapes and actions of individuals and groups with the highest detail in space and time. The PI Narasimhan and Co-PI Sheikh are experts in all aspects of reconstruction, tracking, imaging, illumination, image and video analysis, physically based modeling, and large data collection and analysis. They have served previously as PIs or Co-PIs of several ONR, NSF, DARPA grants as well as numerous industry projects. They have a strong track record in knowledge dissemination via publications, seminars, workshop and conference organizations, websites, code release, and have advised and graduated several doctoral students and post-doctoral scholars. The project is expected to impact and inspire new research in event and social behavior understanding, battlefield decision-making, as well as public interest news analysis.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 08, 2016
- Source ID
- N000141512358
Entities
People
- Srinivasa G. Narasimhan
Organizations
- Massachusetts Institute of Technology
- Office of Naval Research
- United States Navy