Computer Vision and Scene Understanding with Light Field Imaging
Abstract
A key goal of computer vision and scene understanding is to reconstruct the visual world in 3D, and to understand objects and scene"" structure, as well as material properties. This is relevant in many applications, such as automated image understanding, maritime o""r aerial surveillance.However, all computer vision methods essentially assume a single monocular image, or perhaps at best binocul""ar observation. This assumption was reasonable in the past, with bulky cameras. However, we are increasingly seeing a revolution whe""re new types of multi-camera imaging sensors are available, along with the computational power to take advantage of them. For exampl""e, current mobile devices such as the iPhone 7+ include two cameras (dual lenses), with images combined using computational photogra"phy. Mobile camera arrays have been proposed by vendors such as Pelican Imaging. Consumer light field cameras have become available" from Lytro and Raytrix. These cameras capture the full 4D light field, corresponding not just to 2D pixel intensity, but also captu"ring the angular distribution of light. This in turn allows for photographic applications such as refocusing after capture and small" shifts in viewpoint within the aperture of the main lens. Moreover, almost every major camera vendor has released a virtual reality"" camera, which in general can be viewed as a light field sensor. For virtual reality, having the full light field is crucial to give"" a sense of immersion, since the observer can look in any direction and move about the scene. Indeed, a workshop on light fields for"" computer vision (L4CV) is now a regular part of computer vision conferences, and the PI will be giving one of the keynote talks at"" CVPR 2017.In this proposal, we go beyond simple photography, focusing on computer vision and scene understanding with light field"" imaging systems, and expect the insights to be broadly applicable to future multi-camera/video systems. We have made initial effort""s on this problem for shape from light fields, and a novel theory of differential invariants for physics based vision with non-Lambe""rtian reflectance. However, computer vision and scene understanding with light field imaging involve much greater theoretical and pr""actical challenges, which we address in a comprehensive fashion.Specifically, we develop theoretical foundations, including a form""al characterization of the ambiguity space for light fields, analogous to the bas-relief transformation for Lambertian surfaces. Nex""t, we explore new practical applications, including motion blur as a converse of rendering, light fields for scattering with applica""tions to underwater imaging in naval applications, and potential applications in material recognition and scene segmentation. One ke""y goal is to combine physics-based modeling with modern machine-learning techniques based on Convolutional Neural Networks,to enabl""e enhanced scene understanding.Finally, a core problem with higher-dimensional imaging like light fields is lower resolution, sinc"e the sensor now needs to capture both spatial and angular variation. We will explore new approaches to light field interpolation fo"r both images and videos, considering applications in virtual reality and remote sensing/imaging. This proposal has significant pote""ntial for future naval relevance, since new sensors could give new capabilities in the maritime domain. Since multicamera or light f""ield sensing is passive, it can be explored in the wild, as required in the ocean.The PI is uniquely qualified for this proposal,"" having developed some of the foundational results on reflectance and lighting in computer vision, including unified multi-cue shape" from light fields. He was recently awarded an endowed chair and elected an IEEE fellow. His earlier work has been recognized with t"he ACM SIGGRAPH Significant New Researcher Award in Computer Graphics, and the ONR PECASE and Young Investigator Awards in physics-b"ased
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jul 07, 2017
- Source ID
- N000141712687
Entities
People
- Ravi Ramamoorthi
Organizations
- Office of Naval Research
- United States Navy
- University of California, San Diego