Shape Recovery and Image Understanding with non-Lambertian Reflectance
Abstract
Shape Recovery and Image Understanding with non-Lambertian Reflectance Ravi Ramamoorthi University of California, San Diego A grand challenge for image analysis and machine vision is to fully understand and interpret the visual world, with applications in automated image understanding, surveillance and reconaissance. Real-world surfaces have a range of complex appearances, such as the shiny reflections from a velvet cushion, the translucent appearance of human skin, or the glossy reflections of a polished metal surface, all of which are common challenges in the maritime domain. Modeling a range of objects in natural outdoor and indoor conditions requires understanding the physics of light reflection from surfaces of varying material properties—more formally, we must have a handle on non-Lambertian reflectance. Moreover, current systems for higher-level computer vision tasks such as shape recovery, recognition of people and actions, or segmentation of objects are all reaching the point where progress requires treating the physics of image-formation as a first-class primitive. New sensing primitives available in the past couple of years give us hope of substantially addressing this challenge. First, light field sensors or multi-camera arrays have or will soon become available to the consumer market (either stand-alone as in the Lytro microlens light field camera or integrated into mobile phones as in the Pelican Camera Array). This is a revolution in imaging, wherein a number of closely related viewpoints are acquired in a single shot capture. This allows us to tease out the effect of specularity, based on the different appearance from nearby views, a capability that has not previously existed. Moreover, acquisition is passive, allowing the methods we develop to work in the wild and in the outdoors, which is key in naval sensing applications. In a similar vein, one can imagine systems to position nearby differential light sources (perhaps in a mobile device with close by flashes, integrated with light field sensors). The PI’s initial work on differential photometric stereo shows that a method that has theoretical guarantees of shape recovery for arbitrary reflectance properties can be formulated. The proposal will address threemajor technical challenges related to non-Lambertian reflectance from light field and differential image sensors. We will develop theoretical foundations, providing precise information about what properties of shape can be recovered from general BRDFs, leveraging our foundational results on differential photometric stereo and optical flow, and generalizing to differential viewpoint change, and numerically stable differential invariants. We will build practical algorithms for shape recovery from light field data (a nearby or differential set of viewpoints). A key idea is that one has multiple cues simultaneously in a single shot like depth from defocus and correspondence; also multiple viewpoints for specular and transparent objects. Finally, we will investigate image understanding and 3D model building. The potential of easy 3D acquisition for general non-Lambertian objects, potentially enables a rich new ecosystem of 3D geometry instead of still images. More interesting, 3D information especially around edges can greatly enhance performance of higher-level image understanding tasks like recognition and segmentation. The PI is uniquely qualified for this proposal, having developed some of the foundational results on reflectance and lighting in computer vision, including spherical harmonic lighting and differential invariants. He has also been recognized with the ACM SIGGRAPH Significant New Researcher Award in Computer Graphics. Much of this work was done under earlier ONR PECASE and Young Investigator Awards; the success of these projects also show that he can work with ONR and appreciate basic research with future naval relevance.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 12, 2016
- Source ID
- N000141512013
Entities
People
- Ravi Ramamoorthi
Organizations
- Office of Naval Research
- United States Navy
- University of California, San Diego