Sparse Multi-View and First Order Computer Vision and Image Understanding
Abstract
Computer Vision seeks to reconstruct the 3D visual world, enabling scene understanding and material perception for autonomous agents and robots. Applications include automated image understanding of maritime and aerial surveillance imagery, and semantic search of related visual databases. Most computer vision methods are designed for single images or videos (or perhaps a stereo image), and remarkable successes have been achieved on these topics in recent years. However, we are now rapidly transitioning to a world where sparse multi-view imagery is ubiquitous. For example, current mobile phone cameras (Apples iPhone and Googles Pixel flagship phones) have multiple cameras for photographic applications like portrait mode. Indeed, recentwork by the PI and others has shown that even entire 4D light fields, capturing the full spatial and directional flow of light in a scene can be reconstructed from a sparse set of multi-view images. A second major development is that computer graphics photorealistic rendering algorithmshave become accurate and predictive enough that their results can be used to match real images,and these results can be computed at real-time frame rates on current graphics hardware. However, most physics-based computer vision algorithms continue to make assumptions of Lambertian reflectance with no cast shadows or interreflections, and modern deep learning techniques often ignore the physics of light transport altogether. In this proposal, we seek to enable inverse lighttransport for computer vision with full computer graphics photorealistic rendering.This proposal develops a comprehensive approach to the above problems, ranging from theoretical foundations (such as limits for acquisition of light fields from sparse multi-view images, and effects of occlusion, specularity etc.) to algorithms that take in sparse multi-view imagery and recover shape and appearance, including non-Lam ertian effects. We will also consider theimpact on higher-level vision tasks such as segmentation and scene understanding, as well as duals to sparse multi-light imagery. We will explore differentiable rendering and optimization as a general paradigm to solve computer vision problems by refining the physics-based or learning solutionin an optimization pass, and consider theoretical formulations of shape optimization as an Euler-Lagrange variational problem. From a technical point of view, the proposal will combine existing expertise in physics-based computer vision, with physically-motivated deep learning and first order differentiable rendering/vision algorithms. While this proposal represents basic research, there is significant potential for future naval relevance. Maritime surveillance operations involve combining images and reconstructing 3D scenesfrom sparse multi-view imagery. These reconstructions could also be used by autonomous agents and robotic devices. The use of photorealistic scene simulations and first order methods will refine the solutions, enabling accurate shape and material maps which can guide artificial agents, and enable them to fully reason about their environment. Moreover, we have the ability to considerscattering and radiative transfer, as required for underwater imaging. The PI is uniquely qualified for this proposal, having developed some of the foundational results on reflectance and lighting in computer vision, including differential invariants for 3D reconstructions, and unified multi-cue shape from light fields. He was recently named an IEEE, ACM Fellow and inducted into the SIGGRAPH Academy. Approved for Public Release.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 08, 2020
- Source ID
- N000142012529
Entities
People
- Ravi Ramamoorthi
Organizations
- Office of Naval Research
- United States Navy
- University of California, San Diego