THIS IS A CONTINUATION OF N00014-14-1-0316 Visual Material Understanding
Abstract
isual Material Understanding PI: Ko Nishino Project Summary We propose a research program that aims to establish the theoretical and computational foundations to fully leverage materials as visual context for image understanding in computer vision. Our goal is to make materials the central player in computer vision to effect transformative advances in image understanding. The material determines how an object of particular shape would appear in our world, ranging from its surface texture to the subtle gleam of a highlight. We, as human beings, can easily tell a wool glove from a hand, or even a leather glove despite the fact that they all have the same hand shape. This indicates that there is abundant visual cues characteristic to the underlying materials that awaits us to harness for understanding the image contents. We will explore two separate approaches to extract material information from images. The first is a generative approach geared towards recovering characteristic radiometric properties pertaining to real-world materials. In particular, we will focus on estimating the reflectance as the unique optical signature of different materials. The main focus of our research is to enable the estimation of reflectance in arbitrary images, without knowing the other radiometric constituents, namely the illumination and geometry, of the scene. We will achieve this by bootstrapping the estimation with initial geometry estimates, either in the form of coarse depth data acquired with RGB-D sensors or rough geometry estimates obtained from multiple images via structure from motion, and by deriving novel statistical priors of materials conditioned on their reflectance, scene, and surface geometry. We will establish a sound theoretical foundation with a probabilistic formulation that will enable the incorporation of these strong material priors to solve the otherwise severely illposed problem. The second approach is a discriminative one where we aim to directly classify each pixel or image region into different materials. For this, our research will focus on deriving the canonical representation of materials that captures the wide appearance variability faithfully but concisely. Key research thrusts consist of automatically discovering material representations that tailor the bases for covering the appearance variations compactly; and exploiting the dependency of materials on scenes (e.g., the types of materials we observe in a desert is very different from those in a cityscape) and their relative spatial configurations in both 2D and 2.5D (RGB-D) data (e.g., a cup induces liquid surrounded by ceramic) in the classification process. We will also investigate the inter-dependency of objects and materials as well as scenes and materials to derive an iterative algorithm that reinforces the recognition of materials with the recognition of objects and scenes. Finally, we will combine the generative and discriminative approaches to devise practical computer vision tools to exploit material as visual context in existing paradigms of image and scene understanding. For this, we will focus on deriving a material-aware local image descriptor, a compact key point and region representation that encodes not just the shape but also the material information in the local neighborhood. This will be achieved by assuming rough geometry of the local region to obtain a rough estimate of the reflectance which is then classified into material groups based on a novel material representation. We believe such a material-aware image descriptor has the potential of transforming current image understanding methods that heavily rely on feature detection and correspondence matching.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jun 03, 2016
- Source ID
- N000141612158
Entities
People
- Ko Nishino
Organizations
- Drexel University
- Office of Naval Research
- United States Navy