Layered Semantic 3D Modeling from Large-Scale 3D Point Clouds for Indoor and Outdoor Environments
Abstract
SRI International (SRI) proposes to research new techniques that exploit the semantics in thescene to improve the quality of 3D mod"els for both indoor and outdoor environments using atop-down, layered (coarse-to-fine) approach. SRI proposes to investigate and ex"ploit deeplearning techniques to detect objects in the scene and perform a top-down reasoning about thelayout of buildings and scene structures that will enable the 3D models to overcome limitationsof bottom-up segmentation approaches.We propose to extract spa"ces (e.g., rooms, hallways) aligned to a canonical world coordinatesystem. Then, the spaces are segmented into structural elements"" (walls, doors) and clutterobjects, such as furniture. A similar hierarchical decomposition can be done for outdoor scenes,first p""ortioning into ground plane and non-ground plane structures, then further dividing intobuildings, roads, trees, street fixtures, an"d road signs.SRI will research and use probabilistic grammars that enable us to automatically learn how tocombine semantic primiti"ves into higher-level objects in a layered, hierarchical representation.The techniques we will develop will significantly overcome" limitations of current state-of-the-artalgorithms and system for automatically creating 3D models. We will be agnostic to the way the3D information is extracted. SRI will handle both 3D point clouds created by structure frommotion and multi-view stereo algori"thms from videos and imagery, as well as LIDAR data withoptional images co-collected.SRI has a very strong framework, algorithms a"nd software for mapping indoor and outdoorspaces using robots or dismounts that are freely moving around. SRI has a large portfolio" ofexisting algorithms for 3D point cloud processing, region segmentation, 3D modeling for indoorsand outdoors using a variety of" platforms that will form the basis of the bottom up processinglayer for the propose effort. SRI has also been using deep learning" from images and videos for avariety of tasks such as: generic object recognition, semantic scene segmentation, fine-grainedrecogn""ition, 3D modeling and structure classification.SRI proposes a three-year effort as follows:Year 1: Develop new classifiers based" on point-cloud and image data to robustly extractdifferent surface components and label these components. Use top-down knowledgeencapsulated in grammars characterizing layout of building to build simplified polygonal modelswhile inferring surfaces for missing" data and occluded surfaces. Use the classification to extractand label top level building structures, such as rooms, passage-ways,"" stairs, lobbies, etc. thesecan be represented as separable 3D models allowing further simplification of the primarybuilding struc""tures such as doors, windows, ceilings, floor etc. The semantic 3D modeling willenable us to automatically model cluttered scenes."" In Year 1, we will focus on INDOOR scenes.Year 2: Do fine grain modeling and extract of finer object classes such as furniture, fi""xtures,etc. for INDOOR scenes. Extend SRI~s building, foliage, ground extraction algorithms to exploitsemantic categorization base"d top-down inference to extract better polygonal models forOUTDOOR scenes. This will allow better polygonal models for indoors and" outdoors in areas ofclutter. When there are trees near building or traversable roads occluded by foliage, the semantictop-down re"asoning layer will allow us to create separable simplified structures. This will allowcompact polygonal modeling without having to go down to ultra-high resolutions (< 2-3cm).Year 3: Develop framework for rapidly updating 3D models based a sparse set of images orRGBD images. We will exploit semantic inference layer infer changes and 3D to update themodels. We will also improve the indoor and outdoor modeling methods developed in Years 1and 2.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 29, 2017
- Source ID
- N000141712949
Entities
People
- Bogdan Matei
Organizations
- Office of Naval Research
- SRI International
- United States Navy