AI-based Fast 3D Large-scale Outdoor Scene and Instances Reconstruction for VR/AR
Abstract
"Approved for Public Release"High-quality virtual environments play an important role in an immersive experience in virtual reality,(VR) and augmented reality (AR) applications. The most widely used methods of 3D content creation for virtual environments are model,ed either manually by artists or using 3D scene reconstruction techniques (e.g. image-based photogrammetry/LIDAR). Both of them are,time-consuming and costly, especially for large-scale terrains. In particular, reconstructed 3D scenes usually leave behind corrupte,d data in the form of holes and gaps where captured data density is low or occluded. These inaccuracies need to be cleaned up before, being used in downstream applications. Reconstructed scenes also need semantic/instance segmentation to provide meaningful interact,ion experiences in AR/VR applications. We propose an end-to-end AI-based system of large-scale 3D scene reconstruction using images,captured by unmanned autonomous vehicles (UAV). Employing UAVs for such objectives is popular thanks to their ability to scan data o,ver a vast area in a short time, with minimum human effort. Our system will merge the scene reconstruction and semantic segmentation, into one network under the observation that the two tasks share the same photogrammetry information. By representing the large-scal,e scene in volumetric space and adopting a coarse-to-fine strategy, our reconstruction will be efficient and fast. As the reconstru,cted scene usually includes holes, artifacts, and erroneous geometries with relatively low-resolution details. We will further corre,ct and complete the scene automatically. Finally, high-resolution instances (cars, buildings, and humans) will be inferred using dif,ferentiable rendering and shape priors (i.e. a synthetic database for each object category) separately using the generated instance,segmentation labels. Our proposed automatic system will reconstruct terrain efficiently with multiple instances/objects reconstructe,d without intersection and in high quality. The system makes direct use of 2D UAV images and will be cost-effective and fast. The re,constructed 3D instances can be further used for animation and interaction. The generated contents may also be used to build dataset,s that can support other applications such as virtual environment simulation systems developed for One World Terrain (OWT) and objec,t detection/recognition.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Dec 10, 2021
- Source ID
- N000142212020
Entities
People
- Yajie Zhao
Organizations
- Office of Naval Research
- United States Navy
- University of Southern California