Complete Human Digitization and Unconstrained Performance Capture
Abstract
As virtual and augmented reality (VR/AR) is growing into the next generationplatform for warfighter training, simulation, and collaboration, there is a growing need forgenerating detailed visual representations of ourselves. Although participants are commonly embodiedby digital avatars, the modeling and animation of these CG characters typically involvea time-consuming production pipeline. With the democratization of mobile cameras and depthsensors (e.g., iPhone X, Kinect), 3D capture technologies are now accessible to everyone, butexisting methods are restricted to the acquisition of raw non-parametric surfaces (e.g., KinectFusion).While the captured scans are suitable for 3D visualization and playback, they cannot beeasily animated or authored for real interactive applications and immersive experiences.Our goal is to enable the automatic creation of dynamic and parametric full body avatars, includinggarments, and facilitate their animations through new interfaces based on body gestures.We will focus on single-view consumer depth sensors and the recordings should be performedin unconstrained environments. The system must be easily deployable to various naval trainingfacilities. We propose to explore the digitization of fully clothed subjects, including their performancesunder complex garments. Our aim is to reconstruct animated human bodies withseparate geometric models for clothing. While computational models exist for anatomical structuressuch as unclothed bodies, they cannot be extended to handle the shape variation of garments.We plan to explore new data-driven and deep learning techniques for modeling the immensevariation of shapes caused by the intricate structures and interactions of human bodies and garments.Wewill introduce novel convolutional neural networks, that can process geometric models,and new training procedures, that use a combination of synthetic and real-world data, leveragingUSC ICT???s Light Stage 6 high-end acquisition device [49]. In collaboration with the Naval HealthResearch Center, we will capture various performances of real marines with different types of uniformsand equipment. Our proposed research will not only make the creation of highly realisticdigital humans scalable and accessible for immersive training purposes, but would directly impactour existing ONR funded research on automated fit-for-duty assessment and injury preventiontechniques for warfighters. If successful, this research could lead to new discoveries beyond thescalable generation of highly realistic digital humans, such as fundamental recognition, synthesis,and segmentation tasks for 3D deep learning.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jul 10, 2018
- Source ID
- N000141812349
Entities
People
- Hao Li
Organizations
- Office of Naval Research
- United States Navy
- University of Southern California