Deep Volumetric Video From Very Sparse Multi-View Performance Capture
Abstract
We present a deep learning-based volumetric capture approach for performance capture using a passive and highly sparse multi-view capture system. We focus on a template-free, per-frame 3D surface reconstruction from as few as three RGB sensors, where conventional visual hull or multi-view stereo methods would fail. State-of-the-art performance capture systems require either pre-scanned actors, large number of cameras or active sensors. We introduce a novel multi-view Convolutional Neural Network (CNN) that maps 2D images to a 3D volumetric field that encodes the probabilistic distribution of surface points of the captured subject. By querying the resulting field, we can instantiate the clothed human body at arbitrary resolutions. Our approach also scales to different numbers of input images, which yield increased reconstruction quality when more views are used. Though only trained on synthetic data, our network can generalize to real captured performances. Since high-quality temporal surface reconstructions are possible, our method is suitable for low-cost full body volumetric capture solutions for consumers, which are gaining popularity for VR and AR content creation. Experimental results demonstrate that our method is significantly more robust and accurate than existing techniques where only very sparse views are available.
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 06, 2018
- Accession Number
- AD1158373
Entities
People
- Chloe Legendre
- Chongyang Ma
- Hao Li
- Jun Xing
- Linjie Luo
- Tianye Li
- Weikai Chen
- Yajie Zhao
- Zeng Huang
Organizations
- University of Southern California