Large-scale Immersive Virtual Reality for Military Training And Development of Robust Computer Vision Techniques
Abstract
Military operating conditions and environments vary significantly. While actually creating these different operating conditions and environments is challenging and expensive, simulating military scenarios with virtual environments represents an attractive alternative. However, creating realistic virtual environments requires sophisticated multimodal visualization and rendering tools. The goal of this research is to augment our existing state-of-the-art immersive virtual reality (VR) environment with additional equipment to further enhance its capabilities for interactive military training as well as for the development of robust video and audio analytics techniques. To this goal, we propose to acquire smart lighting equipment and mobile/stationary audio and visual sensors. The smart lighting equipment will enable our current VR system to intelligently vary lighting to illuminate different areas of the space based on the content being displayed and on the tasks users need to perform. The lighting system can also be easily customized to simulate different illumination conditions for military operations. The mobile/stationary audio and visual sensors allow us to perform distributed multimodal sensing of the users to monitor their activities and to provide natural interactions with the virtual environment. The enhanced immersive VR environment can provide effective training for a team of users such as military squads. Through the virtual immersive environment, users can acquire new skills with realistic, interactive scenarios accessed through immersive 3D worlds without the potential risks of live-action situations. Furthermore, with the proposed multi-modal sensing and video analysis techniques, training can be made more interactive and fun. Users can use their body gestures and voice to naturally interact with the computer agents in the VR system, to control and manipulate the virtual objects, and to navigate around the virtual environment. The computer agents, in turn, can localize the trainees, recognize their verbal and non-verbal behaviors, infer their cognitive states and intents, assess their performance, and provide appropriate feedback to improve their training performance. The enhanced immersive VR environment can also facilitate the development of robust computer vision and audio processing techniques. With the large-scale multimodal display, computer-controlled intelligent lighting system, and sophisticated rendering techniques, the immersive environment can create arbitrarily complex and realistic scenarios including different landscapes, architectures, terrains, different types of targets, and different background clutters under varying illumination condition. Such dynamic and complex scenes, coupled with corresponding groundtruth annotation, provide a large amount of challenging yet realistic data to develop robust computer vision and auditory processing techniques for such tasks as target detection, tracking, and recognition, human event and activity recognition, and 3D scene recognition and reconstruction. In summary, the proposed purchase of smart lighting equipment and mobile/stationary audio and visual sensors will greatly enhance our existing immersive virtual reality environment. The enhanced VR environment can provide a platform for effective military training and for the development of robust computer vision and auditory processing techniques.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 12, 2017
- Source ID
- W911NF1610222
Entities
People
- Qiang Ji
Organizations
- Army Contracting Command
- Rensselaer Polytechnic Institute
- United States Army