WAR Fighting Performance: Augmented Reality Multi-Modal Interaction Techniques for JTAC and Battlefield Readiness
Abstract
With significant advancements occurring in Augmented Reality (AR) head-mounted displays (HMDs), their usage for Navy and other milit ary applications are quickly becoming a reality. However, with this nascent widespread use, there is little in the way of establishe d interaction techniques and modalities. This means that while some modalities are implemented, they are often not implemented with users in mind. Rather, their goal is often system recognition of the input. Joint Tactical Air Controllers (JTAC) operators are one population that will likely be on the front lines using AR-HMDs. Without a development of user-centric interaction techniques and m ultimodal input options, their performance will suffer. JTAC forces operate in an extremely diverse range of environments, under hig h task load, and with many tools that sometimes require simultaneous use.AR-HMD use take the form of a visualization system such as The Virtual Reality Rehab (SBIR Phase-II) Synthetic Vision Systems for Ground Forc es (SVSGF). For example, SCSGF allows JTACs to visualize targets, splash-zones, mini-maps, and instance, situational awareness. Wit h the wide range of contexts multimodal inputs are vital in maintaining robust interactions with the system. These inputs and input modalities must be developed with the user in mind. The amount of cognitive load they are under, the environments that they are in, the inputs they have available, and the complexity of the task that needs completion will all play into how optimal inputs into the system are formed.There may be a scenario where a JTAC operator needs to maintain the vision constant of a moving target through a zooming visualization aid inside of the device while also needing to insert a splash zone visualization on that target. In this s cenario, we can see that gesture alone would not be enough to allow for input into the AR-HMD visualization system. A combination of gaze tracking to follow the target, gesture inputs for an indication of where to add the splash zone visualization, and speech to i ndicate what the parameters are for the splash zone to be added would be a more robust system allowing for completion of all of the required tasks in unison rather than needing to do them individually. If in that same environment partially through the speech spec ifications of the parameters needed nearby ordinance caused the environment to become too loud than that user could switch from use ientation of the aid.The above scenario poses two big questions: what the appropriate unimodal and multimodal inputs into a system are, and how can that system detect and correct for errors and input transitions in real-time. The research proposed here will answ er those questions, allowing for that system to become a reality. This will be achieved through observation studies of users interac ting in simulated JTAC environments using gaze, speech, gestures, and different combinations of those inputs. The interaction techni ques of use will be measured for how intuitive they are, and the generation of both sets of multimodal interactions as well as unimo dal interactions will be generated. By also measuring the patterns of unimodal and multimodal inputs in unconstrained environments ( i.e., when and input used is acceptable), the contexts of use and strengths of the combinations and individual modalities can be mea sures. Lastly, from the above findings, the interactions will be implemented and tested against each other to find the most efficien t input system. This proposal seeks to use two systems to study unimodal and multimodal interaction for JTACs and develop new method s of interaction.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Oct 22, 2021
- Source ID
- N000142112949
Entities
People
- Francisco Ortega
Organizations
- Colorado State University
- Office of Naval Research
- United States Navy