Effective Novelty-Based Human-Machine Interactions in Open-World Outdoor ISR Domains
Abstract
AbstractEffective Novelty-Based Human-Machine Interactions in Open-World Outdoor ISR Domains(Approved for Public Release)Task-basedhuman-machine interactions in open-world outdoor domains, in particular, ISR domains require machines to have substantial situational awareness and interaction capabilities, being able to detect unexpected and unknown aspects in familiar and unfamiliar scenes that need to be investigated and report them to human operators. This requires a tight integration of the artificial agent#s perceptualand language subsystems in order to enable linguistic characterization of unknown percepts, including novel objects and their properties as well as novel agents and their behaviors as they arise in open worlds. It also requires the system#s ability to introspect on these characterizations and generate natural descriptions to be able to effectively communicate with humans about the novel aspects in order to avoid wrong interpretations of visual cues or misunderstandings of their descriptions on the human side. Finally, based on task-based interactions with the human about the novelty, the artificial should then be able to make any necessary adaptationsits planned COA, including learning novel behaviors quickly on the fly.The overarching goal of this effort is to reach an unprecedented level of open-world novelty characterization, communication, and adaptation that (1) significantly improves open-world situational awareness and understanding in artificial systems, (2) enables effective human-like communications in open-world settings, and (3) allows agent to quickly adapt their COA in the interest of meeting their mission objectives. We will develop and integrate the next generation of edge-based neuro-symbolic algorithms within a cognitive robotic architecture, building on our extensive past ONR-funded work on grounding AI agents in our physical world, vision and language processing, and #zero and one-shot learning# of novel objects and their properties, and concepts in general, in open worlds. The integration, utilizing the DIARC cognitive robotic architecture framework, will tackle challenging problems ranging from negotiating novel objects and scenes through dialogues, to learning new concepts quickly from both data and instructions in a way that generalizes across contexts, to quickly adapting to novelties through a combination of different methods, including a hybrid planning reinforcement learning system that can quickly learn novel behaviors. The integrated architecture will be evaluated in a series of increasingly complex outdoor ISR tasks on a fully autonomous Spot robot.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Dec 15, 2023
- Source ID
- N000142412024
Entities
People
- Matthias J Scheutz
Organizations
- Office of Naval Research
- Tufts University
- United States Navy