Counterfactuals and Multiple Rewards: Inducing and Explaining Good Team Behavior for Effective Agent Human Teaming
Abstract
In this work we propose two new paradigms, counterfactual based stepping stone rewards and multireward learning, that enable robust human machine teaming for long term autonomy. Though increased machine intelligence significantly improves the performance of many real world systems, maintaining human oversight is critical for many safety critical applications. Consequently, incorporating human insight and elucidating agent decisions are critical in machine intelligence and even more so for human machine teaming applications where nuanced interactions are often necessary to achieve complex goals. Unfortunately, it is exceedingly difficult for a human operator to inject suggestions into a complex autonomous system or for the human to accept and implement the possibly counter intuitive decisions of an autonomous agent, significantly reducing the effectiveness of human machine teaming for many missions. The proposed work addresses this gap through two related but distinct approaches: stepping stone rewards and multi reward learning. Intuitively, stepping stone rewards use the concept of counterfactual agents to hypothesize, evaluate, and isolate promising actions in domains that require teammates to find a complex joint action. Multi reward learning introduces the concept of alignment to determine “what matters when” and allows an agent to focus on the simplest, most beneficial action that will help its long term goals. Together, these two approaches: (i) enable an agent to learn complex behaviors; (ii) allow a human teammate to suggest potential actions or rewards to an agent; and (iii) enable an agent to justify its actions by isolating the conditions and rewards for which its action will be beneficial. The proposed work bridges the gap between the low level analytical deduction of autonomous agents and the high level mission understanding of human cognition.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 14, 2022
- Source ID
- FA95501910195
Entities
People
- Kagan Tumer
Organizations
- Air Force Office of Scientific Research
- Oregon State University
- United States Air Force