Explainable Systems: Improving confidence in autonomous systems
Abstract
Autonomous and autonomic systems are becoming increasingly important elements of today~s cyber landscape: fromsmart IoT, to robotic""s, to autonomous vehicles, to command and control, to resilience in enterprise systems. Thesesystems are taking increasing responsi""bility for carrying out complex tasks, detecting and repairing faults, andprotecting against attacks.But what happens when autonom"y goes wrong? How do we know what it is doing? How do we determine if it isbehaving correctly? And how do we steer it in the right direction if it is working incorrectly? Today~s systems arehopelessly bad at enabling us to answer these questions. The chief problem is that in today~s systems autonomy isopaque: autonomic decisions are often wired into the implementation of the system where the logic for those decisionsis spread across the code. In other systems autonomy is encapsulated ~ but only as a ~black box~ (for e"xample, trainedthrough machine learning), where the logic behind a decision is hidden.In this research, we propose to develop a pr"incipled approach to ~explainable systems~ ~ systems that can explain theirautonomous behavior in ways that a human can understand. The key idea is that a system retains reflective models ofits autonomous behavior that allow it to retrace and justify its decisio"n history (for past courses of action) and plans (forfuture actions). Critically, these models capture an explicit value system (in" a multi-dimensional quality space) thatforms the basis for determining courses of action. Because the value system is explicit it is possible for the system toprovide a rational explanation of the principles that led to a given plan of action.Focusing specifically on task-oriented autonomy ~ in which a system must plan a sequence of decisions to carry outsome task -- our research will develop a transferrable approach and toolset that will enable: (a) a system to explain itsdecision making and recommendations in a fo"rmally-grounded, but human-understandable way; (b) contrastrecommendations with potential alternative plans of action; (c) support" ~drill down~ to enable more detailedexplanations that link plans to implementations; and (d) permit operators of the system to cor"rect the system whendecision making is misaligned with policy or context, thereby improving its functioning over time.The formal b"asis for this work will be the use of plan synthesis engines based on probabilistic models with multidimensionalreward functions. S"uch engines have been used effectively to create plans that provide optimal utility in anuncertain environment. But, to date, they"" have not been coupled with mechanisms that use their models as a basis forexplanation. We will add those features to these tools,"" packaged as a reusable toolkit, allowing users to understandautonomous behavior, at varying levels of abstraction, without requiri"ng the users to understand the mathematicalunderpinnings of probabilistic state machines or utility functions.The research carried" out in this proposal will directly address the viability of future systems for the Navy, which areincreasingly relying on autonomi""c and autonomous system control, by improving our trust in such systems andallowing human-system coordination at a level not curren""tly possible. As such, it directly addresses ~Supportive andcomplementary approaches~ (TA5) of ONR BAA Announcement # N00014-17-S-B"10
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Sep 29, 2017
- Source ID
- N000141712899
Entities
People
- David Garlan
Organizations
- Massachusetts Institute of Technology
- Office of Naval Research
- United States Navy