Explainable and Trustworthy Intelligent Systems
Abstract
Motivation. Deep learning and neural network architectures have enabled unprecedented breakthroughs in a numberof Artificial Intelligence (AI) applications. However, the lack of decomposability of current generation of deep learningmodels into intuitive and understandable components makes them difficult to interpret or understand. Theeffectiveness of these models is fundamentally limited by their inability to explain their decisions and actions to humanusers.Research Objective. The two main goals of this proposal are: (1) to develop theory, algorithms, and implementationsfor transparent deep neural networks that are able to provide explanations for their predictions, and (2) to study the effect of developed transparent neural networks and explanations on user trust and perceived trustworthiness with Visual Question Answering [6] as our AI testbed. Technical Approach. Our proposed work is divided into three thrusts:~ Thrust 1: Transparent Neural Networks and Explanation Modalities. We will develop neural network models thatconsist of interpretable internal representations or semantically named neurons, and produce the following 3explanations from such models: (a) Neuron Name Paths (~Audit Trails~) or Textual Explanations. This is a ~cat~ because these neurons fired: ~line segment~, ~blob~, . . . , ~eye~, ~head~, ~paw~, ~cat upper-body~, ~cat~. (b) Heat Maps or Visual Explanations. This is a ~cat~ because these pixels are ~paws~, these pixels are ~eyes~, etc. (c) Important Training Instances or Case-based Explanations. This is a ~cat~ because these training images are annotated as ~cat~.~ Thrust 2: Evaluation of Transparency and Explanations. What is the effect of our developed transparency techniqueson user trust and perceived trustworthiness? We will conduct human studies on crowdsourcing websites such asAmazon Mechanical Turk (AMT). One way of determining ~appropriate~ trust is to ascertain whether an explanation enables a user to trust a ~better~ model more often than a ~worse~ model.~ Thrust 3: Beyond VQA: Visual Dialog and Chatbots. We will generalize VQA to a new AI task ~ Visual Dialog ~ anddevelop a large scale dataset, a memory-based deep learning model, and generalize our proposed transparent deep networks and explanation modalities to Visual Dialog. This will enable the next generation of ~seeing~ AI that can hold a back and-forth conversation with a human and explain itself.Anticipated Outcome. If successful, this proposal will result in creation of new scientific knowledge in the form of novelmodels, algorithms, and implementations for transparent probabilistic intelligent systems. In addition to disseminationthrough published papers and released code, one concrete outcome of this proposal will be a a transparent dialogagent that will allow a human operator to ask question about visual content and be able to explain itself.Impact and DoD Relevance. One key barrier to integration of autonomous and intelligent systems in defense,intelligence, civilian applications is lack of transparency and trust. Intelligent systems that are able to explain theirbeliefs and provide support for their predictions have the potential to fundamentally change the way we live byimproving operator and institutional trust. This research best fits under Office of Naval Research (ONR) Division 311,Machine Learning, Reasoning and Intelligence Program, managed by Dr. Behzad Kamgar-Parsi.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 05, 2017
- Source ID
- N000141712435
Entities
People
- Dhruv Batra
Organizations
- Georgia Tech Research Corporation
- Office of Naval Research
- United States Navy