Assured Dependability for Autonomous Systems
Abstract
Sequential diagnostic, operational, and acceptance testing is the normal approach to assuring system developers and users that any kind of system will perform its mission safely, effectively, and reliably. The dependability of an autonomous system, however, depends on the systems decision processes, which interpret sensor data, model the environment, consider mission goals and priorities, choose courses of action, observe outcomes, and potentially modify the systems own logic over time through post-fielding learning. The normal testing approach cannot possibly effectively test, evaluate, verify, and validate system behavior in every decision context an autonomous system could face. A different approach is needed to assure developers, operators, and commanders that autonomous systems will perform dependably in situations that may differ significantly from any that were tested explicitly prior to fielding. Autonomous systems rely on successful integration of many enabling technologies to be dependable. These technologies can include computer vision, sensor fusion, knowledge representation, expert systems, inference engines, path planning, optimization, machine learning, and others. Autonomous systems that team with humans also depend on detailed concepts of operations for how the humans and the machines will interact. All of these represent ways a systems dependability could be threatened; the inputs to each enabling technology generate a novel attack surface, in addition to the usual cybersecurity attack surfaces of advanced systems. For establishing dependability, the environment might as well be an adversary. All of the attack surfaces of autonomous systems can lead to undependable behavior. Attacks could be generated by an adversary or by a complex environment and could involve denial of information (jamming), misleading inputs (spoofing), unauthorized control (hacking), or threats of physical harm (mugging).
Document Details
- Document Type
- Technical Report
- Publication Date
- May 01, 2019
- Accession Number
- AD1122236
Entities
People
- David F Tate
Organizations
- Institute for Defense Analyses