Moral Justification to Foster Human-Machine Trust
Abstract
Robots and artificial intelligence (often called "autonomous agents") are increasingly used in domains such as space operations, cyber defense, disaster response, and medical care, and are envisioned to directly collaborate with human partners in ways that resemble human-human teams. To illustrate, all human communities, groups, and teams have norms that influence and regulate behavior, so autonomous agents that join these communities must be responsive to norms as well-they must know and follow the norms of their community. But even if we succeed in giving autonomous agents such norm competence, we are faced with a significant challenge: Norms can conflict with each other. Examples of such conflicts have been discussed for self-driving cars, military devices, and medical robot assistants. The only way to resolve conflicts between norms is by deciding to uphold one norm-the more important one-and to violate the other, less important norm. This means that whenever an agent (human or machine) resolves a norm conflict, it must commit a norm violation. People respond to such violations with moral disapproval and loss of trust. In this project we investigate one powerful tool humans use-and autonomous agents should use-to mitigate such moral disapproval and repair lost trust: justifications. When an agent must violate a norm in order to resolve a norm conflict, a justification explains why the agent did act in this way and why anybody that shares the community's norms should act in this way. Justifications clarify that the violation of one norm was socially and morally justified because it upheld the other, more important norm. In a series of experiments, we will demonstrate that, after resolving a norm conflict and committing a norm violation, an autonomous agent that justifies its actions-similar to a human who does so-will reduce the moral disapproval and repair the loss of trust that normally result from norm violations.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 21, 2022
- Source ID
- FA95502110359XX0
Entities
People
- Elizabeth Phillips
Organizations
- Air Force Office of Scientific Research
- George Mason University
- United States Air Force