Level of Rigor for Artificial Intelligence Development

Abstract

Throughout the Department of Defense (DoD), artificial intelligence (AI)/machine learning (ML) integration is currently in the form of upgrades to existing programs or new program acquisitions. How do we know these AI/ML-enabled systems will perform as intended? To make this determination, AI/ML product development/acquisition needs a unique evaluation process compared to other traditional software development/acquisition projects. In response, the Naval Ordnance Safety and Security Activity (NOSSA) funded the following research to investigate unique policies, guidelines, tools, and techniques to assess safety concerns within identified AI/ML critical functions. From that effort, 14 key Level of Rigor (LOR) tasks were developed and applied across five stages: (1) Requirements, (2) Architecture, (3) Algorithm Design, (4) Algorithm Code, and (5) Test and Evaluation (T and E). The 14 LOR tasks involve best practice discussions, definitions, measurements, justification documentation, and hazard analysis formats specific to AI/ML systems. These 14 LOR tasks also provide clarity as to why AI/ML software development requires special consideration by the acquisition community. Moreover, this research has the potential of affecting the acquisition community with regard to how they define requirements, create architecture, produce AI/ML algorithm designs, develop AI/ML algorithm code, and perform T and E. The need for the 14 LOR tasks across five stages of development became evident during the development of an acquisition sandbox that researched a route planner deploying AI/ML autonomous systems, as well as having those systems deliver packages with a focus on assessing safety in critical function behavior. The sandbox was designed using a DoD Architecture Framework (DoDAF) and Unified Modeling Language (UML) diagrams that incorporate a variety of AI/ML techniques.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Apr 01, 2022
Accession Number
AD1173626

Entities

People

  • Bruce Nagy

Organizations

  • Naval Air Warfare Center, Weapons Division

Tags

Communities of Interest

  • Autonomy
  • C4I
  • Cyber
  • Electronic Warfare
  • Energy and Power Technologies
  • Materials and Manufacturing Processes
  • Weapons Technologies

DTIC Thesaurus Topics

  • Artificial Intelligence
  • Artificial Intelligence Software
  • Automata Theory
  • Computer Languages
  • Computer Programming
  • Computer Vision
  • Computers
  • Data Curation
  • Data Mining
  • Data Science
  • Deep Learning
  • Game Theory
  • Information Science
  • Machine Learning
  • Network Science
  • Neural Networks
  • Systems Engineering
  • Test And Evaluation

Fields of Study

  • Computer science

Readers

  • Geospatial Intelligence and Artificial Intelligence Analytics
  • Logistics and Supply Chain Management.
  • Software Engineering.

Technology Areas

  • AI & ML
  • AI & ML - DoD AI Strategy
  • AI & ML - Neural Networks
  • Autonomy