Autonomy and Artificial Intelligence Test
Abstract
The Autonomy and Artificial Intelligence Test (AAIT) Project continued test technology development supporting testers in the DoD of Unmanned and Artificial Intelligence-Based Systems. AAIT develops technology to improve ability to develop salient and high-value test plans, increasing safety during live test, to identify safety defects deep inside complex autonomy software, and to improve performance of machine vision systems. The AAIT project collaborated with the Autonomy Community of Interest (COI) Test and Evaluation, Verification and Validation (TEVV) Working Group to ensure that the AAIT project is investing in technologies relevant to the future of autonomous systems. The AAIT Project seeks solutions for legacy topics (test planning, test execution, safety, and performance assessment) but has also expanded interest to find solutions for Artificial Intelligence and Machine Learning systems, topics identified by the intelligence community, and any other topics that are priority for TRMC and OUSD(R&E). The AAIT project continued the Assured DevSecOps of Autonomous Systems (ADAS) effort. ADAS addresses the unique challenges of Autonomy test & evaluation to provide enterprise solutions in support of future programs and joint initiatives. ADAS addresses autonomy test and evaluation verification and validation (TEV&V) needs across the life cycle beginning with mission analysis and engineering and ending with the mission operations. ADAS is a leading pathfinder effort to address gaps identified by the National Security Commission on Artificial Intelligence. The AAIT Project continued investments in robustness testing technology to detect and predict safety-related vulnerabilities and failures within UAS software, in advance of live test. The AAIT project provided the key S&T technology as a basis for the Navy-led CTEIP, “Autonomy, Integration, and Teaming” (AIT), which developed test capabilities to be demonstrated on the Airborne Collision Avoidance System (ACAS-Xu) on Triton, and as a basis for Guardian, a Ground Based Detect and Avoid system, which will allow UAS to achieve certification for use during live test (DO-278A/NAVAIR Cert). The same core technologies are used as a basis for the Army-led CTEIP “Autonomous Systems Test Capability” (ASTC). The AAIT project give testers a more comprehensive means of identifying and reporting on safety vulnerabilities found deep within the UAS software, allowing testers to test for defects that may not have ever been found by traditional testing techniques. The AAIT Project completed development of test technology to improve test planning for surface, sub-surface, ground, and airborne autonomy using optimization algorithms to rapidly generate salient test scenarios. The AAIT project provided the key S&T technology (for test planning) as a basis for the Navy-led CTEIP, “Autonomy, Integration, and Teaming” (AIT). The same core technologies are used as a basis for the Army-led CTEIP “Autonomous Systems Test Capability” (ASTC). The AAIT project, via the CTEIP programs, give testers information about how to choose high-value test conditions. AAIT technology shows exactly where software-based systems are on a performance edge (between mission success and mission failure) and a safety edge (between safety success and safety failure). AAIT helps testers see critical test conditions that they might not have chosen by traditional means. The AAIT Project initiated development of technology to create machine-learned, behavioral copies of autonomy software. This technology creates faster-than-real-time versions of a given autonomy that can then be tested in an accelerated timeline in a simulated environment, and can also be cloned to be tested in parallel-processing fashion. This technology will provide faster, better, and more statistically significant testing data for testers. This technology can also capture human performance, for example a pilot, or a ground radar operator) to be used as more realistic elements of a simulated environment. The AAIT project developed machine vision test technologies to identify where a machine vision system shows brittleness – inconsistent identification – of elements in its field of view. This technology can be used to improve performance of machine vision systems by identifying test data (images or video) to be used for focused testing and also can be used to re-train a brittle system for improved performance. The AAIT Project developed technology to use functional architecture data to identify safety faults, and build safety fault trees) for complex autonomy software systems. Fault tree development has been traditionally built by hand. This technology will identify faults and build a fault tree more comprehensively and thoroughly than humanly possible, saving resources and improving the identification of safety risks in advance of live test. The AAIT Project developed technology to assist with the validation and verification of a learning-in-the-field AI-based system. This technology will assist testers by advising when a learning system has learned sufficiently different information to the point where it is no longer valid for use. This technology can also be used to determine if a system trained in one domain (urban, for example) is valid for use in another domain (desert). The AAIT project initiated technology development to support AI hubs verification, validation, test and evaluation.
Document Details
- Document Type
- Accomplishment
- Publication Date
- Oct 01, 2025
- Source ID
- 0496534dcbe1af0740b2cbf08213cd8a