AISimEval: Qualifying Simulator Environments for the Testing and Training of Artificial Intelligence Components
Abstract
Nowadays, machine learning components are increasingly used in autonomous systems, such as cars, drones, and vessels. Those vehicles are typically safety-critical systems; thus, it is cru-cial to verify the correctness and trustworthiness of the system systematically. Besides test drives and field tests, the verification and training process is heavily complemented with arti-ficially synthesized data, where sensor inputs of the autonomous vehicle (camera, radar, or sonar image) are created by a simulator tool using the rules of physics. Therefore, the simula-tor can evaluate the correctness of the developed components more cost-effectively, even in underrepresented or potentially dangerous situations.However, the precision of simulation-based testing has yet to be discovered and measured. The behavior of a machine-learning component is not guaranteed to be the same in simulators and real life, which significantly hinders the reliability of any simulator-based testing. There-fore, a rigorous analysis is needed to establish the theoretical foundation of systematic simulator-based testing of autonomous systems.Our research aims to systematically discover and quantify the impact of simulators on the validation process of AI components. This includes the measurement of different simulator qualities (e.g., texture details, realistic environment), weather, and lighting configurations and comparing them to accuracy metrics measured on real-life input data. The long-term goal of this research is to discover any evidence that shows crucial differences between simulators and real-life behavior. Furthermore, a systematic evaluation can prove that in specific conditions the difference is limited, thus the simulators can be used to evaluate the safety metrics of AI components. Therefore, our research can establish the foundation of simulator-based machine learning testing techniques and our findings can be used to drive the development of simulation and testing techniques.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- May 15, 2024
- Source ID
- N629092412063
Entities
People
- Oszkir Semerïth
Organizations
- Office of Naval Research
- United States Navy