Performance Evaluation of Sensors on Mobile Vehicles Using a Large Data Repository and Ground Truth
Abstract
Progress in algorithm development and transfer of results to practical applications such as military robotics requires standard qualitative and quantitative measurements for performance evaluation and validation. Although the evaluation and validation of algorithms have been discussed for well over a decade, the research community still faces a lack of well-defined and standardized methodology. In this research, we describe three methods for creating ground truth databases and benchmarks using multiple sensors, for use in mobile robotics. The databases and benchmarks provide researchers with high quality data from suites of sensors operating in complex environments representing real-world problems. At NIST, we have equipped a High Mobility Multi-purpose Wheeled Vehicle (HMMWV) with a suite of sensors including a Riegl ladar, GDRS ladar, stereo CCD, several color cameras, Global Positioning System (GPS), Inertial Navigation System (INS), pan/tilt encoders, and odometry. All sensors are calibrated and registered with each other in space and time. This allows a database of features and terrain elevation to be built. Ground truth information is collected through aerial surveys, from maps, by human annotation, and by previous traverses of the terrain by the vehicle. Ground truth may include terrain elevation information, feature information (roads, road signs, trees, ponds, fences, etc.) and constraint information (e.g., one-way streets). We have implemented our a priori database using One Semi- Automated Forces (OneSAF), a military simulation environment. Using the Inertial Navigation System and Global Positioning System (GPS) on the HMMWV to provide indexing into the database, we extract all the elevation and feature information for a region surrounding the vehicle as it moves about the NIST campus. Ground truth for each sensor can be obtained by projecting this information into the sensors' coordinate systems.
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 01, 2003
- Accession Number
- ADA515319
Entities
People
- Ayako Takeuchi
- Gerry Cheok
- Harry Scott
- Michael Shneier
- Tommy Chang
- Tsai Hong
Organizations
- National Institute of Standards and Technology