Neuromorphic Cognitive Vision Systems for Aerial Navigation and Reconnaissance
Abstract
Image sensors are common in almost every mobile device. They excel at acquiring and coding video streams at high efficiency on the transmitter side, and at high fidelity in reconstruction at the receiver end. They are the result of extensive engineering optimization in serving applications of visual communication where fidelity in pictorial appearance is a main design criterion. In contrast, emerging applications of navigation, spatial mapping, and visually directed search in autonomous mobile systems on robotic and aerial platforms call for a different type of vision sensor, coding visual information rather than appearance, and optimized for performance and efficiency in detecting and interpreting salient features and key events in the visual scene. Such event-driven vision sensors interface naturally with “spiking” neural networks for visual cognition realized using neuromorphic integrated circuits. This project leads towards practical realization and commercial development of versatile vision sensors facilitating effective and efficient navigation and search in autonomous aerial swarm systems. A first aim of the project is development and application of a neuromorphic vision sensor front-end and video analytics back-end tailored for visually guided navigation and search in aerial contextual robotics. Spatial and temporal visual coding schemes inspired by biological vision systems guide the design of the vision sensor front-end. Performance and efficiency of the front-end integrated with a neuromorphic pattern recognition back-end and mounted on an aerial platform (drone) are evaluated for event detection and object recognition. The aerial vision system realized using custom very-large scale integrated (VLSI) neuromorphic vision chips is benchmarked against emulated sensor front-end and recognition back-end models. These back-end models use commercial vision sensors and equivalent external signal processing to explore the impact of the parameter range on various performance metrics. A second aim of the project is to improve on the effectiveness of the back-end in the absence of direct user supervision or intervention, through on-line deep learning. On-line learning is of paramount importance to autonomous cognitive aerial systems for visual reconnaissance—which must make independent decisions purely based on visual input. To date such developments have been lacking. As such, applications of drones have hitherto been limited mostly to cloud-connected human-assisted functions, except for basic navigation skills. The project thus also targets hardware-software codesign of reconfigurable and custom-VLSI accelerators for on-line learning and inference in deep neural networks, as the backbone for the recognition back-end providing the capacity to autonomously acquiring visual reconnaissance functions in unguided aerial navigation. These accelerators serve as versatile testbeds for rapid prototyping of new on-line learning algorithms for large-scale, low precision (binary), sparse (event-driven) neural network architectures, processing, and learning from continuous visual data streams encountered in complex, continually changing and unpredictable mobile aerial environments. The outcomes of the project may lead to new generations of configurable neuromorphic vision systems that can be parameter tuned in-situ for optimal performance and efficiency, towards aerial swarm systems with unprecedented autonomy and resilience in visually guided navigation and search. These outcomes have the potential to revolutionize applications of aerial swarms, such as fully unattended fleet surveillance, homeland security, border patrol, and rescue missions in harmful environments. To demonstrate the potential for impact of the developed neuromorphic cognitive vision systems on relevant applications, the final validation entails expanded aerial navigation/reconnaissance tasks of interest to unmanned autonomous surveillance and rescue operations.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jul 10, 2018
- Source ID
- N000141812248
Entities
People
- Gert Cauwenberghs
Organizations
- Office of Naval Research
- United States Navy
- University of California, San Diego