Dataflow-based Design for Real-Time Scene Understanding at the Edge
Abstract
Real-time scene understanding based on distributed sensor data is required to enable effective decision making in complex, dynamically changing environments. For example, effective sensor networks for scene understanding have important uses for enhancing situational awareness and supporting multi-domain operations in dense urban environments, which are known to be highly dynamic during combat. Motivated by this need, the main objective of this project is to develop streamlined deep learning based approaches that can provide combat arms units in tactical operations with enhanced situational understanding by capturing images and videos using resource-constrained sensing platforms, and then processing the data streams locally with minimal use of a back-end data center. The project will contribute to advancing situational awareness capabilities using multiple video streams that are acquired from different perspectives through distributed, embedded imaging platforms. Effective co-design of algorithms and embedded software is critical in the deployment of scene understanding capabilities across distributed sensor networks. The algorithms employed and their efficient realization in embedded software have a major impact on system-level trade-offs involving knowledge extraction accuracy, latency in adapting to environmental changes, and energy efficiency. In addition to managing these complex trade-offs, system designers for such applications often face very strict constraints on processing capability and memory availability. To address these challenges, this project will investigate the co-design of deep learning architectures, and design optimization techniques for embedded software that enable efficient deployment of these architectures in resource-constrained, mission-critical camera networks. An important direction of emphasis and novelty in this research is the dynamic adaptation of jointly optimized algorithm configurations and embedded software configurations. Such adaptation is critical to ensuring robust and reliable operation in the complex, dynamic environments in which mission critical scene understanding networks are often deployed. A key aspect of the proposed technical approach is the application of dataflow-based design methodologies for signal and information processing systems. The rigorous integration of dataflow-based design optimization techniques with neural networks for multi-view object detection is a major direction of novelty in the proposed work. Dataflow methods will be applied in the project as a formal foundation with which to systematically integrate algorithm-related and implementation-related concerns so as to facilitate co-design of algorithm- and implementation-level optimizations. The results of the research will include models and design optimization algorithms for adaptive signal and information processing that enable autonomous adaptation of embedded systems for robust scene understanding in uncertain environments. The results will also include algorithms for applying deep learning to perform accurate, adaptive scene understanding with limited computational requirements, and with integrated parameterization to enable useful and diverse trade-offs between computational cost and knowledge extraction accuracy.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jun 25, 2021
- Source ID
- W911NF2110258
Entities
People
- Shuvra Bhattacharyya
Organizations
- Army Contracting Command
- United States Army
- University of Maryland