A Multimodal Attention-Based Deep Learning Framework for Real-Time Activity Recognition at the Edge
Abstract
To help enable real-time activity recognition at the edge, hardware accelerators for activity recognition system can play a vital role. Since modern human activity recognition systems are deep learning-based, hardware accelerators can accelerate the DNN layers of human activity recognition system, which are the most compute-intensive part of an activity recognition system. Hardware acceleration for activity recognition is a new area. These couple of works consider a simplistic activity recognition architecture, and accommodate either only a single or two sensor modalities, and thus are not suitable for human activity recognition with continuous time series data from multiple sensor modalities and in different environments and conditions. Therefore, there is a need to design hardware accelerators for multimodal human activity recognition systems comprising of stacked CNN and RNN layers. Real-time human activity recognition from diverse sensor modalities at the edge under varying environmental and-or lighting conditions is an open research problem. To overcome the deficiencies in existing works in this area, this project proposes a deep learning-based framework for realtime human activity recognition at the edge under varying conditions by leveraging various sensor modalities and an attention-based mechanism to fuse sensor data.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Mar 07, 2023
- Source ID
- FA95502210040
Entities
People
- Arslan Munir
Organizations
- Air Force Office of Scientific Research
- Kansas State University
- United States Air Force