Edge Intelligence based Hand Gestures Recognition using Wearable Multimodal

Abstract

Hand gestures recognition (HGR) has achieved great success and opened a new trend in human-machine interaction in recent years. Deployment of some existing HGR systems in practical applications still meets some challenges such as the limited measurable range of sensors; the lack of important information due to the use of a single modality; the high communication cost, latencies, and privacy burdens due to the training of complex deep models. This project aims to overcome these main issues by developing edge intelligence techniques f or hand gesture recognition using wearable multimodal sensors (e.g. accelerometer and camera) with less annotation effort. In this project, we have designed a wearable multimodal prototype that enables the capture of multimodal information such as RGB and motion data. We then designed a set of twelve dynamic hand gestures commonly utilized in the context of human-machine interaction. We collected datasets of such gestures using the designed prototype with fifty subjects in various environmental conditions. To the best of our knowledge, this dataset can be considered the first benchmark dataset for the research community of gesture recognition from wrist-worn multimodal sensors. We deployed various state-of-the-art CNN models for the comparative study of hand gesture recognition using RGB and motion data. The experimental results showed the challenges of the benchmark as well as the best performance of existing models and the room for future improvement. Besides, in the framework of the project, we improved algorithms for hand pose estimation with temporal information and continuous hand gesture recognition. We also conducted fundamental research on shape analysis and Bayesian inference in a hybrid CNN-LSTM model for time-series prediction. We introduced a framework for easy study on federated learning. The prototype and research results have been published in 12 international conferences and submitted to one IEEE Sensor journal.

Open PDF

Document Details

Document Type
Technical Report
Publication Date
Dec 23, 2022
Accession Number
AD1194129

Entities

People

  • Thanh-Hai Tran

Organizations

  • Hanoi University of Science and Technology

Tags

Communities of Interest

  • Autonomy
  • Materials and Manufacturing Processes
  • Sensors

DTIC Thesaurus Topics

  • Artificial Intelligence Software
  • Automata Theory
  • Bayesian Inference
  • Bayesian Networks
  • Computer Languages
  • Computer Vision
  • Convolutional Neural Networks
  • Deep Learning
  • Detection
  • Dimensionality Reduction
  • Human-Machine Interaction
  • Image Processing
  • Information Science
  • Machine Learning
  • Network Science
  • Neural Networks
  • Recurrent Neural Networks
  • Supervised Machine Learning

Fields of Study

  • Computer science

Readers

  • International Relations and Conflict Resolution
  • Neural Network Machine Learning.
  • Software Engineering.

Technology Areas

  • AI & ML
  • AI & ML - Neural Networks