Integrative Multimodal Communication for ALS Patients Using iPad
Abstract
Amyotrophic lateral sclerosis (ALS) is a disease that causes progressive degeneration of the motor neurons and eventually the death of neurons controlling voluntary muscles. As the motor and speech capabilities of a patient at different stages evolves, assistive communication is essential. However, the available capabilities of an ALS patient differ from each other and keep on evolving when the disease progresses. While there is a large space of assistive technologies, they come with major limitations. First, most devices are based on one main technology, such as eye gaze or brain - computer interfaces (BCI), and they may not be able to use all the capabilities of an ALS patient for optimized communication. For example, facial expressions, such as head movement, mouth/jaw movement and blinking, can provide additional inputs for improving communications. A patient may also find one type of communication more comfortable than the other, e.g., BCI vs gaze. There is a lack of a single device or app that can integrate and take full advantages of all the capabilities of a patient. Also, many existing technologies come with a high price tag, ranging from thousands to tens of thousands of dollars, which creates a major hurdle for patients, in particular, for those with socioeconomic disadvantage or those with limited health insurance. Besides, for patients, the communication devices often look different, are big or cumbersome, and may call attention to the patient s disability. The caregiver may not be technology-savvy to set up or program such devices. For example, current BCI-based assistive communication often relies on a complicated device setup with a computer, monitor, software, and complicated operation instructions, and the sophisticated EEG cap placement can also be a major challenge for caregivers. The goal of this proposed project is to develop a lightweight, cost-effective, multimodal-based assistive communication app, EyeCanDo, running on iPad, with an optional consumer-grade wireless EEG headset, to take full advantage of the available capabilities of an ALS patient, including eye gaze, facial expressions, and BCI, for optimal communication and improving the quality of life. A user will be able to download EyeCanDo directly from Apple App Store to an iPad Pro and begin to use the app immediately. EyeCanDo is built upon Apple s Augmented Reality technology, which captures and models a human s face and produces data about eye gaze and facial expressions. Through a statistical learning-based approach, EyeCanDo will fully utilize all the user inputs including gaze and facial expressions for improved performance, boosting both speed and accuracy. We also propose a machine learning-based approach using a user s actions and experiences as feedback to automatically adjust to the user to reduce error rate and the selection time. With full Web access and social network support, EyeCanDo will also engage patients for accessing more information and using different communication channels to seek social support. We will further integrate an alternative BCI-based communication in the same app based on P300, a brainwave component that occurs after a stimulus that is deemed important, such as a flashing character of interest. P300-based typing has the benefit of ease of use, does not require precise eye movement, and is less tiring than gaze. We will explore two alternative consumer-grade EEG headsets, which have been well-evaluated for detecting reliable P300 signals in research. Through a lightweight, deep learning-based approach, the app will provide improved detection of a P300 signal on the iPad, taking advantage of Apple s machine learning framework. The project will be built upon our current EyeCanDo app, which provides the core functionalities and framework for gazed-based communication and has been evaluated with 18 ALS patients at Stony Brook Medicine ALS Clinic. During the course of the project
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Dec 28, 2022
- Source ID
- W81XWH2210408
Entities
People
- Fusheng Wang
Organizations
- Stony Brook University
- United States Army