What Drives Human Behaviour? Multimodal Eye Fixation Related Brain State Analysis for Attention and Emotion Monitoring
Abstract
Peripheral physiological measures such as electrodermal activity (EDA), heart rate and pupil dilation, as well as neurophysiological measures such as electroencephalography (EEG), can inform us about individuals' cognitive and emotional state. We are interested in exploiting such measures in real life situations. A challenge of interpreting physiological measures as markers of mental state in real life is the lack of context information. We here approach this challenge by relating physiological measures to eye tracking. Participants scanned stimuli that induced different levels of workload (small sets of numbers that needed to be added or not) and different types of emotion (neutral, pleasant and unpleasant pictures). EDA, heart rate, pupil size and EEG were related to the first eye fixation on the stimulus. For peripheral measures, response traces across the following 10s were determined and signal amplitudes were compared between the different types of stimuli. EEG signals were compared for the different types of stimuli in the time interval from fixation onset to 1500 ms later using a cluster-based, nonparametric randomization approach. For the peripheral measures, high workload stimuli stood out from all other stimuli in all modalities, with patterns as expected from literature under more traditional experimental conditions: high values of EDA, heart rate, and pupil size for high compared to low workload stimuli. For emotional stimuli, peripheral physiological effects tended to be in the expected direction but were more modest in size. In the EEG signals, a significant late parieto-occipital cluster could be identified with higher amplitudes for high compared to low workload stimuli, as well as for emotional stimuli compared to the neutral stimuli. In future analyses we will combine fixation-locked signals from different modalities to detect mental states elicited by information that is being looked at.
Document Details
- Document Type
- Technical Report
- Publication Date
- Aug 06, 2021
- Accession Number
- AD1145329
Entities
People
- Anne-marie Brouwer
- D Piechnik
- G J Funke
- I V Stuldreher
- K Lingelbach
- M T Tolston
- M Vukelic
- S Huertas Penen