Emotion Classification from Gaze Data Alone: A Deep Learning Approach without Pupil Diameter

Abstract

There is active research on estimating emotions by analyzing facial expressions and pupil responses using facial images. Due to fake facial expressions or changes in environmental brightness, it may not be possible to estimate emotions from facial expressions or pupil diameter. Therefore, we propose a method to estimate emotions from gaze alone. This study proposes a method to classify emotions by fine-tuning gazeNet, a gaze classification model into fixation, saccade, etc. to compensate for the lack of data for deep learning. We evaluated the proposed methods in three ways, classifying emotions into four categories (delighted, relaxed, bored, and afraid), two categories (negative and positive valence), and two categories (low and high arousal), which are based on Russell’s circumplex model. The result of the four-category classification shows that emotions associated with higher arousal, i.e., delighted and afraid, exhibited higher accuracy. Classification along the valence axis suggests that valence classification using gaze data alone is difficult. Classification along the arousal axis suggests that high-arousal states achieve higher recall, precision, and F-measure compared to low-arousal states. In this study, we showed that emotion classification from gaze alone using transfer learning with a pre-trained model that classifies gaze was effective on the arousal axis.

Junichi Nagasawa

Contract Researcher

My research interests include Human-Computer Interaction, Security, eye tracking.