Using Neural-Network-Driven Image Recognition Software to Detect Emotional Reactions in the Face of a Player While Playing a Horror Video Game
Abstract
Image analysis of the webcam images of the player playing horror video games is used to infer his/her emotional state. We initially relied on the received wisdom that there are three distinguishable emotional states: joy, agitation (a combination of fear and surprise) and neutral.
We applied modern Image analysis techniques to extract the player's face in three sets of 50 consecutive frames by first generating 150 feature vectors and then using an auto-encoder neural network to dimension-reduce these to a mapping of 150 points on a 2D manifold. We find that these mapped points are not uniformly distributed in a plane and an overlap between neutral and each of the two affective states is highly improbable.
We derive convex hulls using a k-medoids clustering algorithm; they seem to indicate a difference between the mouth region expression and that around the eyes. We use kernel density estimation and Monte Carlo methods to determine that the affective states joy and agitation have a small probability of overlap (6.6% and 10.6%, respectively), which can be interpreted as a confusion probability.
Our advanced statistical analysis thus confirms that modern image analysis can indeed be used to monitor the player's emotional state while playing.