Synch-Graph : multisensory emotion recognition through neural synchrony via graph convolutional networks
Date
03/04/2020Metadata
Show full item recordAbstract
Human emotions are essentially multisensory, where emotional states are conveyed through multiple modalities such as facial expression, body language, and non-verbal and verbal signals. Therefore having multimodal or multisensory learning is crucial for recognising emotions and interpreting social signals. Existing multisensory emotion recognition approaches focus on extracting features on each modality, while ignoring the importance of constant interaction and co- learning between modalities. In this paper, we present a novel bio-inspired approach based on neural synchrony in audio- visual multisensory integration in the brain, named Synch-Graph. We model multisensory interaction using spiking neural networks (SNN) and explore the use of Graph Convolutional Networks (GCN) to represent and learn neural synchrony patterns. We hypothesise that modelling interactions between modalities will improve the accuracy of emotion recognition. We have evaluated Synch-Graph on two state- of-the-art datasets and achieved an overall accuracy of 98.3% and 96.82%, which are significantly higher than the existing techniques.
Citation
Mansouri Benssassi , E & Ye , J 2020 , Synch-Graph : multisensory emotion recognition through neural synchrony via graph convolutional networks . in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-20) . Proceedings of the AAAI Conference on Artificial Intelligence , AAAI Press , pp. 1351-1358 , Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) , New York , New York , United States , 7/02/20 . https://doi.org/10.1609/aaai.v34i02.5491 conference
Publication
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-20)
ISSN
2159-5399Type
Conference item
Collections
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.