Synch-Graph : multisensory emotion recognition through neural synchrony via graph convolutional networks
MetadataShow full item record
Altmetrics Handle Statistics
Altmetrics DOI Statistics
Human emotions are essentially multisensory, where emotional states are conveyed through multiple modalities such as facial expression, body language, and non-verbal and verbal signals. Therefore having multimodal or multisensory learning is crucial for recognising emotions and interpreting social signals. Existing multisensory emotion recognition approaches focus on extracting features on each modality, while ignoring the importance of constant interaction and co- learning between modalities. In this paper, we present a novel bio-inspired approach based on neural synchrony in audio- visual multisensory integration in the brain, named Synch-Graph. We model multisensory interaction using spiking neural networks (SNN) and explore the use of Graph Convolutional Networks (GCN) to represent and learn neural synchrony patterns. We hypothesise that modelling interactions between modalities will improve the accuracy of emotion recognition. We have evaluated Synch-Graph on two state- of-the-art datasets and achieved an overall accuracy of 98.3% and 96.82%, which are significantly higher than the existing techniques.
Mansouri Benssassi , E & Ye , J 2020 , Synch-Graph : multisensory emotion recognition through neural synchrony via graph convolutional networks . in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-20) . Proceedings of the AAAI Conference on Artificial Intelligence , AAAI Press , pp. 1351-1358 , Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) , New York , United States , 7/02/20 . https://doi.org/10.1609/aaai.v34i02.5491conference
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-20)
Copyright © 2020 Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. This work has been made available online in accordance with publisher policies or with permission. Permission for further reuse of this content should be sought from the publisher or the rights holder. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1609/aaai.v34i02.5491.
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.