Multimodal Neural Network for Sentiment Analysis in Embedded Systems

Quentin Portes, José Carvalho, Julien Pinquier, Frédéric Lerasle

Abstract

Multimodal neural network in sentiment analysis uses video, text and audio. Processing these three modalities tends to create computationally high models. In the embedded context, all resources and specifically computational resources are restricted. In this paper, we design models dealing with these two antagonist issues. We focused our work on reducing the numbers of model input features and the size of the different neural network architectures. The major contribution in this paper is the design of a specific 3D Residual Network instead of using a basic 3D convolution. Our experiments are focused on the well-known dataset MOSI (Multimodal Corpus of Sentiment Intensity). The objective is to perform similar results as the state of the art. Our best multimodal approach achieves a F1 score of 80% with a number of parameters reduced by 2.2 and the memory load reduced by a factor 13.8, compared to the state of the art. We designed five models, one for each modality (i.e video, audio and text) and one for each fusion technique. The two high-level multimodal fusions presented in this paper are based on the evidence theory and on a neural network approach.

Download


Paper Citation


in Harvard Style

Portes Q., Carvalho J., Pinquier J. and Lerasle F. (2021). Multimodal Neural Network for Sentiment Analysis in Embedded Systems.In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, ISBN 978-989-758-488-6, pages 387-398. DOI: 10.5220/0010224703870398


in Bibtex Style

@conference{visapp21,
author={Quentin Portes and José Carvalho and Julien Pinquier and Frédéric Lerasle},
title={Multimodal Neural Network for Sentiment Analysis in Embedded Systems},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP,},
year={2021},
pages={387-398},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010224703870398},
isbn={978-989-758-488-6},
}


in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP,
TI - Multimodal Neural Network for Sentiment Analysis in Embedded Systems
SN - 978-989-758-488-6
AU - Portes Q.
AU - Carvalho J.
AU - Pinquier J.
AU - Lerasle F.
PY - 2021
SP - 387
EP - 398
DO - 10.5220/0010224703870398