Authors:
Hmamouche Youssef
1
;
2
;
Ochs Magalie
2
;
Prévot Laurent
1
;
3
and
Chaminade Thierry
4
Affiliations:
1
Aix Marseille Université, CNRS, LPL, UMR7309, Aix-en-Provence, France
;
2
Aix Marseille Université, Université de Toulon, CNRS, LIS, UMR7020, Marseille, France
;
3
Institut Universitaire de France, Paris, France
;
4
Aix Marseille Université, CNRS, INT, UMR7289, Marseille, France
Keyword(s):
Multimodal Signals Processing, Conversation, Machine Learning, Human-human and Human-machine Interactions, Functional MRI.
Abstract:
Whole-brain neuroimaging using functional Magnetic Resonance Imaging (fMRI) provides valuable data to localize brain activity in space and time. Here, we use a unique corpus including fMRI and behavior recorded when participants discussed with a human or a conversational robot. Temporal dynamic is crucial when studying conversation, yet identifying relationship between the participants’ behavior and their brain activity is technically challenging given the time resolution of fMRI. We propose here an approach developed to extract neurophysiological and behavioral time-series from the corpus and analyse their causal relationships. Preprocessing entails the construction of discrete neurophysiological time-series from functionally well defined brain areas, as well as behavioral time-series describing higher-order behaviors extracted from synchronized raw audio, video and eyetracking recordings. The second step consists in applying machine learning models to predict brain activity on the
basis of various aspects of behavior given knowledge about the functional role of the areas under scrutiny. Results demonstrate the specificity of the behaviors allowing the predictions of the activity in functional brain areas.
(More)