Authors:
Keisuke Yamazaki
1
;
Satoshi Tamura
2
;
Yuuto Gotoh
3
and
Masaki Nose
3
Affiliations:
1
Graduate School of Natural Science and Technology, Gifu University, Gifu, Japan
;
2
Faculty of Engineering, Gifu University, Gifu, Japan
;
3
Ricoh Company, Ltd., Kanagawa, Japan
Keyword(s):
Voice Activity Detection, Human Motion, Speaker Diarization, Dynamic Image, Multi-modal Transfer Module, Conference Video Processing.
Abstract:
In this paper, we propose a visual-only Voice Activity Detection (VAD) method using human movements. Although audio VAD is commonly used in many applications, it has a problem it is not robust in noisy environments. In such the cases, multi-modal VAD using speech and mouth information is effective. However, due to the current pandemic situation, people wear masks causing we cannot observe mouths. On the other hand, utilizing a video capturing the entire of a speaker is useful for visual VAD, because gestures and motions may contribute to identify speech segments. In our scheme, we firstly obtain dynamic images which represent motion of a person. Secondly, we fuse dynamic and original images using Multi-Modal Transfer Module (MMTM). To evaluate the effectiveness of our scheme, we conducted experiments using conference videos. The results show that the proposed model has better than the baseline. Furthermore, through model visualization we confirmed that the proposed model focused much
more on speakers.
(More)