Authors:
Luca Ballan
1
;
2
;
Ombretta Strafforello
3
;
2
and
Klamer Schutte
2
Affiliations:
1
Department of Math, University of Padova, Italy
;
2
Intelligent Imaging, TNO, Oude Waalsdorperweg 63, The Hague, The Netherlands
;
3
Delft University of Technology, The Netherlands
Keyword(s):
Action Recognition, Region Attention, 3D Convolutional Neural Networks, Video Classification.
Abstract:
Long-Term activities involve humans performing complex, minutes-long actions. Differently than in traditional action recognition, complex activities are normally composed of a set of sub-actions, that can appear in different order, duration, and quantity. These aspects introduce a large intra-class variability, that can be hard to model. Our approach aims to adaptively capture and learn the importance of spatial and temporal video regions for minutes-long activity classification. Inspired by previous work on Region Attention, our architecture embeds the spatio-temporal features from multiple video regions into a compact fixed-length representation. These features are extracted with a 3D convolutional backbone specially fine-tuned. Additionally, driven by the prior assumption that the most discriminative locations in the videos are centered around the human that is carrying out the activity, we introduce an Actor Focus mechanism to enhance the feature extraction both in training and i
nference phase. Our experiments show that the Multi-Regional fine-tuned 3D-CNN, topped with Actor Focus and Region Attention, largely improves the performance of baseline 3D architectures, achieving state-of-the-art results on Breakfast, a well known long-term activity recognition benchmark.
(More)