Deep Set Conditioned Latent Representations for Action Recognition

Akash Singh, Tom de Schepper, Kevin Mets, Peter Hellinckx, José Oramas, Steven Latré

2022

Abstract

In recent years multi-label, multi-class video action recognition has gained significant popularity. While reasoning over temporally connected atomic actions is mundane for intelligent species, standard artificial neural networks (ANN) still struggle to classify them. In the real world, atomic actions often temporally connect to form more complex composite actions. The challenge lies in recognising composite action of varying durations while other distinct composite or atomic actions occur in the background. Drawing upon the success of relational networks, we propose methods that learn to reason over the semantic concept of objects and actions. We empirically show how ANNs benefit from pretraining, relational inductive biases and unordered set-based latent representations. In this paper we propose deep set conditioned I3D (SCI3D), a two stream relational network that employs latent representation of state and visual representation for reasoning over events and actions. They learn to reason about temporally connected actions in order to identify all of them in the video. The proposed method achieves an improvement of around 1.49% mAP in atomic action recognition and 17.57% mAP in composite action recognition, over a I3D-NL baseline, on the CATER dataset.

Paper Citation

in Harvard Style

Singh A., de Schepper T., Mets K., Hellinckx P., Oramas J. and Latré S. (2022). Deep Set Conditioned Latent Representations for Action Recognition. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP; ISBN 978-989-758-555-5, SciTePress, pages 456-466. DOI: 10.5220/0010838400003124

in Bibtex Style

@conference{visapp22,
author={Akash Singh and Tom de Schepper and Kevin Mets and Peter Hellinckx and José Oramas and Steven Latré},
title={Deep Set Conditioned Latent Representations for Action Recognition},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP},
year={2022},
pages={456-466},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010838400003124},
isbn={978-989-758-555-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP
TI - Deep Set Conditioned Latent Representations for Action Recognition
SN - 978-989-758-555-5
AU - Singh A.
AU - de Schepper T.
AU - Mets K.
AU - Hellinckx P.
AU - Oramas J.
AU - Latré S.
PY - 2022
SP - 456
EP - 466
DO - 10.5220/0010838400003124
PB - SciTePress