Behavioral Analysis Through Computer Vision: Detecting Emotions and Hand Movements to Aid Mental Health

Aditya Gupta, Geetanjali Bhola

2025

Abstract

This paper presents an advanced behavioral pattern analysis methodology developed for a mental health diagnosis task using the CNN model with an additional Attention Layer constituting an enhanced VGG Model. To improve combined performance, a Transformer has been integrated into the CNN. The VGG-Transformer model fine-tunes and trains over 3,500 images from the dataset available in the RAF-DB dataset for optimal feature extraction and classification. Our sophisticated model uses a repository called DeepFace, with a VGG model in it, followed by the MediaPipe library to process video inputs from the user. The system accurately recognises various emotions and hand movements from frames, with a final great training accuracy of 94.33% and validation accuracy of 95.76%. The DeepFace integration supports detail and precision in the recognition of facial emotions, while MediaPipe hand tracking enables deep hand movement analysis. Such a complementary approach will make one delve into understanding behavior in depth, where the results can aid in detecting the possibility of mental health-related problems and supporting therapists in their diagnosis and treatment. This work has proven that state-of-the-art machine learning in mental health assessment is required and has been proven effective by combining the enhanced VGG16 Model and Transformers, DeepFace, and Medi-aPipe specialised libraries.

Paper Citation

in Harvard Style

Gupta A. and Bhola G. (2025). Behavioral Analysis Through Computer Vision: Detecting Emotions and Hand Movements to Aid Mental Health. In Proceedings of the 3rd International Conference on Futuristic Technology - Volume 1: INCOFT; ISBN 978-989-758-763-4, SciTePress, pages 671-682. DOI: 10.5220/0013583500004664

in Bibtex Style

@conference{incoft25,
author={Aditya Gupta and Geetanjali Bhola},
title={Behavioral Analysis Through Computer Vision: Detecting Emotions and Hand Movements to Aid Mental Health},
booktitle={Proceedings of the 3rd International Conference on Futuristic Technology - Volume 1: INCOFT},
year={2025},
pages={671-682},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013583500004664},
isbn={978-989-758-763-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Conference on Futuristic Technology - Volume 1: INCOFT
TI - Behavioral Analysis Through Computer Vision: Detecting Emotions and Hand Movements to Aid Mental Health
SN - 978-989-758-763-4
AU - Gupta A.
AU - Bhola G.
PY - 2025
SP - 671
EP - 682
DO - 10.5220/0013583500004664
PB - SciTePress