Edge‑Enabled Continuous Multi‑Modal Deep Learning Framework for Robust Real‑Time Sign Language Recognition to Empower Inclusive Communication with the Deaf Community

Sivakumar Ponnusamy, G. Visalaxi, S. Sureshkumar, Lokasani Bhanuprakash, Sibisaran S.

2025

Abstract

This study introduces an edge-enabled, continuous multi-modal deep learning framework designed to deliver robust real-time sign language recognition, transforming the communication experience for the Deaf community. By integrating RGB video, depth sensing, and skeletal key point inputs into a unified convolutional-transformer architecture deployed on lightweight edge devices, the system ensures low-latency inference and high accuracy under diverse lighting and background conditions. Continuous gesture segmentation and adaptive fusion strategies enhance the recognition of dynamic signs and phrases, while on-device processing preserves user privacy and enables offline operation. Extensive evaluations on benchmark datasets and in-field trials demonstrate significant improvements in vocabulary coverage, speed, and resilience to occlusion compared to prior approaches. The proposed framework paves the way for more inclusive and accessible human computer interfaces that bridge communication gaps in real-world settings.

Paper Citation

in Harvard Style

Ponnusamy S., Visalaxi G., Sureshkumar S., Bhanuprakash L. and S. S. (2025). Edge‑Enabled Continuous Multi‑Modal Deep Learning Framework for Robust Real‑Time Sign Language Recognition to Empower Inclusive Communication with the Deaf Community. In Proceedings of the 1st International Conference on Research and Development in Information, Communication, and Computing Technologies - Volume 1: ICRDICCT`25; ISBN 978-989-758-777-1, SciTePress, pages 20-26. DOI: 10.5220/0013856900004919

in Bibtex Style

@conference{icrdicct`2525,
author={Sivakumar Ponnusamy and G. Visalaxi and S. Sureshkumar and Lokasani Bhanuprakash and Sibisaran S.},
title={Edge‑Enabled Continuous Multi‑Modal Deep Learning Framework for Robust Real‑Time Sign Language Recognition to Empower Inclusive Communication with the Deaf Community},
booktitle={Proceedings of the 1st International Conference on Research and Development in Information, Communication, and Computing Technologies - Volume 1: ICRDICCT`25},
year={2025},
pages={20-26},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013856900004919},
isbn={978-989-758-777-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 1st International Conference on Research and Development in Information, Communication, and Computing Technologies - Volume 1: ICRDICCT`25
TI - Edge‑Enabled Continuous Multi‑Modal Deep Learning Framework for Robust Real‑Time Sign Language Recognition to Empower Inclusive Communication with the Deaf Community
SN - 978-989-758-777-1
AU - Ponnusamy S.
AU - Visalaxi G.
AU - Sureshkumar S.
AU - Bhanuprakash L.
AU - S. S.
PY - 2025
SP - 20
EP - 26
DO - 10.5220/0013856900004919
PB - SciTePress