Hands-Free VR

Jorge Askur Vazquez Fernandez, Jorge Askur Vazquez Fernandez, Jae Joong Lee, Santiago Andrés Serrano Vacca, Alejandra J. Magana, Radim Peša, Bedrich Benes, Voicu Popescu

2025

Abstract

We introduce Hands-Free VR, a voice-based natural-language interface for VR that allows interaction without additional hardware just using voice. The user voice command is converted into text using a fine-tuned speechto-text deep-learning model. Then, the text is mapped to an executable VR command using an LLM, which is robust to natural language diversity. Hands-Free VR was evaluated in a within-subjects study (N = 22) where participants arranged objects using either a conventional VR interface or Hands-Free VR. The results confirm that Hands-Free VR is: (1) significantly more efficient than conventional VR interfaces in task completion time and user motion metrics; (2) highly rated for ease of use, intuitiveness, ergonomics, reliability, and desirability; (3) robust to English accents (20 participants were non-native speakers) and phonetic similarity, accurately transcribing 96.7% of voice commands, and (3) robust to natural language diversity, mapping 97.83% of transcriptions to executable commands.

Paper Citation

in Harvard Style

Vazquez Fernandez J., Lee J., Serrano Vacca S., Magana A., Peša R., Benes B. and Popescu V. (2025). Hands-Free VR. In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: HUCAPP; ISBN 978-989-758-728-3, SciTePress, pages 533-542. DOI: 10.5220/0013115100003912

in Bibtex Style

@conference{hucapp25,
author={Jorge Vazquez Fernandez and Jae Lee and Santiago Serrano Vacca and Alejandra Magana and Radim Peša and Bedrich Benes and Voicu Popescu},
title={Hands-Free VR},
booktitle={Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: HUCAPP},
year={2025},
pages={533-542},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013115100003912},
isbn={978-989-758-728-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: HUCAPP
TI - Hands-Free VR
SN - 978-989-758-728-3
AU - Vazquez Fernandez J.
AU - Lee J.
AU - Serrano Vacca S.
AU - Magana A.
AU - Peša R.
AU - Benes B.
AU - Popescu V.
PY - 2025
SP - 533
EP - 542
DO - 10.5220/0013115100003912
PB - SciTePress