Andrea Bottino, Matteo De Simone



The development of new interaction paradigms requires a natural interaction. This means that people should be able to interact with technology with the same models used to interact with everyday real life, that is through gestures, expressions, voice. Following this idea, in this paper we propose a non intrusive vision based tracking system able to capture hand motion and simple hand gestures. The proposed device allows to use the hand as a “natural” 3D mouse, where the forefinger tip or the palm centre are used to identify a 3D marker and the hand gesture can be used to simulate the mouse buttons. The approach is based on a monoscopic tracking algorithm which is computationally fast and robust against noise and cluttered backgrounds. Two image streams are processed in parallel exploiting multi-core architectures, and their results are combined to obtain a constrained stereoscopic problem. The system has been implemented and thoroughly tested in an experimental environment where the 3D hand mouse has been used to interact with objects in a virtual reality application. We also provide results about the performances of the tracker, which demonstrate precision and robustness of the proposed system.


  1. Akyol S., Alvarado P., 2001. 'Finding Relevant Image Content for mobile Sign Language Recognition', Proc. Signal Processing, Pattern Recognition and Application, pp. 48-52
  2. Bottino A., Laurentini A., 2007. 'How to Make a Simple and Robust 3D Hand Tracking Device Using a Single Camera', CSCC 2007, Agios Nikolaos, Greece.
  3. Bray M., Koller-Meier E., Van Gool, L., 2004, 'Smart particle filtering for 3D hand tracking', Proc. IEEE Intern. Confer. on Automatic Face and Gesture Recognition, pp. 675 - 680
  4. Bradski G. R., 1998. 'Computer Vision Face Tracking For Use in a Perceptual User Interface', Intel Technology Journal (2), pp. 215.
  5. Chen F.-S., Fu C.-M., Huang C.-L., 2003. 'Hand Gesture Recognition Using a Real-Time Tracking Method and Hidden Markov Models', Image and Vision Computing vol. 21, August, pp. 745-758
  6. Cheng Y., 1995. 'Mean shift, mode seeking, and clustering', IEEE Trans. PAMI., vol. 17, pp. 790-799
  7. Cui Y., Weng J., 2000. 'Appearance-Based Hand Sign Recognition from Intensity Image Sequences'. Computer Vision Image Understanding, vol. 78, February, pp. 157-176
  8. Drummond T., Cipolla R., 2002. 'Real-time visual tracking of complex structures'. IEEE Trans. PAMI, vol. 24, July, pp. 932-946.
  9. Erol A., Bebis G., Nicolescu M., Boyle R. D., Twombly X., 2007. 'Vision-based hand pose estimation: A review'. Computer Vision and Image Understanding vol. 108, pp. 52-73
  10. Gumpp T., Azad P., Welke K., Oztop E., Dillmann R., Cheng G., 2006. 'Unconstrained Real-time Markerless Hand Tracking for Humanoid Interaction'. Proc. of 6th IEEE-RAS Intern. Conf. on Humanoid Robots, pp. 88-93
  11. Haiting Zhai, Xiaojuan Wu, Hui Han, 2005. 'Research of a Real-time Hand Tracking Algorithm'. Proc. of ICNN&B 2005
  12. Heap T., Hogg D., 1996. 'Towards 3D hand tracking using a deformable model', Proc. of International Conference on Automatic Face and Gesture Recognition, pp. 140-145
  13. Isard M., Blake A., 1998. 'CONDENSATION - conditional density propagation for visual tracking', Int. J. Computer Vision, vol. 29, pp. 5-28
  14. Isard M., Blake A., 1998. 'ICondensation: Unifying LowLevel and High-Level Tracking in a Stochastic Framework', Proc. ECCV98, pp. 5-28
  15. Letessier J., Bèrard F., 2004. 'Visual tracking of bare fingers for interactive surfaces', Procs. of UIST 7804: 17th Annual ACM symposium on User Interface Software and Tec
  16. Liu N., Lovell B., Kootsookos P., 2003. 'Evaluation of hmm training algorithms for letter hand gesture recognition', Proc. ISSPIT 2003, Darmstadt, Germany
  17. Liu Y., Jia Y., 2004. 'A Robust Hand Tracking and Gesture Recognition Method for Wearable Visual Interfaces and Its Applications', Proc. of 3rd Int. Conf. on Image and Graphgics IEEE
  18. Mahmoudi F., Parviz M., 2006. 'Visual Hand Tracking Algorithms', Geometric Modeling and Imaging--New Trends, vol. 5/6, pp. 228 - 232
  19. Oka K., Sato Y., Koike H., 2002. 'Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems', Proc of FGR 7802, p. 429.
  20. Olafsdottir, H. et al. Is the thumb a fifth finger? A study of digit interaction during force production tasks. Exp Brain Res (2005) 160: 203-213
  21. Pantrigo J.J., Montemayor A.S., Sanchez A., 2005. 'Local search particle filter applied to human-computer interaction', Proc. of ISPA'05, pp. 279 - 284
  22. Shan Lu, Metaxas D., Samaras D., Oliensis J., 2003. 'Using multiple cues for hand tracking and model refinement', Proc. IEEE Conf. on Computer Vision and Pattern Recognition 2003, vol. 2, pp. 443-450.
  23. Shan C., Wei Y., Tan T., Ojardias F., 2004. 'Real time hand tracking by combining particle filtering and mean shift', Proc. FG2004, pp. 669-674
  24. Stenger B., Mendonca P.R.S., Cipolla R., 2001. 'Modelbased 3D tracking of an articulated hand', Proc. IEEE CVPR 2001 vol. 2, pp. II-310 - II-315
  25. Stenger B., Thayananthan A., Torr P. H. S., Cipolla R., 2004. 'Hand pose estimation using hierarchical detection', Proc. of Intl. Workshop on HumanComputer Interaction 2004.
  26. Tomasi C., Petrov S., Sastry A., 2003. 783D Tracking = Classification + Interpolation', Ninth IEEE International Conference on Computer Vision, vol, 2, pp. 1441-1448
  27. Yang Liu, Yunde Jia, 2004. 'A robust hand tracking for gesture-based interaction of wearable computers', Proc. ISWC 2004, pp. 22 - 29
  28. Yang Liu, Yunde Jia, 2004. 'A robust hand tracking and gesture recognition method for wearable visual interfaces and its applications', Proc. International Conference on Image and Graphics ICIG 2004, pp. 472 - 475

Paper Citation

in Harvard Style

Bottino A. and De Simone M. (2009). A FAST AND ROBUST HAND-DRIVEN 3D MOUSE . In Proceedings of the Fourth International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2009) ISBN 978-989-8111-69-2, pages 567-574. DOI: 10.5220/0001746005670574

in Bibtex Style

author={Andrea Bottino and Matteo De Simone},
booktitle={Proceedings of the Fourth International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2009)},

in EndNote Style

JO - Proceedings of the Fourth International Conference on Computer Vision Theory and Applications - Volume 1: VISAPP, (VISIGRAPP 2009)
SN - 978-989-8111-69-2
AU - Bottino A.
AU - De Simone M.
PY - 2009
SP - 567
EP - 574
DO - 10.5220/0001746005670574