Facial Feature Tracking and Occlusion Recovery in American Sign Language

Thomas J. Castelli, Margrit Betke, Carol Neidle



Facial features play an important role in expressing grammatical information in signed languages, including American Sign Language (ASL). Gestures such as raising or furrowing the eyebrows are key indicators of constructions such as yes-no questions. Periodic head movements (nods and shakes) are also an essential part of the expression of syntactic information, such as negation (associated with a side-to-side headshake). Therefore, identification of these facial gestures is essential to sign language recognition. One problem with detection of such grammatical indicators is occlusion recovery. If the signer’s hand blocks his/her eyebrows during production of a sign, it becomes difficult to track the eyebrows. We have developed a system to detect such grammatical markers in ASL that recovers promptly from occlusion. Our system detects and tracks evolving templates of facial features, which are based on an anthropometric face model, and interprets the geometric relationships of these templates to identify grammatical markers. It was tested on a variety of ASL sentences signed by various Deaf 1 native signers and detected facial gestures used to express grammatical information, such as raised and furrowed eyebrows as well as headshakes.


  1. J. Ahlberg. “CANDIDE-3: An Updated Parameterised Face.” Linköping University Image Coding Group, Linköping, Sweden, Technical Report Number LiTH-ISY-R-2326. January 2001.
  2. Video Analysis of American Sign Language webpage. http://www.cs.bu.edu/fac/ betke/research/asl.html
  3. V. Athitsos and S. Sclaroff. “Estimating 3D Hand Pose from a Cluttered Image.” CVPR, Madison, WI, 2003. Pp. 432-439.
  4. C. Baker-Shenk and D. Cokely. American Sign Language: A Teacher's Resource Text on Grammar and Culture. Gallaudet University Press. 1991.
  5. B. Bauer and K-F. Kraiss. “Towards an Automatic Sign Language Recognition System Using Subunits.” in I. Wachsmuth and T. Sowa (eds.) Gesture and Sign Language in HumanComputer Interaction. Springer-Verlag. 2002. Pp. 64-75.
  6. M. Black and Y. Yacoob. “Tracking and Recognizing Rigid and Non-rigid Facial Motions Using Local Parametric Models of Image Motions.” ICCV, Cambridge, MA, June 1995. Pp. 374-381.
  7. F. Bourel, C. C. Chibelushi, and A. A. Low. “Robust Facial Feature Tracking.” British Machine Vision Conference, Bristol, Sept. 2000. Pp. 232-241.
  8. G. R. Bradski. “Computer Vision Face Tracking For Use in a Perceptual User Interface.” Intel Technology Journal, Q2, 1998. 15pp.
  9. T. F. Cootes and C. J. Taylor. “Statistical Models of Appearance for Medical Image Analysis and Computer Vision.” SPIE 2001. Pp. 236-249.
  10. G. R. Coulter. “Raised Eyebrows and Wrinkled Noses: The Grammatical Function of Facial Expression in Relative Clauses and Related Constructions.” 2nd Nat'l. Symp. Sign Language Research and Teaching, CA, 1978. Pp. 65-74.
  11. L. G. Farkas (Editor). Anthropometry of the Head and Face. Raven Press. 1994.
  12. Intel Corporation. Open Source Computer Vision Library: Reference Manual. July 2002. http://www.intel.com/research/mrl/research/opencv/
  13. T. Kanade, J. F. Cohn, and Y. Tian. “Comprehensive Database for Facial Expression Analysis.” Face & Gesture Recognition Conf., France, 2000. Pp. 46-53.
  14. J. Lombardi and M. Betke, “A Self-Initializing Eyebrow Tracker for Binary Switch Emulation.” Boston University, CS-TR-2002-023. 2002.
  15. S. J. McKenna, S. Gong, R. P. Würtz, J. Tanner, and D. Banin. “Tracking Facial Feature Points with Gabor Wavelets and Shape Models”. Inter'l Conf. Audio- and Video-Based Biometric Person Identification, 1997. Pp. 35-42.
  16. National Center for Sign Language and Gesture Resources webpage. http://www.bu.edu/ asllrp/cslgr/
  17. C. Neidle, S. Sclaroff, and V. Athitsos. “SignStreamTM: A Tool for Linguistic and Computer Vision Research on Visual-Gestural Language Data.” Behavior Research Methods, Instruments, and Computers: 33(3), 2001. Pp. 311-320.
  18. C. Neidle, J. Kegl, D. MacLaughlin, B. Bahan, and R. G. Lee. The Syntax of American Sign Language: Functional Categories and Hierarchical Structure. Cambridge, MA: MIT Press. 2000.
  19. T. Ohtsuki and G. Healey. “Using Color and Geometric Models for Extracting Facial Features.” J. Imaging Science and Technology: (42) 6, 1998. Pp. 554-561.
  20. T. Starner and A. Pentland. “Real-Time American Sign Language Recognition from Video Using Hidden Markov Models.” Proc. International Symposium on Computer Vision, Coral Gables, FL, 21-23 November 1995. Pp. 265-270.
  21. J. Tang and R. Nakatsu. “A Head Gesture Recognition Algorithm.” Int'l Conf. Advances in Multimodal Interfaces, Beijing, 2000. Pp. 72-80.
  22. V. Vezhnevets. “Face and Facial Feature Tracking for Natural Human-Computer Interface.” Graphicon 2002. Nizhny Novgorod, Russia, 16-21 September 2002.
  23. C. Vogler and D. Metaxas. “ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis.” ICCV, Mumbai, 1998, Pp. 363-369.
  24. C. Wang, W. Gao, and S. Shan. “An Approach Based on Phonemes to Large Vocabulary Chinese Sign Language Recognition.” FGR Conf., 2002. Pp. 411-416.
  25. J. Woodward. “Deaf Awareness.” Sign Language Studies:3, 1973. Pp. 57-60. CVPR, San Diego, 1989. Pp. 104-109.

Paper Citation

in Harvard Style

J. Castelli T., Betke M. and Neidle C. (2006). Facial Feature Tracking and Occlusion Recovery in American Sign Language . In 6th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2006) ISBN 978-972-8865-55-9, pages 81-90. DOI: 10.5220/0002471700810090

in Bibtex Style

author={Thomas J. Castelli and Margrit Betke and Carol Neidle},
title={Facial Feature Tracking and Occlusion Recovery in American Sign Language},
booktitle={6th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2006)},

in EndNote Style

JO - 6th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2006)
TI - Facial Feature Tracking and Occlusion Recovery in American Sign Language
SN - 978-972-8865-55-9
AU - J. Castelli T.
AU - Betke M.
AU - Neidle C.
PY - 2006
SP - 81
EP - 90
DO - 10.5220/0002471700810090