American Sign Language Recognition in Game Development for Deaf Children

HeleneBrashear1 , Valerie Henderson 1

1 Georgia Institute of Technology GVU Center College of Computing Atlanta, Georgia, USA

(brashear, sylee, vlh, thad)@cc.gatech.edu

Kwang-HyunPark2, HarleyHamilton3 ,

2 Korea Advanced Institute of Science and Technology Daejeon, Republic of Korea

akaii@robotian.net

SeungyonLee1, ThadStarner1

3 Center for Accessible Technology in Sign Atlanta Area School for the Deaf Clarkston, Georgia, USA

hhamilto@doe.k12.ga.us

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

The article presents a gesture recognition game to help deaf children learn America Sign Language (ASL). A data set characterized by disfluencies inherent to continuous signaling is used. Hand signals are recorded through visual and accelerometer data which is used to train a Hidden Markov Model (HMM).

Deaf children of parents who themselves have no hearing deficiencies need intensive instruction in signaling at an early age. This is thought particularly important for children not to suffer any developmental deficits. The only training these children may receive will be at school. To augment this Copy Cat is a game like program that encourages signaling of complete phrases. Real time video feedback and line gesture recognition is used to correctly train signs.

As no ASL recognition engine exists a wizard of OZ technique was employed in the creation of one. A human expert was used as the wizard who would sit out of sight to provide the corrections through the computer game environment. In this way the machine learning algorithm is provided with a set of examples and solutions. The role of the wizard eventually would be taken over by the computer. The input system had to be both rugged and cheap to sustain the riggers of the classroom environment. For this reason a combination of video and accelerometer data acquisition was chosen.

The ASL engine is able to handle both single and double handed signs, but can’t deal with facial expressions used for classifier manipulation. The data collection window is opened with a push to sign input from the user. This is taken from previous push to speak interfaces. The structure of the paradigm results in data which is much more natural as it is a product of interaction and not out of context in as in a laboratory environment.

Five children were used in the study, all of them played all three levels of the games at least five times. 90% of the data was used for training the remaining 10% for testing. For the user dependent study 93.4% accuracy was achieved, and 86.4 for the user independent study.

Discussion:

This is a very interesting system which sets out to fill an important gap in the training of deaf children who do not have sufficient access to a teacher during their developmental period. The system not only recognizes correct signs but encourages sentences and interaction in a natural manner. The hand segmentation is robust in varying lighting and does not involve unrealistic apparatus for individual purchase and use.