American Sign Language Recognition in Game Development for Deaf Children

HeleneBrashear1 , Valerie Henderson 1

1 Georgia Institute of Technology GVU Center College of Computing Atlanta, Georgia, USA

(brashear, sylee, vlh, thad)@cc.gatech.edu

Kwang-HyunPark2, HarleyHamilton3 ,

2 Korea Advanced Institute of Science and Technology Daejeon, Republic of Korea

akaii@robotian.net

SeungyonLee1, ThadStarner1

3 Center for Accessible Technology in Sign Atlanta Area School for the Deaf Clarkston, Georgia, USA

hhamilto@doe.k12.ga.us

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

The article presents a gesture recognition game to help deaf children learn America Sign Language (ASL). A data set characterized by disfluencies inherent to continuous signaling is used. Hand signals are recorded through visual and accelerometer data which is used to train a Hidden Markov Model (HMM).

Deaf children of parents who themselves have no hearing deficiencies need intensive instruction in signaling at an early age. This is thought particularly important for children not to suffer any developmental deficits. The only training these children may receive will be at school. To augment this Copy Cat is a game like program that encourages signaling of complete phrases. Real time video feedback and line gesture recognition is used to correctly train signs.

As no ASL recognition engine exists a wizard of OZ technique was employed in the creation of one. A human expert was used as the wizard who would sit out of sight to provide the corrections through the computer game environment. In this way the machine learning algorithm is provided with a set of examples and solutions. The role of the wizard eventually would be taken over by the computer. The input system had to be both rugged and cheap to sustain the riggers of the classroom environment. For this reason a combination of video and accelerometer data acquisition was chosen.

The ASL engine is able to handle both single and double handed signs, but can’t deal with facial expressions used for classifier manipulation. The data collection window is opened with a push to sign input from the user. This is taken from previous push to speak interfaces. The structure of the paradigm results in data which is much more natural as it is a product of interaction and not out of context in as in a laboratory environment.

Five children were used in the study, all of them played all three levels of the games at least five times. 90% of the data was used for training the remaining 10% for testing. For the user dependent study 93.4% accuracy was achieved, and 86.4 for the user independent study.

Discussion:

This is a very interesting system which sets out to fill an important gap in the training of deaf children who do not have sufficient access to a teacher during their developmental period. The system not only recognizes correct signs but encourages sentences and interaction in a natural manner. The hand segmentation is robust in varying lighting and does not involve unrealistic apparatus for individual purchase and use.

An Empirical Evaluation of Touch and Tangible Interfaces for Tabletop Displays

Aure ́lien Lucchi, Patrick Jermann, Guillaume Zufferey, Pierre Dillenbourg

CRAFT - EPFL aurelien.lucchi, patrick.jermann, guillaume.zufferey, pierre.dillenbourg @epfl.ch

Comments:

Manoj’s Blog

Franck's Blog

Summary:

The article sets out to make a comparison between a touch and a tangible interface. In general touch interfaces are faster than touch interfaces. As touch interfaces take advantage of a broader range of human abilities. Most studies are based on Fitt’s law where time to acquire target is a function of the distance divided by the target size. In this article the digital desk is used in combination with the Bricks system.

Recently there has been an increase in the popularity of touch systems. Tangible systems are seen as more task specific. For the purpose of comparison, both the touch and the tangible systems were made as similar as possible creating equivalent virtual objects for the scale models. A variety of gestures were compared: Addition, Selection, Translation, Rotation, Scaling, Removal, Undo previous action, Submit, Select All, and Adjustment.

Forty experimental subjects were used in the experiment. They were give a total of forty warehouse layouts to implement on both interfaces. their time and accuracy were measured and compared. Overall, the tangible interface was significantly quicker than the touch interface. However, some actions were quicker in the touch interface than in the tangible one such as scaling walls. The poly sensory feedback provided by tangible objects, combined with our natural familiarity handling them was attributed to the overall increase in performance of the tangible interface.

Discussion:

An interesting comparison. However, I feel that it is a little unfair as we have all had a life time experience with tangible interfaces, by comparison our acquaintance with touch interfaces is very short. Given more time I feel that the performance on the touch interface would improve and potentially surpass its tangible counterpart. Additional feedback, other than visual, would have certainly improved the performance of the touch interface.

Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound

Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda

Department of Information Physics and Computing Graduate School of Information Science and Technology The University of Tokyo Eng.Bldg.6, 7-3-1, Hongo, Bunkyo Ward, Tokyo, Japan {iwa,tatezono,shino}@alab.t.u-tokyo.ac.jp http://www.alab.t.u-tokyo.ac.jp/∼ iwa

Comments:

Drew’s Blog

Franck’s Blog

Summary:

The article presents a tactile device that produces 3D stress fields combined with a stereoscopic display to provide 3D tactile feedback for interacting with 3D visual objects. With developments in graphics, physical stimulation and display technology. There is a demand for haptic devices that integrate sight and touch.

Current devices enable the visualization and handling of 3D objects. In addition to this, the ability to physically sense them would enrich the experience. Constant contact systems are not very effective as there is never an absence of touch. Robot systems of feedback are bulky, non portable, and difficult to set up. This paper will present an airborne ultrasonic alternative which has a high temporal and spatial resolution.

The apparatus consists of an array of 96 ultrasonic transducers which are grouped into 12 sets. These are arranged in rings where each members is equidistant from the center of the array. They are able to be used directly on the skin where the cumulative force can be focussed to a point.

The most effective stimulation was obtained through bursts of ultrasonic sound. Continuous activation resulted in only the onset of sound being perceived. 20mm spatial resolution was achieved but the ultrasonic array lacked sufficient strength for a continuous pressure to be perceived.

Discussion:

An interesting approach, where tactile feedback can be applied without continuous contact. This setup does not allow for 2D phase control of the array which would permit the point of focus to be manipulated.

There are many possible applications where feedback would be beneficial as it is non intrusive and does not require anything to be worn. As the array is fixed in space the feedback is anchored to a particular position, similar to real objects.

FreeDrawer – A Free-Form Sketching System on the Responsive Workbench

Gerold Wesche, Hans-Peter Seidel

Comments:

Frank's Blog

Paul's Blog

Summary:

The article presents a virtual reality spline based free-form surface sketching system where the user uses a tracked stylus as an input device. The user draws directly in 3D with tools adapted to the needs of designers. The power and utility of 3D environments for design are well recognized. However, most CAD systems are geared toward elaborate shape design and lack support for concept development. FreeDrawer is a sketching tool dedicated to the initial stages of design, enabling direct transfer into a corresponding computer representation.

Unlike direct surface modeling which is based on free-form solids, free form curves or voxel based virtual clay, FreeDrawer constructs models from a skeleton. This approach avoids both computational load and unwanted artifacts when off axis changes are made. Space curves and 2D curves projected on a plane are supported. Standard editing features are accessed from a hand held wand.

A curve network is used to snap new curves to a position in space relative to other curves already drawn, so in effect they are an approximation of the original. When changes are make to a curve, locally connected curves will adapt up to the next junction. Globally connected curves would undergo a collective adaptation.

Filling in surfaces is achieved within closed loops. There is a closed loop algorithm that searches according to geometric criteria. Kuriyama surfaces and n-sided domains can be selected. However, for surface deformation Catmull-Clark surfaces are closely defined to match the Krriyama surfaces. Curve smoothing, sharpening and dragging are achieved with hand gestures where the region of influence is definable.

A novel tool pallet is presented in the form of a set of radially organized pointers rather like fingers, from which the operator can chose. The point of the desired tool is contacted to the point of application and the selection button pressed. An example of a car seat drawn quickly is given to illustrate the performance of the system.

Conclusions:

The utility and at the same time difficulty in implementation of a force feedback system in this type of environment are mentioned. The tool palate is also identified lacking, and in the process of being redesigned based on experience from the current word. An important case is made that unreasonable demands are placed on the ease and usability of 3D drawing environments for the uninitiated.

Discussion:

FreeDrawer seems to target a highly specific window in the design phase which has not yet be liberated from pen and paper. The features and available tools are all tailored to this end and make a highly compelling case.

It is not surprising that the tool palate did not survive in its original form. Trying to touch a particular object would be awkward particularly for some of the tools, not to mention how one would deal with objects close together.

Furthermore, some question is raised as FreeDrawer’s suitability for all designers. Without a test battery this is hard to judge.

COMPUTER VISION-BASED GESTURE RECOGNITION FOR AN AUGMENTED REALITY INTERFACE

Moritz Störring, Thomas B. Moeslund, Yong Liu, and Erik Granum

Computer Vision and Media Technology Laboratory,

Aalborg University Niels Jernes Vej 14,

DK-9220 Aalborg East, Denmark

{mst,tbm,yliu,eg}@cvmt.aau.dk

Comments:

Drew’s Blog

Manoj’s Blog

Summary:

This paper presents a computer vision based gesture recognition system for an augmented reality interface. It is claimed that wearable computing will soon be able to use head mounted displays (HMDs) in all sorts of applications. This application uses a combination of HMDs and head mounted cameras (HMCs) to overlay information on the view of the real world. Hand gestures were chosen as a mode of communication with the augmented reality interface.

Two traditional types of optical state based gesture recognition are described. The first is model based recognition where the image of the hand is frame by frame fitted to a pre constructed model. The second is an appearance based approach which requires a set of training examples and a classifier. These two methods are dismissed as not being suitable for AR applications due to their high computational requirements.

The goal of the system presented in this article is to provide the foundation for a multiuser round table presentation system. Pointing and clicking are identified as two gestures which are fundamental to any interface. The system will be able to identify a total of six discrete gestures. The plane of sight is limited to 2D in the interests of economy.

Segmentation is achieved by the use of a color pixel approach. HSV and normalized RGB are used to overcome color variation caused by variations in illumination intensity. This method enables colors to be separated from their intensity. Region growing is used to find completed areas.

The image is geometrically segmentation with a pseudo polar transformation. A discrete number of concentric lines are explored to speed up processing. The area of interest is demarcated between the smallest to the greatest radius of skin. A temporal filter is used to differentiate gestures. For pointing, once the gesture has bee identified, the vector is triangulated from the two HMCs. The clicking gesture is identified by the use of a bounding box to identify when the thumb is opposed. The algorithm is described as robust and efficient for gesture recognition.

Discussion:

This paper proposes a comprehensive wearable augmented reality system.

The approach chosen for segmentation with the use of a transformation was of particular interest. The ability to count fingers by transforming the coordinate system into pseudo polar and then counting the number of spikes is robust and independent of finger length.

It is not clear how this would be adapted to three dimensions.

Technology and Dancing

I have tried to imagine what could improve upon looking in the mirror while dancing with someone who is trying to teach one. If one for the sake of simplicity looks at the technical side of mastering given moves or routines. These depend an performing a series of movements in a particular order where the components occur at a particular time. This is a tremendous simplification and can be viewed as a first step. For example the the same approach could be used for learning to play the piano. By itself, it would not lead to music, yet it is a necessary prerequisite.

So as a tool to teach dance moves or routines, video motion tracking, combined with vibrotactile stimulation could be very beneficial for refining movements and postures. The routines could be practiced without music until a certain familiarity and comfort had been attained. One would first watch a dance instructor perform an example which has been broken down into its component parts. One would then attempt to imitate with vibrotactile feedback based on visual motion tracking information to refine ones actions.

The final step of applying the routines to music may not be something that can be taught.

VARK

After taking the test it was no surprise that my score was quite representative of how I think I learn. However, I don’t think it is objective as the questions are rather transparent so it is quite easy to gravitate towards what one thinks one would like to do in a particular situation. No doubt to avoid this the battery of questions would have to be much larger.

At any given time the learning approach we use may depend on the particular problem which we are confronted with. Under different circumstances, such as familiarity or the lack there of, a totally different approach may be deployed.

Motion Editing with Data Glove

Wai-Chun Lam, Feng Zou, Taku Komura

City University of Hong Kong 83 Tat Chee Ave Kowloon, Hong Kong

Comments:

Drew’s Blog

Manoj’s Blog

Summary:

This paper describes a method to edit captured human motion data with a data glove.

Motion editing is frequently used to generate mew motion data from existing captured data. This procedure can be applied to characters with different body sizes using retargeting. Overlapping and blending motions is achieved with motion wrapping. These methods are traditionally based on keyframe postures, where the editor determines when the posture must be edited. The changes are then determined with inverse kinematics. This approach has the following drawbacks: In order to determine the effect of a change the motion has to be replayed. Drastic changes need the insertion of several keyframes. Keyframe editing is a static editing method whereas motion is obviously dynamic.

A new dynamic editing method using a data glove has been therefore developed. First the real human motion data that is to be used for the basic motion is prepared. The animator then mimics the motion that is viewed to create a mapping function between the hand gestures and the real human motion data. New motions not present in the original data can then be created from hand gestures alone. This method maps finger movements to the actions of the whole body and is also able to function in real time for virtual reality, gaming etc.

A P5 data glove is used where the index and middle fingers correspond to the legs and shoulders of the collected human movement data. Matching is then made by synchronizing the tips and pits of the motion curves. This stage can be repeated if required to improve the mapping function. As the degrees of movement of the human body are much greater than those of the fingers, individual finger movements are mapped to generalized coordinates for the body. The arm motions are conjugate with opposite leg actions. During the capture and mapping stages, Fourier series expansion is used to eliminate noise and high frequency data. Additionally, the domain of the mapping function can be scaled to cope with new hand gestures outside the original domain.

Evaluation & Results:

An ordinary walking motion was captured for the model data. The animator was asked to mimic this action in order for the mapping function to be generated. The animator was then asked to execute a hopping zigzag running stride. The most prominent features noted in the resulting animation were the absence of lateral body tilt and foot contact with the ground. This indicated the need to extract specific constraints from the original data and preserve them in the newly generated motion. In its current form, for the newly generated motion to be realistic it has to have a similar topographic structure to the original data. As a next step, a more detailed joint matching approach between hand and body will be explored.

Discussion:

The use of a dynamic control system in place of a static one when dealing with a dynamic system is logically very sound. However, a walking finger puppet will only be able to directly influence the legs of the human model. All other dynamics will have to be slaved to the legs or independently controlled.

In order to understand the results more thoroughly a more rigorous comparison needs to be presented. A comparison to a traditional static keyframe approach would be useful to identify advantages and shortcomings.

However, a very interesting approach. To be able to control and modify a complex motion with a simple and intuitive action surely has many applications.

EyePoint: Practical Pointing and Selection

Using Gaze and Keyboard

Manu Kumar, Andreas Paepcke, Terry Winograd

Stanford University, HCI Group 353 Serra Mall, Stanford, CA 94305-9035 {sneaker, paepcke, winograd}@cs.stanford.edu

Comments:

Drew’s Blog

Manoj’s Blog

Summary:

The article presents EyePoint which is a combination of gaze based pointing and keyboard triggered selection. The goal is to develop a simple system which is accurate and fast enough to be a viable alternative to traditional devices such as the mouse. The motivation for the project was to use gaze information to augment traditional input devices, and not serve as a substitute. The visual modality was seen as an input channel, where its use for motor control would be directly at odds with the users natural inclination. This is best illustrated by traditional dwell based systems, which by their very nature have lacked performance rendering them as non viable alternatives.

In the development of EyePoint, research began exploring how users use the mouse for pointing. It was universally seen that the mouse was the preferred method for all object manipulation tasks. Furthermore, for any gaze based system to be viable it would have to be able to execute all the clicking actions such as single, double, right, over, etc. In contrast to deliberate motor actions such as mouse and keyboard input which need no disambiguation. Eye movements need filtering to remove micro and involuntary scads to avoid the midas touch effect. Current gaze based systems suffer from a variety of intrusive functional byproducts undermining their viability.

EyePoint, tries to overcome this by employing a two step progressive refinement procedure in order to compensate for the inaccuracy of eye gaze trackers. The user looks at the desired target and holds down the appropriate hot key for the desired click action. The area being looked at is zoomed the user looks again and releases the hot key to execute the action. Drag is a two step process, where the destination is selected with another hot key. To abort an action the user simply looks outside the zoom area. An interesting refinement is the projection of a focussing grid on the zoomed window to stabilize gaze.

The system is based on four principles:

1. Not slaving any action directly to eye movements.

2. Using zooming to overcome accuracy problems.

3. Fixation and smoothing algorithm.

4. Efficient activation mechanism.

Refinements:

1. Ensuring that the zoom window is bound by the limits of the screen.

2. An animated zoom the reduce secondary saccades.

3. A focussing grid overlay to reduce jitter.

4. A current gaze feedback marker was tested and seen to be distracting.

Evaluation:

20 participants. Three variables: Focus points. Gaze marker. without focus points

1. 10 - 15 min training phase

2. Web Study navigate through 30 pages.

Subjects thought speeds were comparable.

3. Pointing only task to click on the red balloon.

Subjects felt the mouse was faster and more accurate.

4. Mixed typing and pointing with a mouse only measuring the time to point not type.

Subjects liked the reduction in hand movement, but preferred the accuracy of the mouse

Results:

Web Study:

EyePoint with focus marker was 20% slower than the mouse.

EyePint with focus markers was 10%, without was 13% greater error than the mouse.

Balloon Study:

Not a great deal of difference in speed here.

10x the error of the mouse.

Mixed Study:

Again not much difference in times.

20x the error of the mouse.

Discussion:

EyePoint is a well thought out use of gaze controlled pointing. An improvement on previous overloaded approaches. This system maximizes the modalities natural attributes without overburdening them.

However, the traditional mouse uses a single button for several actions single double, drag. EyePoint replaces this with multiple buttons which surely is less economic. It may have been better to use a single hot key similar in function to the traditional model. This brings up the question of ease of use for disabled users.

In the conclusion the data gathered for test three is reinterpreted more favorably by making assumptions. Surely it would be simpler to repeat test three with Eye point to get a direct comparison.

Monocular Eye Tracker

First Impressions:

Reasonable comfortable to wear could be used for an extended period of time.

Does not impede vision

easy to set up

consistent pupil lock

Ideas to exploit the device:

I have always been interested to see what is being looked at while a particular task is being completed.

A good measure of sensory attention

could be a interesting to set up an accuracy evaluation task for the eye tracker itself

Limitations of the device:

auto calibration is a little too fast paced, which may be why accuracy is questionable particularly in the vertical domain.

The CyberGlove

First impressions:

Comfortable to wear, does not impede movement, could be used for an extended period of time.

Setup, calibration, getting data from separate joints and setting vibrating transducers seemed straight forward. I need to familiarize myself with C# syntax, but completing the assignment was not hard, particularly with sample code to follow.

Ideas to exploit the device:

Exploring what tactile features are used in object identification tasks.

Vibro-tactile feedback.

Limitations of the device:

Quick calibration was not very accurate.

Would be curious to determine its consistency.

It seems very effective for its purpose.

The Monocular Augmented Reality Goggles

First impressions:

Much more user friendly than the binocular pair. Very little adjustment required to see clearly. In particular, none of the double vision problem of the first device. Images were evenly visible throughout the display field. In spite of being monocular the sense of depth was greater than with the previous binocular system.

Ideas to exploit the device:

As before all sorts of application can be thought of, with either additions to or subtractions from the environment.

Limitations of the device:

similar to the binocular pair however to a less extent. The field of view may be a little narrower. However, much more usable.

The Binocular Augmented Reality Goggles

First impressions:

Adjustments are very important to get any result. The vertical and proximal positioning of the unit on one’s head combined with correct interocular separation are critical. The head strap also has to be secure for any slight movements of the unit on one’s head and all bets are off.

There was an initial double vision when looking at objects at different depths of field. This is due to a fixed convergence on the cameras and a fixed focal plane with the eye piece displays. A short time is required to accommodate in order to maintain integration of left and right images.

Furthermore, one has to adjust to exploring the environment with the use of head movements as eye movements are not possible.

Ideas to exploit the device:

All sorts of application can be thought of, with either additions to or subtractions from the environment.

Limitations of the device:

The field of view is narrower and the focal length is longer than normal which gives the impression that objects are further away than they really are. It is a little like looking through a telescope.

A light shield around the eye pieces to block off light leaking in would improve clarity. This would also eliminate competition from one’s peripheral vision (seeing around the outside of the device).

Contrast and color representation need experimentation. It was hard to see the yellow book on the table etc.

In spite of these comparatively trivial points a most interesting experience.