The $3 Recognizer: Simple 3D Gesture Recognition on Mobile Devices

Sven Kratz

Deutsche Telekom Laboratories, TU Berlin Ernst-Reuter-Platz 7 10587 Berlin, Germany sven.kratz@telekom.de

Michael Rohs

Deutsche Telekom Laboratories, TU Berlin Ernst-Reuter-Platz 7 10587 Berlin, Germany michael.rohs@telekom.de

Comments:

Akshay’s Blog

Franck’s Blog

Summary:

The article presents the $3 gesture recognizer algorithm which is designed to be implemented quickly, is device independent, does not require any special toolkits or frameworks, in stead relies solely on trigonometric and geometric calculations. With the inclusion of 3D accelerometers in many mobile devices the range of motion is no longer restricted by the size of the device. This work is an extension of Wobbrock et al.’s $1 2D algorithm, resulting in a 3D recognizer..

The gesture data is obtained by collecting a series of acceleration values which are the differences between consecutive measurements. This set is then compared to the gesture library and a set of matching scores determined by the heuristic which are then used to determine if a gesture has been correctly identified.

As with 1$ resampling is performed to equalize the number of points between the candidate gesture and the members of the library. As with $1 a rotation is applied along an axis between the first point and the centroid of the gesture. The size is then normalized by scaling the gesture to fit in a set cube space. Mean Squared Error is used to match candidate gestures with those in the library. Rotational differences are compensated for with Golden Section Search.

There is a heuristic in $3 which leads to a reduction in false positives over $1. The scoring heuristic is able to deal with the reduction in precision of the accelerometer data as compared with a tablet entry.

Results:

Considerable recognition in false positives over $1. However, an actual figure is not stated.

Discussion:

The extension to $1 from 2D to 3D is indeed of interest. The main extension seems to be attributes to a scoring heuristic which is used to determine whether a library match has been made. The difference in application to less precise accelerometer data is also interesting.

However, the absence of any quantitative data to support the claimed improvements over $1 is concerning.

Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes

Jacob O. Wobbrock

The Information School University of Washington Mary Gates Hall, Box 352840 Seattle, WA 98195-2840 wobbrock@u.washington.edu

Andrew D. Wilson

Microsoft Research

One Microsoft Way Redmond, WA 98052 awilson@microsoft.com

Yang Li

Computer Science & Engineering University of Washington The Allen Center, Box 352350 Seattle, WA 98195-2350 yangli@cs.washington.edu

Comments:

Frank’s Blog

Manoj’s Blog

Summary:

Introduction:

The article presents $1 a cheap, compact gesture recognizer which involves only basic geometry and trigonometry in its implementation. With increasing opportunities for using pens, fingers, and wands, there is a need for a gesture recognition algorithm that does not require expertise in AI or pattern matching. In this article $1 will be compared to two other well known gesture recognition algorithms.

Criteria for the $1 recognizer:

1. Resilient to variations in sampling due to movement speed or sensing;

2. support optional and configurable rotation, scale, and position invariance;

3. require no specialized mathematical techniques;

4. short implementation;

5. run fast enough for interactive use;

6. have the ability to learn new gestures;

7. return an N-best list with a [0,1] score, independent of the number of input points;

8. provide competitive performance with previous algorithms in HCI over a set of gestures.

$1 has four steps in its execution:

1. The path is resampled by determining an average increment between points, and adding a new point when the actual increment exceeds this value. The candidate gesture and the template will thus have the same number of points. This enable point by point comparison between the template and candidate paths.

2. $1 searches for the angle that results in the best alignment between the two point paths. A rotation trick is used to increase performance. This involves a rotate to zero of the angle from the first point to the centroid of the gesture.

3. The gesture is then scaled to a reference square. This ensures that pairwise point differences between the candidate and template paths are due to the rotation and not aspect ratio. A translate to origin is then performed.

4. A global minimum search is performed for the best angle and template example according to the cumulative difference between the point positions of the candidate and the template path. This may involve further rotation of the candidate path.

Rotation Invariance:

Iterations in matching candidate and template paths are minimized by the use of a seed and search approach. Hill climbing was found to always find a mach where similar gesture pairs were concerned. With dissimilar pairs this was not the case as the results were plagued with local minima and a sharp increase in the number of iterations. A middle ground solution was found by using the Golden Section Search, which guarantees to terminate after 10 iterations.

Limitations of the $1 Recognizer:

$1 is a geometric template matching algorithm. It can’t tell the difference between gestures which depend on orientation, or time.

Evaluation:

Ten subjects were chosen.

An HP iPAQ h4355 Pocket PC was the tablet on which they were tested.

16 Gestures were used.

After one practice gesture the subjects were asked to perform three sets of four gestures at slow medium and fast speeds. All gestures were to be performed as accurately as possible.

The performance of 1$ was compared to Rubine and Dynamic Time Wrapping.

Results:

Recognition errors for 1$ vs Rubine were 0.98% (SD=3.63) and 0.85% (SD=3.27).

Recognition rates for 1$ vs Rubine were 92.02% vs 99.15%

Discussion:

The motivation behind 1$ is particularly good. There are many developers who would be much more willing to combine gesture recognition into applications, knowing that they could make a start with a relatively simple algorithm. The comparison with other algorithms, a discussion of the algorithms limitations, and the pseudocode of the algorithm at the end of the article is particularly helpful.

American Sign Language Recognition in Game Development for Deaf Children

HeleneBrashear1 , Valerie Henderson 1

1 Georgia Institute of Technology GVU Center College of Computing Atlanta, Georgia, USA

(brashear, sylee, vlh, thad)@cc.gatech.edu

Kwang-HyunPark2, HarleyHamilton3 ,

2 Korea Advanced Institute of Science and Technology Daejeon, Republic of Korea

akaii@robotian.net

SeungyonLee1, ThadStarner1

3 Center for Accessible Technology in Sign Atlanta Area School for the Deaf Clarkston, Georgia, USA

hhamilto@doe.k12.ga.us

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

The article presents a gesture recognition game to help deaf children learn America Sign Language (ASL). A data set characterized by disfluencies inherent to continuous signaling is used. Hand signals are recorded through visual and accelerometer data which is used to train a Hidden Markov Model (HMM).

Deaf children of parents who themselves have no hearing deficiencies need intensive instruction in signaling at an early age. This is thought particularly important for children not to suffer any developmental deficits. The only training these children may receive will be at school. To augment this Copy Cat is a game like program that encourages signaling of complete phrases. Real time video feedback and line gesture recognition is used to correctly train signs.

As no ASL recognition engine exists a wizard of OZ technique was employed in the creation of one. A human expert was used as the wizard who would sit out of sight to provide the corrections through the computer game environment. In this way the machine learning algorithm is provided with a set of examples and solutions. The role of the wizard eventually would be taken over by the computer. The input system had to be both rugged and cheap to sustain the riggers of the classroom environment. For this reason a combination of video and accelerometer data acquisition was chosen.

The ASL engine is able to handle both single and double handed signs, but can’t deal with facial expressions used for classifier manipulation. The data collection window is opened with a push to sign input from the user. This is taken from previous push to speak interfaces. The structure of the paradigm results in data which is much more natural as it is a product of interaction and not out of context in as in a laboratory environment.

Five children were used in the study, all of them played all three levels of the games at least five times. 90% of the data was used for training the remaining 10% for testing. For the user dependent study 93.4% accuracy was achieved, and 86.4 for the user independent study.

Discussion:

This is a very interesting system which sets out to fill an important gap in the training of deaf children who do not have sufficient access to a teacher during their developmental period. The system not only recognizes correct signs but encourages sentences and interaction in a natural manner. The hand segmentation is robust in varying lighting and does not involve unrealistic apparatus for individual purchase and use.

An Empirical Evaluation of Touch and Tangible Interfaces for Tabletop Displays

Aure ́lien Lucchi, Patrick Jermann, Guillaume Zufferey, Pierre Dillenbourg

CRAFT - EPFL aurelien.lucchi, patrick.jermann, guillaume.zufferey, pierre.dillenbourg @epfl.ch

Comments:

Manoj’s Blog

Franck's Blog

Summary:

The article sets out to make a comparison between a touch and a tangible interface. In general touch interfaces are faster than touch interfaces. As touch interfaces take advantage of a broader range of human abilities. Most studies are based on Fitt’s law where time to acquire target is a function of the distance divided by the target size. In this article the digital desk is used in combination with the Bricks system.

Recently there has been an increase in the popularity of touch systems. Tangible systems are seen as more task specific. For the purpose of comparison, both the touch and the tangible systems were made as similar as possible creating equivalent virtual objects for the scale models. A variety of gestures were compared: Addition, Selection, Translation, Rotation, Scaling, Removal, Undo previous action, Submit, Select All, and Adjustment.

Forty experimental subjects were used in the experiment. They were give a total of forty warehouse layouts to implement on both interfaces. their time and accuracy were measured and compared. Overall, the tangible interface was significantly quicker than the touch interface. However, some actions were quicker in the touch interface than in the tangible one such as scaling walls. The poly sensory feedback provided by tangible objects, combined with our natural familiarity handling them was attributed to the overall increase in performance of the tangible interface.

Discussion:

An interesting comparison. However, I feel that it is a little unfair as we have all had a life time experience with tangible interfaces, by comparison our acquaintance with touch interfaces is very short. Given more time I feel that the performance on the touch interface would improve and potentially surpass its tangible counterpart. Additional feedback, other than visual, would have certainly improved the performance of the touch interface.

Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound

Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda

Department of Information Physics and Computing Graduate School of Information Science and Technology The University of Tokyo Eng.Bldg.6, 7-3-1, Hongo, Bunkyo Ward, Tokyo, Japan {iwa,tatezono,shino}@alab.t.u-tokyo.ac.jp http://www.alab.t.u-tokyo.ac.jp/∼ iwa

Comments:

Drew’s Blog

Franck’s Blog

Summary:

The article presents a tactile device that produces 3D stress fields combined with a stereoscopic display to provide 3D tactile feedback for interacting with 3D visual objects. With developments in graphics, physical stimulation and display technology. There is a demand for haptic devices that integrate sight and touch.

Current devices enable the visualization and handling of 3D objects. In addition to this, the ability to physically sense them would enrich the experience. Constant contact systems are not very effective as there is never an absence of touch. Robot systems of feedback are bulky, non portable, and difficult to set up. This paper will present an airborne ultrasonic alternative which has a high temporal and spatial resolution.

The apparatus consists of an array of 96 ultrasonic transducers which are grouped into 12 sets. These are arranged in rings where each members is equidistant from the center of the array. They are able to be used directly on the skin where the cumulative force can be focussed to a point.

The most effective stimulation was obtained through bursts of ultrasonic sound. Continuous activation resulted in only the onset of sound being perceived. 20mm spatial resolution was achieved but the ultrasonic array lacked sufficient strength for a continuous pressure to be perceived.

Discussion:

An interesting approach, where tactile feedback can be applied without continuous contact. This setup does not allow for 2D phase control of the array which would permit the point of focus to be manipulated.

There are many possible applications where feedback would be beneficial as it is non intrusive and does not require anything to be worn. As the array is fixed in space the feedback is anchored to a particular position, similar to real objects.

FreeDrawer – A Free-Form Sketching System on the Responsive Workbench

Gerold Wesche, Hans-Peter Seidel

Comments:

Frank's Blog

Paul's Blog

Summary:

The article presents a virtual reality spline based free-form surface sketching system where the user uses a tracked stylus as an input device. The user draws directly in 3D with tools adapted to the needs of designers. The power and utility of 3D environments for design are well recognized. However, most CAD systems are geared toward elaborate shape design and lack support for concept development. FreeDrawer is a sketching tool dedicated to the initial stages of design, enabling direct transfer into a corresponding computer representation.

Unlike direct surface modeling which is based on free-form solids, free form curves or voxel based virtual clay, FreeDrawer constructs models from a skeleton. This approach avoids both computational load and unwanted artifacts when off axis changes are made. Space curves and 2D curves projected on a plane are supported. Standard editing features are accessed from a hand held wand.

A curve network is used to snap new curves to a position in space relative to other curves already drawn, so in effect they are an approximation of the original. When changes are make to a curve, locally connected curves will adapt up to the next junction. Globally connected curves would undergo a collective adaptation.

Filling in surfaces is achieved within closed loops. There is a closed loop algorithm that searches according to geometric criteria. Kuriyama surfaces and n-sided domains can be selected. However, for surface deformation Catmull-Clark surfaces are closely defined to match the Krriyama surfaces. Curve smoothing, sharpening and dragging are achieved with hand gestures where the region of influence is definable.

A novel tool pallet is presented in the form of a set of radially organized pointers rather like fingers, from which the operator can chose. The point of the desired tool is contacted to the point of application and the selection button pressed. An example of a car seat drawn quickly is given to illustrate the performance of the system.

Conclusions:

The utility and at the same time difficulty in implementation of a force feedback system in this type of environment are mentioned. The tool palate is also identified lacking, and in the process of being redesigned based on experience from the current word. An important case is made that unreasonable demands are placed on the ease and usability of 3D drawing environments for the uninitiated.

Discussion:

FreeDrawer seems to target a highly specific window in the design phase which has not yet be liberated from pen and paper. The features and available tools are all tailored to this end and make a highly compelling case.

It is not surprising that the tool palate did not survive in its original form. Trying to touch a particular object would be awkward particularly for some of the tools, not to mention how one would deal with objects close together.

Furthermore, some question is raised as FreeDrawer’s suitability for all designers. Without a test battery this is hard to judge.

COMPUTER VISION-BASED GESTURE RECOGNITION FOR AN AUGMENTED REALITY INTERFACE

Moritz Störring, Thomas B. Moeslund, Yong Liu, and Erik Granum

Computer Vision and Media Technology Laboratory,

Aalborg University Niels Jernes Vej 14,

DK-9220 Aalborg East, Denmark

{mst,tbm,yliu,eg}@cvmt.aau.dk

Comments:

Drew’s Blog

Manoj’s Blog

Summary:

This paper presents a computer vision based gesture recognition system for an augmented reality interface. It is claimed that wearable computing will soon be able to use head mounted displays (HMDs) in all sorts of applications. This application uses a combination of HMDs and head mounted cameras (HMCs) to overlay information on the view of the real world. Hand gestures were chosen as a mode of communication with the augmented reality interface.

Two traditional types of optical state based gesture recognition are described. The first is model based recognition where the image of the hand is frame by frame fitted to a pre constructed model. The second is an appearance based approach which requires a set of training examples and a classifier. These two methods are dismissed as not being suitable for AR applications due to their high computational requirements.

The goal of the system presented in this article is to provide the foundation for a multiuser round table presentation system. Pointing and clicking are identified as two gestures which are fundamental to any interface. The system will be able to identify a total of six discrete gestures. The plane of sight is limited to 2D in the interests of economy.

Segmentation is achieved by the use of a color pixel approach. HSV and normalized RGB are used to overcome color variation caused by variations in illumination intensity. This method enables colors to be separated from their intensity. Region growing is used to find completed areas.

The image is geometrically segmentation with a pseudo polar transformation. A discrete number of concentric lines are explored to speed up processing. The area of interest is demarcated between the smallest to the greatest radius of skin. A temporal filter is used to differentiate gestures. For pointing, once the gesture has bee identified, the vector is triangulated from the two HMCs. The clicking gesture is identified by the use of a bounding box to identify when the thumb is opposed. The algorithm is described as robust and efficient for gesture recognition.

Discussion:

This paper proposes a comprehensive wearable augmented reality system.

The approach chosen for segmentation with the use of a transformation was of particular interest. The ability to count fingers by transforming the coordinate system into pseudo polar and then counting the number of spikes is robust and independent of finger length.

It is not clear how this would be adapted to three dimensions.