COMPUTER VISION-BASED GESTURE RECOGNITION FOR AN AUGMENTED REALITY INTERFACE

Moritz Störring, Thomas B. Moeslund, Yong Liu, and Erik Granum

Computer Vision and Media Technology Laboratory,

Aalborg University Niels Jernes Vej 14,

DK-9220 Aalborg East, Denmark

{mst,tbm,yliu,eg}@cvmt.aau.dk

Comments:

Drew’s Blog

Manoj’s Blog

Summary:

This paper presents a computer vision based gesture recognition system for an augmented reality interface. It is claimed that wearable computing will soon be able to use head mounted displays (HMDs) in all sorts of applications. This application uses a combination of HMDs and head mounted cameras (HMCs) to overlay information on the view of the real world. Hand gestures were chosen as a mode of communication with the augmented reality interface.

Two traditional types of optical state based gesture recognition are described. The first is model based recognition where the image of the hand is frame by frame fitted to a pre constructed model. The second is an appearance based approach which requires a set of training examples and a classifier. These two methods are dismissed as not being suitable for AR applications due to their high computational requirements.

The goal of the system presented in this article is to provide the foundation for a multiuser round table presentation system. Pointing and clicking are identified as two gestures which are fundamental to any interface. The system will be able to identify a total of six discrete gestures. The plane of sight is limited to 2D in the interests of economy.

Segmentation is achieved by the use of a color pixel approach. HSV and normalized RGB are used to overcome color variation caused by variations in illumination intensity. This method enables colors to be separated from their intensity. Region growing is used to find completed areas.

The image is geometrically segmentation with a pseudo polar transformation. A discrete number of concentric lines are explored to speed up processing. The area of interest is demarcated between the smallest to the greatest radius of skin. A temporal filter is used to differentiate gestures. For pointing, once the gesture has bee identified, the vector is triangulated from the two HMCs. The clicking gesture is identified by the use of a bounding box to identify when the thumb is opposed. The algorithm is described as robust and efficient for gesture recognition.

Discussion:

This paper proposes a comprehensive wearable augmented reality system.

The approach chosen for segmentation with the use of a transformation was of particular interest. The ability to count fingers by transforming the coordinate system into pseudo polar and then counting the number of spikes is robust and independent of finger length.

It is not clear how this would be adapted to three dimensions.