Real-Time Hand-Tracking as a User Input Device

Robert Y. Wang

Computer Science and Artificial Intelligence Laboratory,

Massachusetts Institute of Technology

Cambridge, MA USA

Comments:

Franck’s Blog

Summary:

Introduction:

This paper presents an affordable 3D articulated user input system for the hand. An ordinary cloth glove which has been imprinted with colors is used which facilitates the effectiveness of a KNN approach. This implementation allows the user to perform true 3D gestures which are not constrained to a single plane. This system is a compromise position between a bare hand capture and a wearable motion detection system. The graphical pattern on the glove facilitates faster and more accurate pose estimation.

Pose Estimation:

The system is based on single frame pose estimation. Unlike bare hand estimation where very different poses can map to the same image without intensive evaluation. The use of the colored glove ensures the very different poses always map to different images. The implementation of the KNN is adapted use a Hausdorff-like distance metric.

Database Sampling:

Low dispersion sampling of hand poses is used to provide a maximum bound on the estimation error that the algorithm can make. The result is to have a database where the distance from the KNN to the query image is minimized.

Fast Nearest Neighbor Search:

KNN for large databases can be computationally expensive. To speed classification, each pose is compressed into a 128-bit binary code.

Database Coverage Evaluation:

The average performance of 50 test poses which have been randomly sampled is measured. The distance to the nearest neighbor is measured for each pose. Both pose and image distance nearest neighbor improve with increasing database size.

Applications:

This application was intended for 3D direct manipulation tasks such as 3D modeling. Other application could be 3D character control which would certainly be an improvement on the paper summarized earlier in the semester where the implementation was based around a P5 data glove. Interaction with large displays from a distance etc.

Discussion:

This is a very nice approach with the use of a simple yet highly effective solution in the colored fabric glove. It would have useful to have included some data to illustrate the performance of the implementation. This type of feature identification for video tracking is very powerful and obviously has no limit to its applications.