Webcam Mouse Using Face and Eye Tracking in Various Illumination Environemts

Yuan-Pin Lin, Chung-Chih Lin, Jyh-Horng Chen

Institue of Electrical Engineering,

National Taiwan University

Yi-Ping Chao,

Department of Computer Science and Information Engineering,

Chang Gung University

Comments:

Manoj’s Blog

Summary:

Introduction:

This article presents a way of using a screen mounted webcam to aquire human gestures as a substitute input for the mouse. K-Nearest neighbor classifier in combination with an adaptive skin model is used to provide real time tracking. The system implemented on a standard laptop is able to perform at 15fps.

This system is proposed as an alternative interface for the disabled. Unlike traditional solutions it is minimally intrusive, not requiring the user to wear additional devices. The system requires a dedicated machine to perform the visual tracking whose data is forwarded to the computer that is being operated by the user.

Face tracking is performed using a non linear skin color transform to overcome variations in lighting. Iris tracking is used to identify the eyes and their position. The cursor is then calculated as a product of the eye position relative to the head position. Turning the head has the effect of moving the cursor from side to side.

Feature Recognition:

After defining the KNN features for recognizing illumination conditions. An elliptical model was used with 10 images under different illumination conditions. An average of 92% accuracy for skin detection was achieved. The face was distinguishable due to its clustered appearance.

Eye tracking was then performed with a frame capture comparing the centers of the face and eyes.

Results:

The system runs on a standard 2.4-GHz laptop dedicated to tracking the position of the head and eyes at 15fps occupying 45% of the windows resources. It successfully tracks the head and eyes under a variety of different lighting conditions with complex real world backgrounds.

Discussion:

Calculating the vectors of the face and eyes separately and then determining the point of gaze as a product of the eye vectors relative to the face is a good approach which should lead to improved accuracy. Although it is mentioned that the system is under development. It would have been beneficial to have provided some details as to how the system was tested.

Real-Time Hand-Tracking as a User Input Device

Robert Y. Wang

Computer Science and Artificial Intelligence Laboratory,

Massachusetts Institute of Technology

Cambridge, MA USA

Comments:

Franck’s Blog

Summary:

Introduction:

This paper presents an affordable 3D articulated user input system for the hand. An ordinary cloth glove which has been imprinted with colors is used which facilitates the effectiveness of a KNN approach. This implementation allows the user to perform true 3D gestures which are not constrained to a single plane. This system is a compromise position between a bare hand capture and a wearable motion detection system. The graphical pattern on the glove facilitates faster and more accurate pose estimation.

Pose Estimation:

The system is based on single frame pose estimation. Unlike bare hand estimation where very different poses can map to the same image without intensive evaluation. The use of the colored glove ensures the very different poses always map to different images. The implementation of the KNN is adapted use a Hausdorff-like distance metric.

Database Sampling:

Low dispersion sampling of hand poses is used to provide a maximum bound on the estimation error that the algorithm can make. The result is to have a database where the distance from the KNN to the query image is minimized.

Fast Nearest Neighbor Search:

KNN for large databases can be computationally expensive. To speed classification, each pose is compressed into a 128-bit binary code.

Database Coverage Evaluation:

The average performance of 50 test poses which have been randomly sampled is measured. The distance to the nearest neighbor is measured for each pose. Both pose and image distance nearest neighbor improve with increasing database size.

Applications:

This application was intended for 3D direct manipulation tasks such as 3D modeling. Other application could be 3D character control which would certainly be an improvement on the paper summarized earlier in the semester where the implementation was based around a P5 data glove. Interaction with large displays from a distance etc.

Discussion:

This is a very nice approach with the use of a simple yet highly effective solution in the colored fabric glove. It would have useful to have included some data to illustrate the performance of the implementation. This type of feature identification for video tracking is very powerful and obviously has no limit to its applications.

Liquids, Smoke, and Soap Bubbles -

Reflections on Materials for Ephemeral User Interfaces

Axel Sylverter

AGIS,

University of Hamburg,

33537 Hamburg, Germany

Tanja Döring, Albrecht Schmidt

Pervasive Computing,

University of Duidburg-Essen,

45117 Essen, Germany

Comments:

Drew’s Blog

Franck’s Blog

Summary:

Introduction:

This article presents an interface based on the creation manipulation, and short life of bubbles. The choice of this implementation encourages the users to interact with a medium that is very fragile, has a short life and exhibits random behavior.

The use of different materials to provoke specific qualities in human computer interaction is the key point explored by this work.

Implementation:

A 20 inch circular pool with an under mounted camera is used on which bubbles are floated. The bubbles can be either clear or smoke filled. The under mounted camera is able to track the diameter and position of the bubbles by the ring that is formed by surface tension between the skin of the bubble and the liquid in the pool. The user can move the bubbles by either blowing or touching them.

The system was used to control the lighting in a room where the size of a bubble was used to control brightness.

Discussion:

An interesting direction to go. The random component of the setup could add interest and a life of its own to the application. The artistic component and soft imprecise nature of bubbles could be a nice contrast to many applications where the interface could provide both control and feedback.

EyeDraw: Enabling Children with Severe Motor Imparements to Draw with Their Eyes

Anthony J. Hornof

Computer and Information Science,

University of Oregan,

Eugene, OR 97403 USA

Anna Cavender

Computer Science and engineering,

University of Washington,

Seattle, WA 98195 USA

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

Introduction:

This is an application that enable children with severe motor disabilities to draw pictures with their eyes. The application runs on a computer equipped with an eye tracking device.

In order to draw a line the start and end points are identified from successive gaze positions. The line is then automatically created by connecting the dots. The midas touch problem is avoided by the use of dwell time accompanied with visual feed back in the form of a change in the shape of the cursor.

Drawing Commands:

Alternation between viewing and drawing is performed by dwell time. The cursor changes from green for viewing to red for drawing. The dwell time threshold is adjustable based on experience but starts at a half second. A drawing command can be cancelled by extending the dwell for another half second.

Version 1:

This is a minimal control version which has the following tools

Line drawing,

Circle drawing,

Undo Button,

Grid to assist with dwell stabilization,

Save and open drawings.

Evaluation of Version 1:

participants without disabilities were first recruited. After calibration of the eye tracker and a short familiarization with the controls the participants were asked to make some drawings.

500ms was found to be the preferred dwell time.

The easiest functions were clicking buttons.

The hardest functions were drawing and controlling the drawing.

The grid was found to be useful.

Overall the task required a lot of focussed attention.

A second evaluation was performed by impaired users which lead to the second version of the program.

Version 2:

This version of the program was in response to the needs of the impaired users and contained the following additions:

Image of what the camera sees so the user can stay in an optimal position while drawing.

More user defined settings such as dwell time etc.

Audi feedback on the current state of the cursor while drawing.

Rectangle and polygon tools were added.

Colors.

On off switch for the eye tracking.

Evaluation of Version 2:

As before the evaluation was divided into two groups. Both found the rich palate of tools engaging, with the consequence that an extended period of familiarization was required. The next version will have a feature that reveals tools one at a time to enable first time users to start drawing and not be over faced by the apparent complexity of the application.

Discussion:

This is a very interesting article. The evaluation of the first two versions of the application is very thorough in its descriptions of each individuals reaction to the system. Some information about the implementation of the eye tracking would have been of interest. One of the users is reported to have written text at the bottom of their picture. Again, some information on how the keyboard was represented and controlled would have been of interest. Very little is mentioned about participants reaction to using eye movements deliberately. This is only touched upon by mentioning that a lot of attention was required.

Towards Natural Gesture/Speech HCI: A case Study of Weather Narration

Indrajit Poddar, Yogesh Sethi, Ercan Ozyidiz, Rajeev Sharma

School of Computer Science and Engineering,

Pennsylvania State University,

University Park PA 16802

Comments:

Drew’s Blog

Franck’s Blog

Summary:

Introduction:

This paper is concerned with developing recognition techniques which are capable of handling continuous natural gesture and speech inputs. Previous studies have focussed on presenting data which is bounded with a clear beginning and end. This paper uses an HMM approach with naturalistic data collected from a weather narrator. A continuous gesture recognition framework has been combined with a co-occurrence analysis of speech to determine the interaction between the two modalities.

Gestures and Speech:

Three gesture types were identified for recognition as clearly conveying information:

Pointing

Area

Contour

Each of these can be further decomposed into:

Preparation phase in the beginning

Retraction phase at the end

Actual stroke phase in the middle

Compound gestures can therefore be formed through the combination of any of the above mentioned gesture types. The HMMs used in the implementation are based on this premise.

The streaming video was segmented to extract critical features of the presenters face and two hands. Bootstrapping was employed to improve performance. Gestures were mostly single handed but it was important to identify which hand was being used. As expected continuous gesture recognition has a lower recognition rate than isolated gesture recognition.

Keywords:

here, up here, down here

east(ern), west(ern), etc

Name of some place

To examine the interaction of speech with gestures initially needed the identification of keywords and their interrelationship with gestures.

Evaluation:

85% of the time a meaningful gesture is made it is accompanied by a spoken keyword.

83% of the time a co-occurrence of the keyword with a gesture phase.

56% of the time a co-occurrence of the keyword with “point”.

Discussion:

The use of a multimodal input in HCI interfaces is a very interesting concept. In norma human interaction this is a critical element to communication. It is all to familiar that a face to face conversation is much more revealing than one conducted over the telephone of some other restricted modality channel.

Coming to Grips with the Objects We Grasp:

Detecting Interactions with Efficient Wrist-Worn Sensors

Eugen Berlin, Jun Liu, Kristof van Laerhoven, Brent Schiele

Department of Computer Science,

Techniche Universität Darmstadt,

Darmstadt, Germany

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

Introduction:

This article describes a wrist worn sensor that is able to identify nearby objects with RFID tags and also the gestures performed by the user through accelerometer data. In particular this paper presents the technical challenges encountered in the development of a wrist worn device to perform this task.

System:

RFID antenna

M1-mini SkyeTek reader interface circuitry

3D accelerometer

Skin temperature sensor

Two ambient light sensors

Challenges:

Extending the range of the RFID from the wrist to the hand.

Choice of antenna to best perform.

Density of RFID tags and frequency of measurements to be able to determine which item is being picked up.

Classification of 3D accelerometer data.

The Box Test:

Used in optimizing the antenna.

Objects with RFID tags were placed in a box and then individually picked up. The Hit Rate was then calculated as the number of correct identifications divided by the overall number of attempts. This test had the advantage that it had many elements which would de present in a real world application and was therefore more representative.

Variables to optimize using the box test:

Antenna shape

Q-Value, the quality or reading range

RFID Reading Frequency

Optimizing Accelerometer Data (Inertial Sensing):

How the essence of the gestures was captured. The segmentation and classification phase were combined. The data is compacted into a series of linear segments which are then compared to members from a known set with a sliding window paradigm.

Gardening and Domestic Cleaning Scenarios:

These were chosen as test beds because of their real environments containing multiple pre tagged tools and tasks.

Discussion:

This paper presents a more practically usable implementation of an RFID/accelerometer bracelet combination for identifying objects and gestures. The various challenges encountered in the development are outlined leading to an efficient lightweight solution. A classification method for dealing with the accelerometer data was also adapted to run on the reduced resources of the mobile platform.

The Peppermill: A Human-Powered User Interface Device

Nicolas Villar,

Steve Hodges,

Microsoft Research

Camebridge, UK

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

Introduction:

A user interface that sources its power from the effort required to operate it. This is a proof of concept device called the pepper mill which is used as an input device for multimedia browsing applications. The action involved in using the device as a viable alternative to batteries, with economic and ecological benefits. The transducer both senses the users intent and also generates the required power for the device to function. These characteristics are requirements of what are called human powered devices.

Background:

The Space Commander, developed by the Zenith corporation in 1955 as a TV remote control. This device was mechanical generating high frequency sounds which were received by a microphone and decoded. Simpler devices such as windup flash lights and radios have been around for a longtime. Body-worn systems to harvest energy to power cell phones etc are more recent additions. Another example which is mentioned is the MIT self powered button.

The Implementation:

The circuit diagram is presented showing the relatively simple implementation.

Standard components are used with a motor and reduction gear assembly providing the source of power.

The PepperMill Controller:

The controller is designed as a generic controller reminiscent of a culinary pepper mill hence the name. There is a total of four input DOF. the three buttons acting as modifiers to the rotary action. A micro controller, and radio transmitter are also incorporated in the package. A receiver is required on the other end which in this case was a USB serial device.

Evaluation:

An artificial scenario video browsing application was developed. Users could choose channels, and adjust volume. The rate of rotation determined the speed of selection of items on the screen.

Discussion:

A very interesting idea. The Zenith remote from the 50s is a brilliant use of available resources. Why they chose a human powered device when they could have just as easily had a radio version is additionally interesting. Small human powered devices have limitless applications, one is always needing to change batteries when one least need to. The smaller the devices the harder and more fragile this procedure can be. A very interesting paper.

The Wiimote with Multiple Sensor Bars:

Creating an Affordable, Virtual Reality Controller

Torben Sko,

Henry Gardner,

School of Computer Science,

College of Engineering and Computer Science,

Canberra, ACT, 0200, Australia

Comments:

Manoj’s Blog

Franck’s Blog

Summary:

Introduction:

The attributes of the Wiimote are summarized along with the multi various applications to which it has been adapted. The paper focusses on extending the limited field of view of the IR camera by the use of multiple sensor bars so that the Wiimote can be used as a VR controller.

System:

1 Wiimote

5 Sensor bars

1 Nunchuk

The Wiimote has beeb modified into a gun with a grip, the Nunchuck is used in the left hand to move the character position.

A mode based reflex agent structure is used to track the position of the Wiimote. This includes a dynamic model of what it has seen and a static model of the layout of the sensor bars. At least two IR sources have to visible at all times and no more than four. When the system is initialized it aligns itself with the central sensor bar, that is to say the center of the static model. Moving from pair of sensor bars to another is performed by comparing models and discarding the one that is inconsistent. In this way errors in localization can be minimized. The system is also able to deal with the Wiimote loosing view of all sensor bars as long as the last one seen returns.

User Testing Setup:

A two wall back projection display system was used to evaluate performance. The VR Wiimote controller was integrated into a first person shooter game engine. A 3D real time cursor was also provided to give visual feedback.

Two games were used in the evaluation:

A custom built walk and explore game

half-Life 2

Evaluation:

12 users were tested all of which had had experience with games 8 of which with the Wii. They were given a simple explore and locate objects task, followed by a post test questionnaire. The purpose of this test was to determine if the sensor bar array was disturbing. None of the participants found it distracting, some even failed to notice it at all.

The next phase of the testing was to determine the usability of the setup. In particular, how frequently it would need resetting. Most users enjoyed the experience there were frustrations expressed with the controls. Additionally, some reported discomfort with the motion of the scenery.

Finally, after reducing the scenery rotation speed and moving the seat position further back. A subset of the users from the previous test with the highest reset rates were chosen. The new arrangement successfully reduced their error rates by 35%. Post testing the responses were more favorable finding the controller more natural.

Discussion:

Another interesting application of the Wiimote. An affordable VR theatre with a system of extending the field of view of the IR camera for position tracking. This will open up the Wiimote to many applications which before were not viable due to the narrowness of the field of view.

All that is needed now is for a way of increasing the range reducing the sensitivity to ambient light interference.

Whack Gestures: Inexact and Inatentive Interaction with Mobile Devices

Scott E. Hudson

Chriss Harrison

Beverly L. Harrison

Human-Computer Interaction Institute Carnegie Mellon University, Pittsburgh, PA 15213

Anthony LaMarca

Intel Labs Seattle, WA 98105

Comments:

Manoj's Blog

Frank's Blog

Summary:

Whack Gestures is an imprecise inattentive interaction technique which enables the user to interact with devices without the use of detained vision or fine motor skill. An application of this methodology would be interacting with a mobile device without getting it out with crude eye movements or a grasping action. The utility of this approach is that ongoing activities do not have to be interrupted. The purpose of the article is to present such a system and demonstrate its feasibility.

Advances in mobile device technology have broadened the application of mobile computing devices. New interaction methods are required to maximize their effectiveness, and avoid frustrations, which are typified by cell phones ringing at inappropriate times. In situations where interaction with a mobile device creates an interruption it is important that interaction can be carried out with minimal attention being drawn away from the primary task. The current study examines the use of a set of crude communication gestures, some of which do not require the device to be pulled out.

A small vocabulary of gestures to deal with typical events such as silencing a ringing cell phone, responding to a message with yes, no, forward, etc was developed. The gestures were recorded with an accelerometer and consisted of hitting or whacking the mobile device with either the heel or the palm of the hand. Whacks were chosen as they can be organized highly accurately, and can be recorded by accelerometers which are increasingly present in mobile devices.

Whacks by themselves are hard to distinguish from bumps, so to minimize false positives a pair of whacks was used to frame a gesture as follows . In this way by requiring the whacks to be of similar magnitude and temporally close to the gesture a high signal to noise ratio could be achieved.

The three gestures used for evaluation:

whack-whack An empty gesture.

whack-whack-whack Whack as a signal gesture.

whack-wiggle-whack Shaking as a signal gesture.

To implement and test this technique a mobile sensor platform containing a variety of resources including a 3D accelerometer was used. Data was samples at 256Hz and stored on flash memory for post processing. A recognizer was developed to differentiate above background acceleration changes along the approximate vector of impact. Whacks are differentiated by subtracting an exponential decay average from the raw data. The whacks are then characterized by short sharp pulses. Once a whack has been detected the next 300ms is examined. A framing whack must occur within three seconds and with a magnitude of plus or minus 33%. If more than one are detected the final one is treated as the framing closure. The signal portion between the frames is passed to a secondary recognizer. Average energy is used to identify whack wiggle whack, and peak detection with a min separation of 200ms is used for whack whack whack.

Evaluation:

11 volunteers subjects were used.

Given the MSP and shown how to attach it and instructed to go about their normal activities.

In this way 22hours of baseline data was collected.

Subjects were then given a brief instruction of the three gestures, and were asked to perform them three times each. This data was to be used to train the recognizer.

6 of the subjects data was withheld as a validation set.

The ability to avoid false positives was the primary concern.

Detecting the framing whacks was first determined.

Data:

1 false positive / 12hr of use under normal activity from the validation set.

No false positives from the training data?

100% correct classification of the original signals performed artificially by the recognizer.

97% correct classification overall by the recognizer.

Discussion:

The need for gesture based communication with devices which does not require a large amount of attention is a most interesting idea. The choice of hitting the device is sound. However, I am not sure how one would wiggle the device without pulling it out.

A comparison between alternative gestures is mentioned regarding video and survey studies but no information is given. Only that striking or whacking was rated most positively amongst the six alternatives. It would have been interesting to have presented the details of the alternatives as well as the study.

Finally, no information was given on how the evaluation was conducted to collect data during the two hour period:

What sort of stimuli were the subjects to respond to.

How many were presented.

Were the subjects tested under similar conditions.

what were the qualitative reactions to the system.

The applications of a gesture based communication system with minimal attentional demand is of great interest with many applications.