Whack Gestures: Inexact and Inatentive Interaction with Mobile Devices

Scott E. Hudson

Chriss Harrison

Beverly L. Harrison

Human-Computer Interaction Institute Carnegie Mellon University, Pittsburgh, PA 15213

Anthony LaMarca

Intel Labs Seattle, WA 98105

Comments:

Manoj's Blog

Frank's Blog

Summary:

Whack Gestures is an imprecise inattentive interaction technique which enables the user to interact with devices without the use of detained vision or fine motor skill. An application of this methodology would be interacting with a mobile device without getting it out with crude eye movements or a grasping action. The utility of this approach is that ongoing activities do not have to be interrupted. The purpose of the article is to present such a system and demonstrate its feasibility.

Advances in mobile device technology have broadened the application of mobile computing devices. New interaction methods are required to maximize their effectiveness, and avoid frustrations, which are typified by cell phones ringing at inappropriate times. In situations where interaction with a mobile device creates an interruption it is important that interaction can be carried out with minimal attention being drawn away from the primary task. The current study examines the use of a set of crude communication gestures, some of which do not require the device to be pulled out.

A small vocabulary of gestures to deal with typical events such as silencing a ringing cell phone, responding to a message with yes, no, forward, etc was developed. The gestures were recorded with an accelerometer and consisted of hitting or whacking the mobile device with either the heel or the palm of the hand. Whacks were chosen as they can be organized highly accurately, and can be recorded by accelerometers which are increasingly present in mobile devices.

Whacks by themselves are hard to distinguish from bumps, so to minimize false positives a pair of whacks was used to frame a gesture as follows . In this way by requiring the whacks to be of similar magnitude and temporally close to the gesture a high signal to noise ratio could be achieved.

The three gestures used for evaluation:

whack-whack An empty gesture.

whack-whack-whack Whack as a signal gesture.

whack-wiggle-whack Shaking as a signal gesture.

To implement and test this technique a mobile sensor platform containing a variety of resources including a 3D accelerometer was used. Data was samples at 256Hz and stored on flash memory for post processing. A recognizer was developed to differentiate above background acceleration changes along the approximate vector of impact. Whacks are differentiated by subtracting an exponential decay average from the raw data. The whacks are then characterized by short sharp pulses. Once a whack has been detected the next 300ms is examined. A framing whack must occur within three seconds and with a magnitude of plus or minus 33%. If more than one are detected the final one is treated as the framing closure. The signal portion between the frames is passed to a secondary recognizer. Average energy is used to identify whack wiggle whack, and peak detection with a min separation of 200ms is used for whack whack whack.

Evaluation:

11 volunteers subjects were used.

Given the MSP and shown how to attach it and instructed to go about their normal activities.

In this way 22hours of baseline data was collected.

Subjects were then given a brief instruction of the three gestures, and were asked to perform them three times each. This data was to be used to train the recognizer.

6 of the subjects data was withheld as a validation set.

The ability to avoid false positives was the primary concern.

Detecting the framing whacks was first determined.

Data:

1 false positive / 12hr of use under normal activity from the validation set.

No false positives from the training data?

100% correct classification of the original signals performed artificially by the recognizer.

97% correct classification overall by the recognizer.

Discussion:

The need for gesture based communication with devices which does not require a large amount of attention is a most interesting idea. The choice of hitting the device is sound. However, I am not sure how one would wiggle the device without pulling it out.

A comparison between alternative gestures is mentioned regarding video and survey studies but no information is given. Only that striking or whacking was rated most positively amongst the six alternatives. It would have been interesting to have presented the details of the alternatives as well as the study.

Finally, no information was given on how the evaluation was conducted to collect data during the two hour period:

What sort of stimuli were the subjects to respond to.

How many were presented.

Were the subjects tested under similar conditions.

what were the qualitative reactions to the system.

The applications of a gesture based communication system with minimal attentional demand is of great interest with many applications.