Multi-touch interactions are regarded by the public as intuitive and easy to use, yet there is lack of consistency in implementation among research and commercial products. How can intuition be incorporated and evaluated when designing multi-touch gestural interactions? Towards the Establishment of a Framework for Intuitive Multi-touch Interaction Design paper addressed such a question by reviewing published research on multi-touch interactions to construct a framework of factors that influence intuition.
Multi-touch interaction framework
Intuition is defined as
The power or faculty of attaining to direct knowledge or cognition without evident rational thought and inference.
Intuitive interactions minimize the effort required of end users to learn, use, and remember the interactions. The research proposed five factors that influence intuition of multi-touch interactions: direct manipulation, physics, feedback, previous knowledge, and physical motion.
Direct manipulation interactions mimic the natural laws of physical behavior in the real world. Examples include common multi-touch interactions for moving, resizing, rotating, and zooming objects. Direct manipulation interactions avoid the need to learn & train required for indirect input mechanisms such as mouse or joystick and makes use of real-world physical laws familiar to end users. It might be hard to relate for expert computer users, the video below of Windows 95 usability testing illustrates that double-clicking with a mouse isn’t something that comes naturally to everyone.
Gesture size & gesture speed are 2 aspects of physics commonly integrated into multi-touch interactions. Users associate larger gestures with larger interaction outcomes such as navigating and smaller gestures with detail-oriented interactions such as manipulating interface element. Similarly, users expect faster gesture speed to result in faster movement (e.g. move vs. flick)
Feedback in multi-touch interactions is especially important as screens don’t provide tactile feedback and fingers itself can block visual changes. Users use visual or auditory feedback as context to resolve gesture ambiguity.
Depending on end user’s previous experience, they may have different directionality notions for navigating forwards and backwards. Collective research showed that most users preferred one-handed gestures over two-handed gestures when using multi-touch table computers.
Consensus across research & commercial products
Apple iOS, Google Android, and Windows Phone all support some variant of tap, double-tap, touch and hold, drag, flick, and pinch gestures. One finger tap and one finger drag are the most uniform gestures used, supported by all common multi-touch interfaces. Two-finger pinch and spread for zooming or scaling objects are also widespread. There is less consensus with more abstract interactions such as rotation. Collective research studying users show a wide range of variability in expectations for gestures. Move (one finger touch and drag) and selection had the highest agreement among user studies, while deleting objects and zooming had lower levels of agreement.
The video below illustrates multi-touch gestures as used in Google Earth smartphone app.
For fun, the video below demonstrates one finger touch interactions on a HP-150 PC, a touchscreen computer from 1983, even though it ran MS-DOS and lacked a graphical user interface. Computers have come a long way since then…
Remaining research problems
Designing for multiple interactions that require more interactions than there are simple direct manipulation gestures is one problem. A simple photo browsing app can be designed to use a small set of direct manipulation gestures such as moving, scaling, and rotating; a comprehensive photo editing app like Adobe Photoshop with effects, photo cropping, and free-hand selection requires either designing unique multi-touch gestures or using menus & buttons to trigger different interaction modes to re-use gestures. The analysis of published research also revealed a lack of rigorous evaluation of multi-touch interactions. There’s lack of objective, statistically significant evaluation data, limiting ability to quantify existing knowledge.
Future of multi-touch interactions
The five factors that influence interaction design presented in the research suggests that successful mobile app interactions make appropriate use of direct manipulation that behaves in accordance with real-world physical laws, uses menus & buttons as modifiers to re-use gestures for abstract concepts, and provides a combination of visual and auditory feedback. It was surprising to find that except for one finger touch and touch & drag, research showed low levels of agreement for more complex gestures. Mouse & keyboard enables a rich set of interactions through use of modifiers (right-click, shift key + click, etc). One approach may be development of a standard set of modifiers for one finger touch to enable richer interactions. I’ve previously blogged about research that demonstrated viability of using non-dominant hand to modify actions in a real time strategy game context.
Multi-touch interactions are still in their infancy, and it’s important for products to set good precedents early to avoid burdening future users with unintuitive interactions. The QWERTY keyboard layout was designed to minimize typewriter jams. It’s inefficient & limits typing speed, but remains as the most common keyboard layout on computers due to inestimable cost of change. The computer mouse was first introduced in early 1980s. Single left click, double left click, and single right-click on the mouse are standard mouse operations today while triple-click and mouse chording (e.g. hold the left button and click the right button) are rarely used. I’ll be interesting to see which multi-touch gestures will become the standard ones used 25 years from now.