Is it possible to create a real time strategy (RTS) video game using touch input that matches or exceeds the capabilities skilled players have with mouse + keyboard? I came across a research article High-Performance Pen + Touch Modality Interactions: A Real-Time Strategy Game eSports Context (PDF available here) that answered that question by proposing a new pen + touch interaction system for playing Real Time Strategy (RTS) type of video games such as StarCraft. A 3 minute YouTube video summarizes their research.
Background
Real Time Strategy (RTS) is an extremely popular genre of video game worldwide. The Global StarCraft League (GSL) finals engaged 25,000 co-located and 300,000 online viewers. Unlike other competitive video genres such as first person shooter, arcade style fighter, or racing simulation games, RTS games require repeated rapid, precise, complex, and sustained interactions. The research quoted one top ranked player saying “Everything in these games is about APM [actions per minute].” Popular RTS video games use mouse + keyboard (such StarCraft), and no significant RTS video games have been created that use touch + pen input. Video of top ranked players explaining APM show the screen changing so fast from mouse + keyboard interaction that it becomes nauseating to watch.
iPhone and Android smartphones over the past few years have popularized the concept of using finger as input, and tablets like the iPad take advantage of similar interaction model as smartphones. With the recent release of Windows 8 and Microsoft Surface, and the announcement of iPad mini, more and more computers people use will support touch interaction. Similarly, pen input has been successful in niche markets such as graphic design and CAD for decades. Early examples of digitizer graphics tablet include the Apple Graphics Tablet (1979). Early convertible tablet computers with pen input included the Compaq Concerto (1993) and the Compaq TC1000 (2003).
Early tablet computers with pen input from 2003
Deconstructing a RTS video game
The authors did a great job explaining the gameplay aspects common to all RTS video games. (Those who’ve played StarCraft can skip this section :). In general, all RTS games focus on managing units, game entities that player construct & control. Players typically command hundreds of units simultaneously, and make minute adjustments to each unit. The game interface is built around a main view of the game world accompanied by a “mini-map.”
Screenshot from StarCraft 2
In the common mouse + keyboard interaction scheme, the mouse is primarily used to select units and to invoke commands for selected units. The keyboard is used to invoke commands for selected units by advanced players quicker, and also to access saved groups of units. Expert players make extensive use of keyboard shortcuts (hotkeys and macro) to increase the number of actions per minute they can perform to gain an edge on opponents.
Designing pen + touch interaction system adapted for RTS games
The authors designed a pen + touch interaction system optimized for five RTS game mechanics: unit selection, commands, macros, view manipulation, and control groups. Watch a video demonstrating the different components of the interaction system.
Single units are selected with pen taps. Instead of drawing a selection rectangle with mouse groups of units are selected with pen using a unique isosceles lasso select technique. The technique optimizes for a short pen stroke by magnifying the area covered by a pen stroke while preserving a precise & non-precise selection edge. Watch the video at 0:24 that demonstrates the selection method.
Commands are selected using an edge-constrained multi-finger command selection panel instead of using mouse click & keyboard shortcut combinations. Watch the video at 0:35 that demonstrates this.
Macro for commanding factories to construct new units in game are designed as in-context touch menu that appears next to the selected factory object instead of a series of keyboard shortcuts. To support efficient performance by expert players, duplicate view of all factory objects are shown in an array at the bottom of the screen, enabling expert players to save the time required to change the game view to the location of the factory and increasing the number of actions per minute the player can issue.
View manipulation is accomplished using touch, using common gestures for panning, zooming, and rotating views that duplicate gestures on iPhone maps instead of using the mouse. Touch buttons are used to save and recall favorite camera views instead of keyboard shortcuts.
Control groups are programmed & selected using a vertical array of buttons in the user interface instead of keyboard shortcuts.
Did it work?
If given equal training, would a RTS player using pen + touch be competitive against a player using mouse + keyboard? Using actions per minute as a basis for measuring competitiveness, the question can be rephrased as “Would a RTS player be able to perform equal or more actions per minute using pen + touch compared to mouse + keyboard?”
Researchers tested the pen + touch interaction system with participants familiar with playing RTS games using mouse + keyboard. Participants found the pen + touch system to be more fun, engaging, and expressive, while there was more variance in reported results comparing difficulty, speed, and precision. While participants didn’t have time to extensively train using the pen + touch interaction model, all but one of the participants reported they felt they could become as good with pen + touch as they are with mouse + keyboard.
Would this work in real life?
Would a RTS video game released with the pen + touch interaction system be commercially successful? That depends on if players are willing to invest the time to learn a new control system, how the game’s pen + touch interactions match or differ with pen + touch interactions of the operating system, and if computers are available at an appropriate price point with technology that supports pen + touch interactions proposed.
The research demonstrated that expert players could use a pen + touch to play RTS games as well as using traditional mouse + keyboard mechanism. The pen + touch interaction system would require significant training and learning for new players. The researchers acknowledge the interaction system is “not designed for novices, and some interaction techniques are not easily recognizable or intuitive.” Researched provided participants evaluating the system a tutorial with Q&A and 15 minute of practice time in order to learn rules such as a single touch with non-dominant hand in the command selection panel is selection or default command, 2 finger touch is fight command, and 3 finger touch is for constructing new units is not intuitive.
The authors used a Wacom Cintiq 21UX 21” LCD monitor/digitizer hybrid with custom prototype ZeroTouch multi-touch sensor. Touchscreen & pen input technology in popular tablets today don’t match the price & hardware capabilities of the custom hardware used. The $1,999 LCD monitor/digitizer and $2,500 prototype multi-touch sensor is an order of magnitude more expensive than the $127 for the LCD display and touchscreen in the iPad. The Lenovo X230t convertible tablet has a 12.5” screen and has both multi-touch and pen digitizer, but doesn’t accept simultaneous pen & touch input. When the pen digitizer hovers or touches the screen, “palm rejection disables” touch input to allow the palm to rest on the screen while using the digitizer without interfering with selection. Smaller screen size of tablets would not accommodate the large number of software buttons required in the author’s interaction scheme without covering up the majority of the game play area.
What does this mean?
Previous commercial attempts at RTS games using game controllers such as Halo Wars on XBOX 360 received mixed reviews overly simplified gameplay due to limitations of the input mechanisms that didn’t provide opportunities for improving skill with more time spent. The research demonstrated a fun, engaging, and expressive interaction system for playing RTS games using pen + touch, showing:
- an efficient interaction system that uses both dominant and non-dominant hands
- edge-constrained touch buttons for efficient touch targeting with the non-dominant hand
- natural view manipulation using touch
- graphical hotkeys and multi-touch selection panels replacing the keyboard for interaction with non-dominant hand
While the interaction described in the research can’t be directly applied to current generation of tablet hardware, I’m excited about the potential that exists. Current popular touch games like Angry Birds, Cut the Rope, and Plants vs Zombies use only basic direct-manipulation touch interactions that only use the dominant hand, but I’m looking forward to more complex touch games in the future that effectively use the non-dominant hand and can support rapid, precise, and complex interactions.