Gestures are an essential element in the realization of paper-like user interfaces. Unfortunately, poor design and recognition of gestures has impeded the adoption of these interfaces. This paper describes a survey intended to illuminate the problems users have and benefits users enjoy with gesture-based user interfaces. From the results of the survey, we conclude that: users value gestures yet problems with gestures remain; users demand more gestures; and Newtons are used largely as notebooks whereas Pilots are used mostly has personal datebooks and addressbooks. The results of the survey provide insight for designers of pen-based user interfaces and related tools.
3.0 ResultsPen and paper has a long history as a way of recording many kinds of
information. Pen-based user interfaces promise to have many of the benefits
of pen and paper, but current pen-based user interfaces have not lived
up to this ideal. In particular, gestures in current interfaces are poorly
recognized by the computer and difficult for users to learn.
This paper describes a survey designed to discover problems and benefits
of gestures on Personal Digital Assistants (PDAs), especially the Apple
Newton and US Robotics PalmPilot. Specifically, we wanted to find out what
applications are used, how gestures are used in practice, why gestures
are not used, and, in general, what users think about gestures.
We expected to find that gestures are infrequently used because people
have difficulty learning or remembering them, or because they are often
misrecognized by the PDA.
However, most respondents to our survey find gestures valuable and
would like to use them for more operations and applications. At the same
time, gesture recognition and memorability could still be improved.
The remainder of this paper describes the survey methodology, the survey
results, and future work.
Survey participants were solicited from several Usenet newsgroups related to the Newton and Pilot PDAs and to pen-based user interfaces in general. Specifically, we posted the call for participation to the following Usenet newsgroups: alt.comp.sys.palmtops.pilot, comp.sys.palmtops, comp.sys.newton.misc, and comp.sys.pen.
The newsgroup message contained a very general description of the research and a URL for the questionnaire itself, which was a World Wide Web page [11].
Respondents were asked to submit the form only once, but we could not determine a simple method to enforce this constraint. Instead, after the data was collected it was sorted by respondent IP address and examined by hand for multiple submissions. Three respondents submitted the form twice and two submitted the form three times. Multiple submissions were deleted, so the analyzed data contains one entry per respondent.
2.1 Questionnaire
overview
2.0 Method
2.2 Questionnaire details
The questionnaire asked about the following topics:
The majority of questions were multiple choice, but free response questions were also included for general comments about gestures and the survey itself. Respondents were required to answer all multiple choice and demographic questions before the questionnaire could be submitted.
2.2 Questionnaire
details
2.0 Method
2.1 Questionnaire overview
3.0 Results
Answers to most frequency questions (e.g., “How often do you use the delete gesture?”) were multiple choice: “never”, “rarely”, “often”, and “very often.” We decided to use a four choice scale rather than a five choice scale because we wanted to force respondents to state a preference rather than pick the middle choice.
The only question for which these four choices were not used asked the frequency of use of the PDA in general. To get a more quantitative measure, the following answers were provided: “less than once per day”, “once per day”, “2-5 times per day”, “more than 5 times per day.”
For questions involving a value judgement (e.g., “How would you rate
the accuracy of your PDA's gesture recognition?”), the multiple choice
answers provided were “terrible,” “bad,” “good,” and “excellent.”
One section of the survey asked respondents about handwriting.
It asked if respondents used built in handwriting recognition and how they
would rate its accuracy. Also, we asked if respondents used Graffiti1,
and if so how they would rate its accuracy.
A significant part of the survey concerned thirteen operations that
a gesture might invoke, shown in Table 1.
| Operation | Newton gesture | Pilot gesture |
| Delete |
 |
|
| Select |
 |
|
| Insert line/paragraph |
 |
|
| Insert letters/words |
 |
|
| Move cursor |
 |
 |
| Next field |
 |
|
| Previous field |
 |
|
| Open record |
 |
|
| Undo | ||
| Close | ||
| Scroll up | ||
| Scroll down | ||
| Transpose |
For each operation, respondents were asked how often they used a gesture for that operation and why they did not use it more often. The questionnaire did not give any indication as to whether there was a gesture for each operation, or if the operation was even possible on the PDA. We carefully considered reasons why PDA users might not use gestures and included the following ones on the questionnaire:
|
|
The questionnaire also asked how often respondents used six common applications: calendar, address book, to-do list, electronic mail, drawing, and note taking. Respondents were asked to rank these applications according to how often they were used. Instead of solely using a numeric rank, respondents could also select a “not available/never used” answer. Respondents were asked to give each application a unique numeric rank. Unfortunately, the form did not enforce this restriction and some respondents chose the same rank for multiple applications.
To find out about tasks that PDAs might support better, we asked respondents what common task they performed on paper but did not perform on their PDA. In addition, we asked how often they performed this task, and why they did not use the PDA for it.
Several questions asked how PDAs are used in meetings or discussions. Specifically, we asked respondents:
Finally, demographic information about respondents was gathered. These questions asked users to specify the following: age, gender, level of education, technical sophistication, and occupation.
Respondents specified age in years in a free-response box. Four responses were provided for education: “high school”, “some college”, “college degree”, “master’s/professional degree”, “PhD/MD.” For technical sophistication, four numbered choices were provided, with one end labeled “not at all” and the other “extremely.”
Our questionnaire included questions on several different topics. The following subsections present the results about the following topics: demographics, PDA usage, gesture usage, opinions about gestures, handwriting, application usage, paper vs. PDAs, and PDA meeting usage.
3.1 Demographics
3.0 Results
3.2 General PDA usage
One hundred forty-two users responded to the survey. Of these, 42 currently use Newtons, 99 use Pilots, and one uses another PDA. For many questions, responses differed substantially depending on the type of PDA used, therefore Newton users and Pilot users will be analyzed separately and we ignore the one other.
The most common profession was computer programmer/software engineer (38% of Newton users, 27% of Pilot users). The next most common was sales/marketing (10%) for Newton users and manager/executive (20%) for Pilot users. Significantly, one third of Pilot users and half of Newton users had a technical job dealing with computers.
Respondents as a whole were technically sophisticated. On a technical sophistication scale of 1 to 4, only 4% of Pilot users ranked themselves in the lower (less sophisticated) half. Newton users were even more sophisticated, with all but two of the Newton users (5%) giving themselves the most sophisticated rating.
The most common education level for both types of users was a bachelor's
degree (43% of Newton users, 44% of Pilot). Master's and professional degrees
were also common (26% of Newton users, 30% of Pilot).
In terms of gender, the Newton respondents were 7% female and Pilot
users were 9%. This is substantially more skewed than the Internet at large,
whose users are 31.30% female, according to GVU’s WWW User Survey [5].
3.2 General PDA
usage
3.0 Results
3.1 Demographics
3.3 Gesture usage
Respondents reported using their PDAs very frequently, as shown in Table
2. Most of the respondents to our survey had been using their current PDA
for less than one year. The usage times are shown in Figure 4: Time using
current PDA. The average time was 7.5 months for Newton users and 5.4 for
Pilot users. It is interesting that so many Newton users started using
their PDA recently, possibly due to the introduction of the newest model,
the MessagePad 2000.
Another difference between Newton and Pilot users is that Pilot users
did not use gestures as often as Newton users.
Newton users reported fewer problems with gestures than Pilot users.
Several Newton users who “never” or “rarely” used the gesture for “insert
line” indicated a problem with bad recognition by the computer or inability
to remember the gesture. Similarly, the infrequent users of “insert letters/words”
cited poor recognition.
Difficulty remembering gestures was the most common reason given by
Pilot users for infrequent gesture use. Poor recognition of gestures was
also frequently reported.
A surprising result was the relationship between gesture existence
and frequency of use. One would expect that users would answer that they
“never” used gestures that do not actually exist. Most Newton users did
answer “never” for gestures that did not exist. For Newton users, frequency
of use and gesture existence were highly correlated (.94). What is surprising
is that this was not the case with Pilot users. For Pilot users, frequency
of use and gesture existence were completely uncorrelated (.02).
3.4 Opinions
about gestures
3.0 Results
3.3 Gesture usage
3.5 Handwriting
As Table 4 shows, respondents had generally positive feelings about gestures. Newton users agreed with all but two of the eight positive statements made about gestures and Pilot users disagreed with only three of the eight.
Overall, Newton users were slightly more positive about gestures than Pilot users. For all agree/disagree questions, Newton users agreed as much or more than Pilot users.
The responses for both groups of users for all opinion questions were close to normal distributions.
3.5 Handwriting
3.0 Results
3.4 Opinions about gestures
3.6 Application usage
The majority of Newton and Pilot users rated the handwriting recognition on their PDA positively. The average for both sets of users was between “good” and “excellent.” On a scale of 1 to 4, the average ratings were 3.4 and 3.1, for Newton and Pilot users, respectively. Only 7 percent of Newton users and 11 percent of Pilot users rated handwriting recognition negatively.
Graffiti was used by two Newton users and all Pilot users. On average, Graffiti was rated slightly more accurate by Pilot users, at 3.4. Interestingly, 13% of Pilot users did not rate their PDA’s handwriting recognition and Graffiti identically, even though Graffiti is the only handwriting recognition available.
3.6 Application
usage
3.0 Results
3.5 Handwriting
3.7 Paper vs. PDAs
One part of the survey asked how often a set of common PDA applications are used. As seen in Table 5, the most popular Newton applications are note taking, calendar, to-do list, and address book, which are ranked approximately the same. Pilot users ranked calendar, address book, and to-do list as the most often used. Pilot users did note taking substantially less often than other applications and less often than Newton users did.
Users of both PDAs ranked drawing and email as the least often used applications. The application rankings were normally distributed, except for note taking by Newton users, which had spikes at first place (i.e., most often used) and fourth place and very low frequencies elsewhere.
3.7 Paper vs. PDAs
3.0 Results
3.6 Application usage
3.8 PDA meeting usage
Respondents were asked about tasks for which they used paper but did not use their PDA. For users of both PDA types, the single most common response to this question was note taking, as seen in Table 6. Some respondents were specific about the type of note taking they did and some were not. The specific types ranged from short notes of the type typically put on post-it notes to longer notes of the type taken in meetings, lectures, presentations, etc. We put all of these in one category: “note taking.”
For Newton users, drawing was the other task reported as frequently done on paper but not a PDA. For Pilot users, the tasks next most often named were taking telephone messages and drawing. This question used a free-form response which respondents were not required to answer, but most did (60% of Newton users, 74% of Pilot users).
The questionnaire also asked why the task was not done on a PDA. The single most common reason given by Newton users was that the screen is too small (19% of Newton users listed this reason). The other two common reasons listed by Newton users were slow or inaccurate recognition (12%) and inadequate connectivity or compatibility with other computers and applications (10%).
Pilot users gave a wider variety of reasons. The two most popular were that it is faster to use paper (18%) and the small PDA screen (13%). Most respondents were not specific about what they meant by “faster to use paper.” Some specific reasons given by a few are: they do not write quickly with Graffiti, and paper is faster due to the time required to find the Pilot, turn it on, and select or the appropriate application.
The next two common reasons given by Pilot users were that it has poor support for drawing and it is easier to use physical paper or notes, such as post-it notes (10% for both). Some Pilot users prefer physical paper since it is easier to leave a note with a person or in a particular place.
3.8 PDA meeting
usage
3.0 Results
3.7 Paper vs. PDAs
4.0 Discussion
Both Newtons and Pilots are used “often” in meetings. On a scale of
1 (“Never”) to 4 (“Very often”), the averages for Newton and Pilot users
were 3.0 and 3.1, respectively.
Both types of users reported they were less frequently in meetings
where others used PDAs. The average frequencies were 2.0 and 2.3, respectively.
The types of notes taken by users in meetings are shown in Figure 5.
The total usage percentage is greater than 100 since respondents could
indicate more than one type of note. As seen in the figure, there is a
group of four note types that are used substantially more than other note
types. It is interesting that there is little difference between Newton
and Pilot users for all note types.
There are three conclusions we draw from the results presented in the previous section. First, gestures are valuable in current interfaces. Second, PDAs do not currently have enough gestures. And third, people use Newtons and Pilots differently. The following subsections discuss what the benefits and shortcomings of gestures are, why more gestures are needed, how the two PDAs are used differently, and what the limitations of this survey are.
4.1 Benefits
of gestures
4.0 Discussion
4.2 Shortcomings of gestures
Users of Pilot and Newtons alike were very positive about gestures.
Of the eight opinion questions asked, respondents were most critical of
gestures because of the small number available. Both sets of users agreed
that gestures are powerful, easy to learn, efficient, easy to use, convenient.
This positive view of gestures was very surprising to us, since we
thought users had more problems with gestures than they report. When one
considers how the survey data was gathered and the resulting high technical
sophistication of the respondents, this result is less surprising.
4.2 Shortcomings
of gestures
4.0 Discussion
4.1 Benefits of gestures
4.2.1 Gesture recognition
In spite of the technical sophistication of the respondents, there were two areas in which they were neutral or negative about gestures, and one area in which they were negative about PDA interfaces (see Table 4). This subsection will discuss the negative opinions about gestures and the next will discuss the PDA interface.
4.2.1 Gesture recognition
4.2 Shortcomings of gestures
4.2.2 Gesture memorability
4.3 The need for more gestures
Both Newton and Pilot users believe that gestures are not always recognized. Since PDAs were popularized, they have been criticized, fairly or unfairly, for their poor handwriting recognition. It is even more important for gestures to be correctly recognized than for characters, because gestures invoke operations.
Misrecognition of a character is easily perceived by the user. However, if a gesture is misrecognized it will cause an unintended operation to be performed, and users may have difficulty determining what happened. Furthermore, an unintended operation is likely to be more difficult to correct than an incorrectly recognized character. As a Pilot user commented, “cut/copy gestures are risky”.
4.2.2 Gesture memorability
4.2 Shortcomings of gestures
4.2.1 Gesture recognition
4.3 The need for more gestures
Users were also dissatisfied with gesture memorability. Newton users agreed that gestures are easy to remember, but Pilot users disagreed. A few users specifically commented that memorability was a problem. A Newton user wrote, “Need a pop-up list of available gestures.” Another commented, “PDA needs to have small reference sticker about gestures.”
Before conducting the survey, we hypothesized that PDA users might have difficulty with gestures because they are difficult to remember. Unlike many interaction techniques, gestures use recall rather than recognition, which implies that pen-based UI designers must make gestures easy to remember. This goal can be achieved by, for example, designing gestures that are easier to remember and using interaction techniques that help users remember gestures.
4.3 The
need for more gestures
4.0 Discussion
4.2 Shortcomings of gestures
4.4 PDA usage models
Even more than the two areas discussed in the previous section, users were dissatisfied with the number of gestures available. One Newton user wrote, “Need to be able to define new gestures,” and another wrote, “Wish there was a way to add gestures or have a few undefined gestures I could map to specific text-editing tasks.”
Gestures could be very useful on a PDA, where screen space is at a premium and the primary (and often only) input device is a stylus, yet the two most popular PDAs support very few gestures. Both devices offer few gestures that are common to several applications.
It is possible that it is too difficult for novices to learn a gesture, so designers want to minimize the number of gestures. However, in spite of difficulties novices have learning gestures, the additional method of invoking operations would still be advantageous for expert users.
Another reason for the lack of gestures is that it is difficult for the PDA to recognize gestures from a large set. Although this may have been the case for early PDAs, it is no longer an obstacle considering the processing power of modern PDAs.
Finally, it is possible that it is difficult to design good gestures, so designers have only chosen simple, obvious ones. Although gesture input is not a new idea, interface designers do not have the same experience with them as with traditional graphical user interface components. The novelty of gestures for many designers could explain, at least in part, why current pen-based UIs have so few gestures.
4.4 PDA usage models
4.0 Discussion
4.3 The need for more gestures
4.5 Survey limitations
The results on application usage suggest that Newton and Pilot PDAs are used differently. Newton owners use their PDAs as notebooks. Pilot users, on the other hand, use their PDAs as personal datebooks and addressbooks.
There are several reasons for using the two devices differently. The Newton is better suited to be a notebook. It has a significantly larger screen. Users might also prefer the Newton for note taking because it recognizes normal English printing and script, whereas the Pilot only recognizes Graffiti. In addition, the Newton’s built-in software allows the user to draw and include text with the drawings. Conversely, the Pilot’s smaller size makes it more convenient to carry everywhere, which is desirable for a datebook or addressbook.
The difference in application usage between Newtons and Pilots may also be explained, at least in part, by characteristics of the respondents. According to our survey, Newton users are more technically oriented than Pilot users. This effect is shown by the technical sophistication rating but more so by respondents’ occupations. Newton users are much more likely to have technical jobs related to computers than any other job, whereas Pilot users are quite likely to be managers or executives.
Although Newtons are used as notebooks more often than Pilots are, users take the same kinds of notes on both PDAs, at least in meetings. As shown in Figure 5, there is little difference between the kinds of notes that users take. It is interesting that the two kinds of shared notes (i.e., events to share and ideas to share) are the two least used note types. More and better collaborative software is needed.
4.5 Survey limitations
4.0 Discussion
4.4 PDA usage models
5.0 Future Work
An oddity in the Pilot gesture usage is the low correlation between usage and existence. As mentioned in the results section, a surprisingly large number of Pilot users reported using gestures that do not exist on the Pilot. Although we attempted to make it clear what we meant by “gesture,” it is possible that Pilot users misunderstood, perhaps because Graffiti is composed of single strokes that are similar to gestures.
The main limitation of this survey is that the results only have qualitative value because they are not statistically significant. Statistical significance was not achieved because of the small number of respondents, its non-representative sampling of the population of PDA users, and the technical sophistication of the respondents.
The respondents of our survey are not representative because they were self-selected. We could not locate a representative sample of PDA users and ask that they all complete our survey; we posted a request for participation on several Usenet newsgroups. As mentioned earlier, readers of these newsgroups are likely to be technically sophisticated and highly motivated about the technology. Since PDAs are still relatively new, many if not most current owners are “early adopters.” Due to the nature of the respondents, we believe they are more enthusiastic about the technology and more sympathetic to its shortcomings than most users. A broader survey would paint a less rosy picture of PDAs and gestures.
Another limitation of conducting this survey over the web is that no
data verification could be done. Even had it been done in person, some
of the demographic data may not have been conclusively verified, but with
a web-based survey, any respondent could claim to be any age, gender, or
have any profession2. We have no
reason to believe our respondents are dishonest, but lack of verification
is a potential liability.
The great promise of PDAs is the merger of two powerful and popular technologies: information technology and paper. The ideal PDA has the storage, computation, and communication benefits of computers and the versatility, convenience, and portability of paper. Many researchers have discussed or built interfaces that exhibit some paper-like benefits [7,13,17].
An important issue facing researchers today is how to make progress toward the goal of an ideal PDA. Looking at why users choose paper instead of currently available PDAs may show how PDAs can be improved. The results of our survey suggest two avenues for improvement.
One avenue is to improve PDA display size, resolution, contrast, and range of viewing angle. In time, the resolution, contrast, and viewing angle of display devices will no doubt improve. However, small size is one of the best features of PDAs. It would therefore be desirable to investigate software or user interface (UI) techniques that mitigate the drawbacks of small displays. For example, UI designers might make small, low-resolution screens less cumbersome with an interface that uses zooming [2] or focus plus context [1]. Interaction techniques such as gestures [6,7,14], marking menus [16], or pop-up pie menus can be used because they require less screen space than many traditional GUI controls do [15].
The second avenue for PDA improvement is to make it more like paper in terms of speed of use and convenience. Paper is fast and convenient to use because it does not require start-up time; it is always ready to accept writing. As a few respondents to our survey pointed out, this is not the case with PDAs. First, they must be powered on. This only takes a few seconds, but the application that comes up may not be the one the user wanted, in which case the user will have to select the correct application and wait for it. PDAs could solve the power-on problem by having a suspend state, as laptop computers do. Some users also expressed frustration with the speed of the handwriting recognition, especially since, unlike with paper, it is sometimes misrecognized and must be corrected. PUI designers do not need to focus on recognition speed since it will improve as PDA processors become more powerful.
Paper is more convenient because one can easily write not only text, but also drawings, equations, tables, etc. [12] The speed and convenience of paper, especially for informal, temporary notes, might be brought to the PDA with specialized applications or interaction techniques well suited to pen-based UIs. For example, note taking is one of the most popular applications on PDAs. As any user of a modern editor or word processor knows, there is a plethora of operations one could use when entering or editing text. Interaction techniques tailored for pen-based UIs could be used to enable easier access to more sophisticated text processing.
Gestures are promising as a technique for making pen-based UIs more
like paper. The survey shows two ways that gestures could be improved.
Users are not satisfied with gesture recognition accuracy nor with how
easy gestures are to remember. As shown by Frankish, et al and LaLomia,
recognition in pen-based UIs affects user satisfaction [3,8].
Designers of pen-based UIs should attend to the recognizability and memorability
of their gesture sets. Unistrokes [4] and Graffiti
are examples of strokes that were designed to improve recognition of entered
text. Since few interface designers are experts on both gesture recognition
and human psychology, it would be useful to have a tool to aid in the design
of recognizable and memorable gesture sets.
This paper presented the results of a survey of Pilot and Newton users. Four important findings are:
We would like to thank all of the PDA users who took the time to fill
out our questionnaire. Thanks also to members of the U.C. Berkeley community
who gave feedback on early versions of the survey.
1 Graffiti is an alphabet, each of whose characters is a single stroke [10].
2 As the famous cartoon put it, "On the Internet, nobody knows you're a dog."
1. Bartram, L., Ho, A., Dill, J., and Henigman, F. The Continuous Zoom: A Constrained Fisheye Technique for Viewing and Navigating Large Information Spaces. In Proceedings of the ACM Symposium on User Interface and Software Technology (UIST '95), p. 207-215. ACM, 1995.
2. Bederson, B. and Hollan, J. Pad++: A Zooming Graphical Interface for Exploring Alternate Interface Physics. In Proceedings of the ACM Symposium on User Interface and Software Technology (UIST '94), p. 17-26. ACM, 1994.
3. Frankish, C., Hull, R., and Morgan, P. Recognition accuracy and user acceptance of pen interfaces. In Human Factors in Computing Systems, p. 503-510. ACM, Addison-Wesley, Apr. 1995.
4. Goldberg, D. and Richardson, C. Touch-typing With a Stylus. In Human Factors in Computing Systems, p.80-87. ACM, Apr 1993.
5. GVU’s WWW User Survey. Available as http://www.gvu.gatech.edu/user_surveys/survey-1997-04/.
6. Hanne, K. and Bullinger, H. Multimodal Communication: Integrating Text and Gestures. In Multimedia Interface Design, p. 127-138. ACM Press, 1992.
7. Kurtenbach, G. and Buxton, W. Issues in Combining Marking and Direct Manipulation Techniques. In Proceedings of the ACM Symposium on User Interface and Software Technology (UIST '91), p. 137-144. ACM, Nov. 1991.
8. LaLomia, M. User Acceptance of Handwritten Recognition Accuracy. In Human Factors in Computing Systems (Conference Companion), p. 107. ACM, Apr 1994.
9. Landay, J. and Myers, B. Interactive sketching for the early stages of user interface design. In Human Factors in Computing Systems, p. 43-50. ACM, Addison-Wesley, Apr 1995.
10. Lee, Y. PDA users can express themselves with Graffiti. InfoWorld, 16(40):30, Oct 3, 1994.
11. Long, A.C., and Landay, J. PDA User Survey. Available as http://media2.cs.berkeley.edu/PUI-Project/questionnaire.htm.
12. Meyer, A. Pen Computing. SIGCHI Bulletin, 27(3):46-90, Jul 1995.
13. Moran, T., Chiu, P., van Melle, W., and Kurtenbach, G. Implicit structures for pen-based systems within a freeform interaction. In Human Factors in Computing Systems, p. 487-494. ACM, Addison-Wesley, Apr 1995.
14. Morrel-Samuels, P. Clarifying the distinction between lexical and gestural commands. International Journal of Man-Machine Studies, 32:581-590, 1990.
15. Pier, K., and Landay, J. Issues for Location-Independent Interfaces. Technical Report ISTL92-4, Xerox Palo Alto Research Center, December 1992.
16. Tapia, M., and Kurtenbach, G. Some design refinements and principles on the appearance and behavior of marking menus. In Proceedings of the ACM Symposium on User Interface and Software Technology (UIST ’95), p. 189-195. ACM, Nov 1995.
17. Wolf, C., Rhyne, J., and Ellozy, H. The paper-like
interface. In Designing and Using Human-Computer Interfaces and Knowledge
Based Systems, p 494-501. Elsevier, Sep 1989.