Keywords

1 Introduction

The stylus is used across many industries to enhance the productivity and accuracy of interactions. Industrial sectors that have used tablets include retail sales (e.g., assisting customers with product selection), real estate, inventory control, utilities, appliance and machinery repair, design, art, and education. The stylus is also used in the medical industry, and studies have shown that when doctors and nurses used a tablet for simple tasks, the use of a stylus improved productivity and reduced error when compared to use of a finger [10].

A digital stylus can be used with a tablet computer to improve the experience of human computer interaction and precision of work. The stylus can be used with the tablet in many other settings where mobility is required for note taking, drawing, web browsing, social networking or business applications.

The rapid innovation around input devices has led to a wide variety of styli available for purchase on the internet. The range of styli available vary from Passive to Active stylus. The Passive stylus emulates the finger touch interaction and does not include any electronics in it to provide added functionality. In contrast, the Active stylus has electronics that facilitate communication between the stylus and the device with which it is used. It provides added functionality such as hover capability, scrolling, and palm rejection.

While there is a variety of styli available catering to specific functionalities, and others offering a broader scope of functionalities, there is a lack of research that evaluates user experience holistically when using the stylus as an input modality. Prior literature focuses on writing or drawing with a pen or pencil on paper and touches on issues of psychology [11], motor control [12], biomechanics, preference, legibility, speed, and other factors. The published research begins in the early 1900’s and has progressed with technological innovations in research methods. Various studies in the past have provided insights into portions of stylus use and their impact on writing or ergonomics [1, 4, 7, 9, 13]. However, there is no research available that assesses the complete design implementation of a stylus for holistic user experience.

This paper aims to close that gap by presenting a comprehensive stylus evaluation methodology. The methodology is a test suite to characterize the experiences enabled by different stylus implementations. It includes a series of tests for factors that influence a good user experience when using a stylus. The test suite is based on research from various resources including: academic engagements, literature review, user studies, industry recommendations, and online product review sites, from which specific factors important for a good stylus experience were gathered. Factors were identified based on biomechanics, ergonomics, user preference, human comfort, usability/ease of use, motor control, and performance.

The testing suite is split into two main categories: Basic Testing and Advanced Testing. Within each category, there are subsections describing applicable factors and tests for each factor, accompanied by recommendations.

2 Basic Evaluation

The basic testing consists of factors and tests that can be performed manually by a trained evaluator. The basic testing includes four portions, and covers Physical Design Checklist, Palm Rejection, Efficiency & Convenience, and Productivity.

2.1 Physical Design

When considering the design of stylus implementation, the physical affordances are fundamental for using the device as an effective writing/drawing instrument. Pens of different diameters, different cross-sectional shapes, and different weights were tested with college students in Hong Kong as they performed a maze tracing task and a writing task [9]. Another study with employees at Wang tested pens of different diameters and lengths while they drew and wrote on a tablet while watching a monitor [7]. A third study with university students conducted in Taiwan evaluated 12 pens of different lengths and diameter by performing selecting (Fitts’ tapping), writing, and drawing tasks on a tablet computer [22, 23]. Online consumer and tech product reviews also indicated the need for a substantial weighted instrument that felt comfortable when writing [19, 20]. Users also liked the option to choose from a variety of colors. Having a variety of colors to choose from is important because a writing instrument is viewed as a personal item [1, 21].

Based on user preference, performance (low error rate), usability, and ergonomics the following set of recommendations for physical design of a stylus were generated.

  • Weight: Less than 27 grams [9] but No lighter than 11 grams [19, 20].

  • Length: 110–160 mm [7, 22, 23].

  • Diameter at grip: 7.5 mm–11 mm [7, 17, 22, 23].

  • Cross-sectional shape: Circular [9].

  • Grip material: Ribbed grip, matte texture [7].

The tests for the above can be performed by taking physical measurements and making visual observations of the factors mentioned above.

Another aspect related to physical design is palm rejection addressed next since it might impact factors like bezel design which is not addressed in this paper.

2.2 Palm Rejection

Having a robust palm rejection capability in the design implementation of a stylus is another critical consideration when providing a pen or pencil-on-paper-like experience on a digital surface. Being able to comfortably rest the palm on the digital surface while writing is important from both, an ergonomic perspective as well as accuracy. If digital surfaces are to be used on a regular basis, having a robust palm rejection is a must [16]. A stylus would be an ineffective input mechanism if users need to hover their hand over the device surface while using a stylus to avoid unintended actions due to their palm resting on the surface of the device. Since many modern tablets make use of multi-touch recognition, some stylus and app manufacturers have incorporated palm rejection technologies into their products. This works to turn off the multi-touch feature, allowing the palm to rest on the tablet while still recognizing the stylus. A robust palm rejection implementation will enable the users to rest their palm on the touchscreen and use it just as comfortably as using a pen on paper [20].

In order to understand how well the implementation holds up against common writing usages, the researchers looked at various current devices in the market and identified the most common areas of palm rejection errors. When looking at typical right handed stylus usage with a tablet, the researchers found the most errors along the bottom and right edge of the device as the user rested their hand on the screen. Palm rejection errors for left handed users is an area for future exploration. To test for palm rejection with left handed usages, scenarios were developed for the following five specific hand placements as shown in Fig. 1a–e. For each of the hand placement positions, the word ‘Party’ was written and a circle was drawn around the word ‘Party’. The word ‘Party’ was chosen as it included characters that had both descending and ascending parts and a variety of straight and curved parts available in the word [2]. Failures were noted in each run with unintended inking where the hand rested on the screen. Figure 1 above demonstrates the layout of each hand placement position.

Fig. 1.
figure 1

Palm Rejection Hand Placement: a. Bottom left of the screen, b. Middle of the screen, c. Bottom center edge of the screen, d. Bottom right corner of the screen, e. Middle right edge of the screen.

Full size image

In addition to being able to provide a robust palm rejection, there are several other features that can enhance the digital writing and drawing experience.

2.3 Efficiency and Convenience

Additional features, when incorporated into the implementation, provide added convenience and aid efficiency. Many factors were identified as features that users expressed as desirable attributes that would enable the stylus to be used beyond sketching, writing, and selection tasks [19, 20]. The additional features make the stylus multi-functional, providing the user with flexibility, convenience, and increased efficiency during task completion. These features include: (1) An eraser functionality to mimic the traditional pencil on paper experience and leverage existing mental models to quickly erase objects [21], (2) Configurable control buttons so that users can customize the control buttons to perform specific commands [16], (3) A hover capability to provide the same convenience as found in mice to aid in navigation [21], (4) An undo function to be able to quickly revert to previous actions and cut/copy/paste functions to help quick user interactions [8], (5) Auditory feedback to provide confirmation of user actions, (6) Durability of the stylus by providing a protective cover or cap or a way for the stylus to slide into the device shell to protect the stylus tip from any damage [20], (7) Portability of the stylus through a pocket clip, magnetic attachment to the device, or another attachment form as an easy way to stow away the device when on the go [20], (8) Choice of colors that allows for personalization [1], and (9) Longer battery life and easy ways of recharging for additional convenience. The latter list of factors was tested through visual inspection and observation of the feature sets offered.

As noted in previous literature [16, 19, 20] improved efficiency and convenience can lead to a user’s increase in productivity. The next section investigates the aspects of a stylus that enhance productivity.

2.4 Productivity

If the stylus was to provide more of a pen-on-paper-like experience, then it needs to support all functions of using the pen on paper such as writing, reviewing, editing, highlighting documents, signing documents, and freehand drawing [8, 14]. Productivity testing included a series of tests based on task analysis to examine the productivity usage of the stylus. It covers writing accuracy, pointing and selecting functionality and collaboration tasks.

When using a stylus, the perception of accuracy during a writing task is impacted by how the inking is rendered on the screen. The perception of stylus accuracy is highly affected by the appearance of broken inking, incomplete or disjointed inking, and/or rugged inking [1]. To uncover the accuracy in writing, two tests were developed. The first test involved writing the following phrase: ‘The quick brown fox jumps over the lazy fence’ in print using the stylus, as that sentence covers all 26 letters of the alphabet. The second test involved writing a mathematical formula involving symbols with straight and curved strokes. In both cases, failure points were noted in multiple runs of the tests for broken/incomplete inking, non-smooth inking, missing characters or strokes.

In order to understand stylus accuracy when using the device as a pointer and selecting instrument, four user representative tasks involving commonly used Microsoft Office applications were developed. The first one involved using PowerPoint in slide sorter mode to move, delete and add slides. The second task involved moving an image from Word into PowerPoint in multitasking mode. The third task involved watching a YouTube video tutorial on Pivot tables and following along the instructions in multitasking mode while using touch and stylus simultaneously. The final task involved selecting small targets in the system tray of the application (applicable to Windows device only).

To assess how well a stylus could support the collaboration usages, six user tasks involving Microsoft Office Applications and Acrobat PDF were developed. For the first task, the stylus was used to highlight text in PDF, save it and share through available mailing software. The second task involved being able to successfully spell check a Word document. The third task involved editing text of a paragraph in Word. The fourth task involved being able to edit a footnote of a Word document. The fifth task involved being able to highlight and notate a PowerPoint slide in the slideshow mode. The fifth task consisted of playing a video embedded in PowerPoint in the slideshow mode.

The above mentioned tasks were chosen based on task analysis, and are reflective of common productivity tasks across writing, pointing and selecting, and collaboration usages using Microsoft Office applications. Each task was run manually multiple times and was given a score of pass or fail. The usages and tests described in the basic testing section were chosen because they are the most common type of user tasks. The above set of tests were developed to be performed manually by a trained evaluator, and carried out in a manner that is reflective of the user representative task in a reliable manner.

In order to eliminate biases and produce consistent repeatable results, a series of advanced evaluations of the stylus using a robotic arm, Oculus were developed. The advanced testing set up assessed the pen to ink delay, inking accuracy and pressure sensitivity.

3 Advanced Evaluation

The advanced section of the test suite is aimed to evaluate creativity usages of the stylus. A stylus has to enable artistic drawing when using non-touch optimized applications such as graphic design software or a photo editing application [20]. With pressure sensitivity, users also expect their strokes to be rendered smoothly [1]. The tests mentioned in this section were executed by a robotic arm called Oculus, and they cover stroke accuracy, stroke smoothness, and pressure sensitivity.

Oculus is a propriety evaluation setup that consists of a 6-axis robotic arm and a high speed camera. The robot can be fitted with multiple end effectors, including one that can grasp the various stylus pens as shown in Figs. 2 and 3 below. The robot is used to execute specific and repeatable actions in a highly controlled manner, including the specification of interaction speed and angle on the device under test, and the interactions are recorded by a high speed camera for later review and extraction of parameters of interest. The entire setup is non-intrusive and device/OS agnostic. In the stylus evaluation described herein, some of the robotic tests also include the use of an angled mirror to determine when initial contact is made by the robot held stylus on the device under test.

Fig. 2.
figure 2

The robotic arm drawing straight lines with a stylus on the device under test

Full size image
Fig. 3.
figure 3

The robotic arm drawing a spiral with a stylus on the device under test via the high speed camera.

Full size image

When using a stylus, users expects smooth & continuous strokes with no disjointed corners or curves [3]. Also, getting as close to a one-to-one correspondence between the pen and the cursor positions is highly preferred for tablet type devices in order to maintain a perception of accuracy [6].

To check for accuracy, four tests were performed. The first three were done using Oculus as shown in Fig. 4a–c. Oculus was used to draw straight horizontal and diagonal lines as shown in Fig. 4a, spiral lines as shown in Fig. 4b, and a six point star as shown in Fig. 4c. The fourth test was executed manually by a trained evaluator and involved drawing a zigzag line across the screen. The resulting inking was checked for stroke consistency, continuity, and smoothness across all the lines and shapes [2]. In case of the six point star, the angles were evaluated via the ability to produce sharp angles [2]. The six point star has arms of differing lengths (3 mm, 6 mm, 9 mm, 12 mm & 15 mm) to check for accuracy in being able to produce accurate inking of different lengths [1, 2]. The straight lines were checked to ensure they were equidistant to ensure a good mapping.

Fig. 4.
figure 4

Images of objects rendered by Oculus: a. straight, diagonal and spiral lines, b. 6-point star, c. zigzag line.

Full size image

In order to understand the pressure sensitivity enabled by the implementation, two tests were performed. The first was conducted using Oculus and involved using three specific exerted forces to draw three different lines. The forces used were determined by the range typically used for pencil on paper [18]. The second test involved drawing three different lines manually at three different angles (close to 0 degrees, close to 45 degrees, close to 90 degrees). In both tests, the resulting inking was checked for variation in the stroke thickness across the three lines [1, 20, 21].

In addition to checking for accuracy, the Oculus set up was used to calculate latencies. Two types of latencies were calculated: initial latency and average trailing latency. Initial latency is the pen-to-ink delay at the first touch. This latency metric is calculated from when the stylus first touches the screen and the delay in the appearance of the ink on the screen. The trailing latency calculation accounts for the subsequent pen to ink delay of the stylus as it continues to draw the straight lines. It is an average of latencies at the start of the stroke, midpoint and near the end point.

Both initial latency and trailing latency were extracted from the high speed footage by looking through the footage manually in frame-by-frame video editing software. Initial latency was determined by identifying the frame that the stylus makes contact with the device under test with the help of an angled mirror during testing, as well as the frame that ink begins appearing on the device’s screen as shown in Fig. 5. These two frame numbers, in addition to the camera frame rate, are used to determine initial latency.

Fig. 5.
figure 5

Frame counting calculation for initial latency

Full size image

For determining Trailing Latency, a similar process was used. Instead of a contact frame, an initial frame was used, which can be any frame where the robot is in the process of drawing and ink is already showing up on the device. At this point, both the location of where the robot-held stylus tip is on the device under test’s screen and the frame number were marked. After the initial markings, the footage was manually stepped through frame by frame until the ink that was being drawn by the robot crossed the threshold passing where the initial mark was placed on the initial frame. See Fig. 6. This frame number was recorded and labeled as the catch frame.

Fig. 6.
figure 6

Frame counting calculation for average trailing latency

Full size image

Calculating latency is critical to understanding how well the implementation meets user expectation. Latencies impact the perception of the experience of the device. Delay in inking latencies can cause users to write slower so that they could see what they wrote rather than guess what they wrote [2]. Having lower latencies allows for more efficiency. It has been shown that low latency judgments are made visually. Users use the relative movement between the stylus and ink to be able to make an accurate judgment [1]. Studies have shown that the users can perceive latencies as low approximately two to seven milliseconds for non-inking tasks, and as low as approximately 50 ms for inking tasks [2]. Since latency thresholds have been published, newer technologies have been showing more promise to deliver even lower latencies than 50 ms. While these thresholds can be used to determine how close the implementation is to user expectations, more work is needed to understand how lower latencies impact performance and accuracy of non-inking tasks.

4 Conclusion

With touchscreens becoming more prevalent in phones, tablets, laptops and 2-in-1 s, the stylus becomes an integral part of the computing interaction experience. Styli support a multitude of capabilities in addition to the traditional pointing & selecting functionality. An active stylus provides additional capabilities such as programmable buttons, pressure sensitivity, reduced latency, and eraser functions. However, these new capabilities and technical advancements produce complexity of integration and optimization to the interaction with the device. The use of pen and paper has been around for centuries and, therefore, sets a strong level of end-users expectations when using a stylus. In order to meet these expectations, careful design and engineering considerations must be met.

The guidelines presented in this paper are important factors for stylus design, and represent a detailed evaluation guide with user representative tasks as well as an advanced setup to holistically assess a stylus design implementation. The included methodologies and background data can enable better stylus designs that provides a more robust, flexible, efficient, and enriching user experience.