Keywords

1 Introduction

Emerging smartphones are concerned with maximizing the display real estate, leading to a variety of display sizes and blurring the limits between smartphones and tablets. While a trend towards larger displays allows for a better experience, it limits the grasp-ability and one-handed input on such devices [10]. As recent evolution addresses this concern by introducing foldable smartphones, with dedicated apps [16], such as the Royole FlexPai, the first commercialized android-based smartphone that can bend [18]. Once folded these devices offer two displays attached by one edge, leading to two potential uses: either interacting on one display while using the other for output only, or interacting on both displays, which allows to display two complementary applications (e.g. consulting the weather forecast and the agenda to plan a hike, or a map and a list of hotels for booking the closest to a meeting). One-handed input, a common usage scenario, needs renewed consideration on such class of devices. In this paper, we study how two attached displays can be positioned and rotated along two axes to facilitate one-handed user input. Our principle aim is to examine what dual-display configurations facilitate one-handed screen reachability, an outcome that can benefit designers of such devices as well as for the UI of apps built for such devices.

To address our aim, we investigate the impact of the relative position of two palm-size displays when they are articulate along two junction angles (see Fig. 1). We designed and built a running prototype that allowed us to experimentally evaluate thumb reachability with 32 possible dual-display permutations (corresponding to the combinations of two display orientations and angles). Through a preliminary study we examine users grasp patterns for such dual-display configurations. We then investigate the effect of the relative position of the two displays on thumb reachability in a controlled study. We measured the size of both the comfort area (i.e. easily reachable area), as well as the useful area (i.e. reachable area through a large thumb exertion). Our results lead to the most suitable articulations and identify those offering an equal or better reachability than current smartphone-like configurations.

Fig. 1.
figure 1

Left: The 16 dual-display configurations explored for a portrait orientation (we explored the same configurations for a landscape orientation). Center: touchscreens in their 3D printed cases and the 16 connection joints used to create our 32 dual-display configurations, as on the left. Right: illustration of landscape and portrait orientations for (Alpha = 30°, Beta = 60°).

Full size image

Our contributions are: (1) a preliminary study to investigate how users grasp dual-display configurations when articulated across two angles; (2) a study to investigate the effects of the relative position of the two displays on thumb reachability; and (3) a discussion on usage scenarios for such dual-display smartphones.

2 Related Work

Closely related to our work is that on mobile one-handed thumb reachability, multi-display devices and shape-changing devices, which we review briefly.

Studying the thumb reachable area with one-handed operation has led to models predicting the functional area for a given smartphone size, hand size and finger positions [2], even under varying device grips [3]. Solutions proposed in the literature, e.g. ThumbSpace [6], and in commercial products, e.g. the Samsung’s One-Hand Mode [17] reduce the effective display size to the region reachable by the thumb. But these solutions limit the display space leading to concerns of pointing accuracy [5]. Other solutions lead to indirect input (e.g. around-the-device [4], back-of-the-device [12, 13].

Multi- and dual-display prototypes recently emerged, where the device can be physically split [7, 9, 11]. Codex [9] enabled many novel mobile scenarios, including collaborative input. However, most dual-display prototypes fold along one of the device edges [7, 11] and when opened have a form factor closer to a tablet than a smartphone. Studying the manipulation of such dual-display systems has not received much attention.

The above mentioned dual-display mobile devices led to foldable shape-changing devices, such as Paddle [15], a device concept of a highly deformable mobile device, or EXHI-bit, an expandable handheld interface [14]. These devices can dynamically change the arrangement of the displays. However, these works have mostly focused on the advantages of using a shape-changing display to extend the output interaction space, and to our knowledge none of them has explored thumb reachability on such displays. Unsurprisingly, a recent review on the grand challenges of shape-changing interface research [1], underlined the need to further explore the integration between the device form and interaction, with the goal “to realize shape-changing interfaces that capture touch input […] in a quality comparable to today’s handheld computing devices” [1]. This goal drives our exploration of the dual-display thumb reachability, in particular for a dual-axis foldable device.

3 Dual-Display Smartphone Configurations

Our goal is to study the effect of dual-display configurations on thumb reachability. We refer to the display at the bottom, the one being gripped, as “display 1”, and the display at the top as “display 2”. We define a “dual-display configuration” as the relative position of displays A and B that are attached by an edge, point or joint. Given the high dimensionality of the design space for possible angle rotations with each articulated display, we explored a discrete number of static configurations (i.e. configurations where the position of the two displays cannot change). This allowed a more systematic examination of the effect of the relative angle of the two displays on thumb reachability.

3.1 Static Configurations

Three factors drive the design of the static configurations explored in this paper: (1) Orientation of the screen is portrait or landscape; (2) Alpha angle describes a vertical rotation of display 2 towards the user (see rows on Fig. 1-left); (3) Beta angle: describes a horizontal clockwise rotation of display 2 (see columns on Fig. 1-left).

For each angle type, we considered four values: 0°, 30°, 60° and 90°. We did not consider values below 0° as for Alpha it would mean tilting display 2 outside the user’s view (i.e. bent backward); and for Beta it would mean tilting display 2 away from the thumb (i.e. to the left, Fig. 1-left). The configuration where Alpha and Beta are equal to 0° represents a smartphone-like shape, i.e. one flat rectangular device. The combination of these three factors (Orientation, Alpha and Beta) led to 32 dual-display configurations (2 Orientations × 4 Alpha angles × 4 Beta angles). Figure 1-left illustrates the 16 combinations of Alpha and Beta for a portrait orientation.

3.2 Implementation

We implemented our dual-display device using two 2.8” TFT resistive touchscreens, with a resolution of 240 × 320 pixels. Each touch screen was connected to an Arduino MKR1000. We used Arduino tactile screens to take advantage of Arduino’s computing and WiFi capabilities. Once placed side-by-side, our two screens create a single screen of 144 × 58 mm (ratio = 2.5) or 116 × 77 mm (ratio = 1.5) which is close to the size of a Samsung S4, almost equivalent to ZTE AxonM and a bit smaller than the FlexPai once folded.

We then 3D designed and printed the cases for both displays and the required electronic assembly. The final 3D printed case had a size of 77 × 58 × 23 mm. For the connection joints, we 3D designed and printed 16 joints (Fig. 1-center) that could be easily interchanged.

4 Preliminary Study: Device Grasp

In a preliminary study, we investigated how users grasp our dual-display configurations. Since hand grasp affects reachability [3], we examine the set of natural grasps (used in the subsequent study), so that all participants would grasp the configurations similarly. We asked five research students to grasp the 32 dual-display configurations either holding only display 1 (Fig. 2-A), or holding the junction with the index or middle finger (Fig. 2-B, C). For each grasp, we asked them to comment whether they could reach both displays with the thumb. We took pictures of all grasps (Fig. 2).

Fig. 2.
figure 2

Different types of device grasps obtained from participants.

Full size image

Users pointed out that holding display 1 alone made touching display 2 more difficult. We observed that when grasping only display 1, it became difficult to properly hold the phone while releasing the thumb. Instead, placing one finger on the junction ensured a more secure grasp. Therefore, we chose to adopt this “junction grasp” in our subsequent study to avoid any bias due to different users employing different grasps, or the same user changing grasps during the study.

5 Study: Dual-Display Thumb Reachability

5.1 Goal, Task and Instructions

The goal of this study was to assess thumb reachability, i.e. to measure the surface reached with the thumb during one-handed interaction on our dual-display smartphone.

In order to assess the reachable surface for one-handed input on a dual screen smartphone, we emphasized three aspects: Comfort, Useful and Non-Accessible areas. The “Comfort area” of the screen requires only a small input effort (moving the joints of the thumb only). The “Useful area” requires a consequent effort (using the movement of the palm of the hand and not only the thumb articulations). The “Non-Accessible area” of the screen is inaccessible without the use of both hands.

To this end, we asked users to touch the screen with the thumb and move it inside the comfort area, and then to extend the motion to the useful area on each display. They were asked to focus on the gesture, without any time requirement.

5.2 Participants, Experimental Protocol and Collected Data

We recruited 12 participants (7 females) aged 24.3 on average (SD = 4.6), all were right-handed. Participants were recruited among students of a local university (undergraduate or PhD students). Each participant was rewarded with a 10$ Amazon gift card.

The experiment followed a 4 × 4 × 2 within-participant design with Beta angle (0°, 30°, 60° and 90°), Alpha angle (0°, 30°, 60° and 90°) and Orientation (portrait or landscape mode) as factors. We counterbalanced the trials using the Orientation and randomly ordered the Beta and Alpha angles.

The procedure first consisted in explaining to users how to hold the device, using the “junction grasp”, as detailed in the preliminary study. Then, for each configuration, the user explored the comfort area, followed by the useful area. The experimenter manually changed the configurations by repositioning the joints. We used the dual-display device described in Sect. 3. The device was USB-connected to the computer running the experiment. We ensured that the wires did not interfere with the manipulations of the prototype, and attached them so as not to add additional weight to it.

We recorded all the touch points of the finger on both screens. From this point cloud, we computed the reachable areas using a surface-reconstruction approach. We also asked participants to rate the comfort of each configuration on a 7-points Likert scale.

5.3 Results

We use estimation techniques with 95% BCa bootstrap confidence intervals (CI) and ratio analyses as recommended by the APA [20]. Ratio is an intra-subject measurement that expresses the effect size (pair-wise comparison) and is computed between each of the geometric means.

Surface of Comfort and Useful Areas.

We report the main results for the sum of the surfaces on both screens. Indeed the average surfaces measured on display 1 and on display 2 are very similar when considering the Comfort area (display 1: 755 mm2, CI[590; 985]; display 2: 708 mm2, CI[560; 881]) and the Useful area (display 1: 1306 mm2, CI[1072; 1502]; display 2: 1246 mm2, CI[1032; 1414]).

Interestingly, the results establish that the orientation of the screens (portrait or landscape) does not affect the size of the comfort area (landscape: 1491 mm2, CI[1212; 1850]; portrait: 1437 mm2, CI[1140; 1826]) or the useful area (landscape: 2550 mm2, CI[2177; 2882]; portrait: 2554 mm2, CI[2175; 2941]). This remains true for any given configuration. Hence, we will not differentiate the results for both orientations in the remaining analysis.

Overall, the size of the Useful area is 87% larger than the Comfort area (1464 mm2, CI[1198; 1827] vs. 2552 mm2, CI[2220; 2886]). The available display surface of one screen being 2596 mm2, 49% of the dual-screen display is usable. This is higher than the results obtained in prior work involving a single screen: over different smartphone sizes, an average of 36.4% of the screen was reachable with the thumb without losing grip stability (see Table 3 in [13]).

We considered the Alpha angle, representing the relative vertical position of each screen. On average, (Fig. 3-left) configurations involving the two highest Alpha angles (60°, 90°) allow a larger useful area (2651 mm2, CI[2305; 2994]) than the two smallest Alpha angles (2453 mm2, CI[2127; 2807]). The intra-subject ratio analysis (see Fig. 3-right) clearly confirms this trend (ratio > 1, CI not intersecting (1): in comparison to configurations based on Alpha = 0°, the average useful area is 9% larger with Alpha = 90°, and 6% larger when Alpha = 60°. This trend also holds for the comfort area alone.

Fig. 3.
figure 3

Average finger reach surface in mm2 (top) according to the Alpha angle (left), and surface ratio (with 95% CIs) of the Alpha angle (ratio > 1 means bigger finger reach than with Alpha = 0°).

Full size image

We also considered the effect of the Beta angle, depicting the horizontal relative position between the two screens. The average size of the useful area seems to diminish when Beta increases (see Fig. 4-left): in particular, this area is clearly larger with Beta = 0° (2774 mm2, CI[2432; 3149]) than with Beta = 90° (2341 mm2, CI[2015; 2656]). The intra-subject analysis based on the ratio strongly confirms this trend (see Fig. 4-right): the useful surface is 15% to 19% larger with Beta = 0° than with Beta = 60° or 90°, while there is no visible difference with Beta = 30°.

Fig. 4.
figure 4

Average finger reach surface in mm2 (top) according to Beta angle (top), and surface ratio (with 95% CIs) of the Beta angle (ratio > 1 means smaller finger reach than with Beta = 0°).

Full size image

The four better dual-display configurations are those combining a large Alpha angle (60°, 90°) and a small Beta angle (0°, 30°). When comparing each of the 15 possible configurations to the baseline (Alpha = Beta = 0°), the intra-subject ratio confirms this outcome as the ratio range of these four configurations are among the 5 smaller ratios obtained, i.e. leading to an equivalent or larger useful area (Fig. 5-left). The resulting 4 best configurations are illustrated in portrait orientation in Fig. 5-right.

Fig. 5.
figure 5

Left: ratio of the finger reach surface (with 95% CIs) for each configuration Cij where i = Alpha angle/30 and j = Beta angle/30 (ratio > 1 means smaller finger reach surface than with Alpha = 0° + Beta = 0°). Right: illustrating the portrait orientation of the 4 best configurations.

Full size image

User Preference.

Similarly to thumb reachability, the scores given by the participants to the different configurations was not affected by the orientation of the screens (landscape: 4.4, CI[4.1; 4.9]; portrait: 4.4, CI[4.0; 4.8]). Based on the intra-subject ratio analysis, results establish that configurations based on Alpha = 0° obtained an average score 12% (CI [4; 19]) better than for Alpha = 30°, and 16% (CI[4; 28]) better than for Alpha = 90°. More interestingly, the scores obtained with regards to the Beta angle are in line with the Useful area: it is clearly maximized when Beta is minimized (Beta = 0°: 5.4, CI[5.0; 5.9]; Beta = 90°:3.6, CI[3.3; 4.1]).

Summary.

This experiment revealed two surprising results: (1) the orientation of the displays (portrait or landscape) does not affect the thumb-reachable zone; and (2) the reachable zone on each display is very similar: 50% of each display can be reached during a one-handed interaction. Concerning the junction angle, we observed that the thumb-reachable surface is maximized for 4 particular configurations that combine a large vertical rotation (Alpha = 60° or 90°) and a small horizontal rotation (Beta = 0° or 30°). These configurations offer the same, or slightly higher (5%), thumb reachability than a smartphone-like configuration (Alpha = Beta = 0°). The overall user rating on a 7 point Likert scale is 4.4 (4.9 for the 4 configurations highlighted above). Interestingly, user preference is correlated with all these results but one: users prefer the configuration without vertical rotation (Alpha = 0°).

6 Discussion

6.1 Potential Usages of Articulated Dual-Display Devices

Our results indicate that screen orientation (portrait or landscape) does not impact thumb reachability or users’ preference. Therefore, the orientation can be associated to predefined modes (e.g. apps requiring text input, games, etc.). As four configurations were also identified as particularly beneficial, up to 8 modes might thus be predefined through specific articulations of the dual-display device. We also demonstrated that thumb-reachability is similar on both screens. Hence, both screens can be used equally and applications can be displayed on one or the other screen without constraints.

To identify potential usage situations for such articulated dual-display devices, we organized two brainstorming session with 10 HCI Master and PhD students. In several scenarios, participants considered display 2 as a dynamically adjustable peephole on the content or tasks displayed on Display 1: for example, in the case of map navigation, placing display 2 perpendicular to display 1 (Beta = 90°) would display the north-east of the map displayed on display 1. Alternatively, articulated dual-display devices might be used to extend the current display surface around display 1: the position of display 2 according to display 1 would correspond to a specific application. As opposed to existing around-device solutions [8] this would provide visual feedback and could be organized along the two angles considered in this experiment. Finally, participants of the brainstorming also highlighted the fact that the device might be used in different configurations according to the user task: totally folded on itself, the device is small, can be placed in a pocket and is sufficient to read a SMS. If unfolded, it can be used to read an email or browse the web.

6.2 Limitations and Future Work

First, smartphones come in multiple sizes, so future work should validate our results with different sizes of displays. Second, we will conduct a number of technical improvements to the current prototype. Tactile screens with better resolution then the ones we used are required to allow a real use of existing mobile applications. Inserting an IMU to detect in real-time the relative orientation of the two screens could also improve the encoding of the articulation. Finally, the thickness of the device, the size of the bezels, the shape of the chassis and how it sits in the hand need to be improved to correspond to more realistic settings. These improvements will allow us to further study the use of articulated dual-display devices with real applications. Another important issue is the definition of the content layout of non-horizontal display screens, in which little work has been done so far [19]. In addition, the current prototype is based on rigid, predefined joints: instead, a kneecap-like joint or an actuated joint might be used to dynamically adjust the configuration. Finally, holding the devices was feasible but we will explore other solutions to improve grasping, placing a ring on the rear of the case.

7 Conclusion

In this paper, we explored the concept of an articulated dual-display smartphone to overcome reachability issues on large mobile screens. We designed 32 different configurations of an articulated dual-display smartphone and studied them through the analysis of the thumb-reachable surface. Participants’ ratings were also positive towards such devices. Despite the limitations of the current prototype, results also establish that the thumb-reachable surface is close to 50%, which is higher than those previously found on large mobile displays. Articulated dual-display devices are therefore a promising solution to lessen the discomfort and reachability issues of the recent trend in increasing mobile display sizes, through foldable touchscreens.