Keywords

1 Introduction

The quality of applications that are running on mobile devices is one of the essential predictors of their acceptance by end users and thus their success. More specifically, some particular features of mobile technology (such as connectivity, convenient user interface, touch screen, context awareness, supported devices, etc.) affect users’ satisfaction and loyalty and should therefore take part in the process of evaluating the quality of mobile applications [36]. Compared to personal computers, several additional characteristics of mobile devices have to be considered in that context, including mobility, screen size, virtual keyboard, and social interconnectivity [1]. According to Huang [15], both hardware and software related challenges need to be tackled when designing mobile devices. While challenges related to hardware encompass limited input facilities (like keyboard, touch screen, and scroll wheel), limited output facilities (such as small screen size, audio output, etc.) and design for mobility, software related challenges mainly cover navigation, search engine, images and icons. As one of the outcomes of their study, Flora et al. [9] found that more than 75 % of respondents strongly believe that positive user experience (e.g. feeling comfortable in interaction with gadget and feeling smart while performing assignments without tutorials or other types of assistance) importantly contributes to the perceived mobile quality.

Kang et al. [20] distinguish two types of mobile applications: (1) web-based applications composed of pages optimized for mobile devices, and (2) native applications originally developed for different mobile devices like smartphones and tablets. An important effect on quality of native and web-based mobile applications has user experience that can be considered within following two categories: the context (hardware affordances, platform capabilities and user interface conventions, and an environment in which an application is used), and the implementation (performance, design, and integration with platform features) [3]. The use of social web applications on mobile devices offer new opportunities for interaction and collaboration among users that are continuously connected [10]. Web 2.0 applications (such as wikis, blogs, microblogs, social bookmarking sites, social networking sites, mashups, podcasting applications, e-portfolios, virtual worlds, online office suites, and knowledge management applications) support various breeds of interaction among users as well as creation, sharing, organization and integration of different artefacts [30]. In the ecosystem of social web applications, the users are no longer the consumers of content but contribute both actively (by creating content in their diverse forms) and passively (by using services that are available to the community of users) to the development of Web 2.0 applications [37]. In addition, the authors pointed out that the ability to use Web 2.0 applications on multiple platforms (PC, smartphone, PDA, etc.) represents an additional challenge to the design of a user experience and is therefore important to identify user needs in a timely manner. Li and Choi [21] stated that due to particularities of mobile devices (such as smaller screen and limited input methods) users have a number of issues (e.g. they can see only small part of the document) with the employment of applications meant for collaborative writing (e.g. Google Docs and Zoho Docs, etc.) when used on them. In that respect, the authors concluded that the use of social web applications on mobile devices is limited to chat, making group discussions, brain storming or informal collaborative writing. Drawing on a feedback received from students, Cochrane and Bateman [4] concluded that the choice of a smartphone type represents a key factor in the acceptance and use of social web applications on mobile devices as well as their integration in educational context with the collaboration as the main objective.

Considering all the aforementioned, quality attributes proposed in recent standard on software quality assessment (e.g. ISO/IEC 25010 [18]) cannot be used as guidelines for the design and development of high quality mobile applications [38]. Apart from user experience which reflects hedonic dimensions of quality, usability is focused on pragmatic facets of quality [2]. It is therefore important to introduce novel quality models that will be consisted of dimensions related to both usability and user experience. The main objective of this paper is to identify factors that significantly affect mobile quality in the context of social web applications.

The remainder of the paper is organized as follows. Next section offers a brief overview of prior studies focused on the assessment of particular dimensions of mobile quality. Proposed framework for measuring mobile quality of social web applications is introduced in the third section. Employed research methodology and study findings are presented in the fourth section. The key contributions, limitations of the study and future research directions are discussed in the last section.

2 Related Work

Baharuddin et al. [2] emphasize that for the purpose of designing and evaluating mobile applications, one should take into account specific properties and limitations of following four contextual factors: user, environment, technology, and task/activity. By understanding mentioned factors, it is possible to determine which usability dimensions (effectiveness, efficiency, satisfaction, usefulness, aesthetic, learnability, simplicity, intuitiveness, understandability, and attractiveness) should be considered when designing mobile applications. The same authors also stated that the use of web applications on mobile devices which have not been initially designed for them makes users weary and eventually affects the extent to which mobile usability of those applications is perceived by users.

According to Zamfiroiu [39], there are two groups of factors that contribute to the quality of mobile application. Battery life, RAM memory of a mobile device, CPU processing power, users’ expectations, the distribution way of applications, level of the developer’s involvement, and memory on hard disk of the mobile device are external factors because they have indirect impact on quality of mobile applications. The group of internal factors is related to specificities of mobile applications including the source code, used technologies, the information volume provided, commands rapidity, and testing level. As relevant quality characteristics of a mobile application Rabi’u et al. [35] pointed out battery life, navigation, robustness of operating system, and screen size. They also suggested that the usability of mobile applications should be measured by means of following attributes: understandability, learnability, operability, and attractiveness.

Based on the analysis of prior empirical studies, Coursaris and Kim [6] found that usability of mobile applications is most commonly measured with following attributes: efficiency, errors, ease of use, effectiveness, satisfaction, and learnability. As a follow up, Harrison et al. [12] proposed PACMAD model meant for evaluating usability of mobile application that includes following 7 attributes: effectiveness, efficiency, satisfaction, learnability, memorability, errors, and cognitive load. They also pointed out that the first three of mentioned attributes must be evaluated while remaining four can be hard for measuring.

By adapting scales from the Information System Success Model [8], Özata [34] discovered that system quality, information quality and perceived enjoyment are important determinants of users’ satisfaction in the context of using Facebook as a mobile application for smartphones and tablets. However, it appeared that construct flow (the extent to which an individual becomes deeply absorbed in an activity) do not have direct significant influence on users’ satisfaction. Results of the study carried out by Hussain and Kutar [16] uncovered that features of mobile devices have strong effect on users’ satisfaction related to mobile applications. More specifically, the users expressed higher level of satisfaction when they used iPhone than when they employed O2 Orbit device. The rationale behind this results is that users experienced a greater number of issues when they used OS2 Orbit (e.g. font size, navigation, and interface design) than when they had an interaction with iPhone (e.g. hypersensitivity of virtual keyboard). Gikas and Grant [10] have investigated the use of social media and Web 2.0 applications on smartphones for learning purposes in higher education institutions and found that students had difficulties with data entry due to too small virtual keyboards. By combining adapted scales from TAM model [7] and an expectancy - confirmation theory [23], Ohk et al. [22] confirmed that perceived usefulness, interactivity, and perceived ease of use have positive impact on consumer satisfaction while in turn consumer satisfaction positively affects consumers’ intentions to continuously use mobile applications. Results of the research conducted by Hsiao et al. [14] revealed that users’ satisfaction, tight connection with others, and hedonic motivation significantly contribute to continuance intention of social applications as well as that users’ satisfaction and habit have full mediation effects between perceived usefulness and intention to continue use social applications. Based on the analysis of data collected from 330 KakaoTalk and 311 Facebook users, Ha et al. [11] discovered that users’ attitude towards mobile social networking sites is influenced by hedonic, integrative, and mobile convenience gratifications while cognitive, hedonic, integrative, and social interactive gratifications are affected by mobile convenience.

Jung and Yim [19] have found indirect effect of user interface design on the perceived usability of smartphone applications by means of following mediating variables: learnability of applications and interactions between users and smartphone applications. In addition, they discovered that user interface design has direct effect on learnability of applications and interactions between users and smartphone applications which in turn have direct impact on perceived usability of smartphone applications. Taking the aforementioned into account the authors concluded that mobile applications will be easy to use and learnable only if their user interface is simple and consistent.

Ickin et al. [17] conducted qualitative and quantitative study on the sample of three types of smartphones (Motorola, HTC, and Samsung) that all had preinstalled Android operating system. The study participants employed mentioned mobile devices in order to use 13 different categories of mobile applications including social network applications (OkCupid, Cooliris, Foursquare, Facebook, Twitter, Tumblr, and Touiteur). Analysis of collected data has shown that application’s interface design contributes to the quality of experience because the users reported number of issues related to this construct including the position and location of the keys on the smartphone screen, difficulty with resizing, web-page scrolling, inefficient manual input (e.g., “fat finger” problem). According to the same authors, the second relevant construct is application performance because users commonly used expressions such as “freeze”, “sloppy”, “sluggish”, “speed”, “performance”, “usage of memory”, and “sdcard” when they wanted to emphasize that particular application has not met requirements of this construct to the acceptable extent. They also found that users who previously used these applications on their personal computers reported lower level of quality of experience because they had much higher expectations. Apart from application’s interface design and application performance, the quality of experience is also affected by battery, phone features, apps and data connectivity cost, user’s routine, and user’s lifestyle [17].

Hoehle and Venkatesh [13] uncovered that usability of mobile applications accounts for 47 % of variance in continued intention to use social media on mobile devices. According to the outcomes of their study, relevant predictors of continued intention to use mobile social media applications are following six usability attributes: application design (branding, data preservation, instant start, orientation), application utility (collaboration, content relevance, search), user interface graphics (aesthetic graphics, realism, subtle animation), user interface input (control obviousness, de-emphasis of user settings, effort minimization, fingertip-size controls), user interface output (concise language, standardized user interface element, user-centric terminology), and user interface structure (logical path, top-to-bottom structure). It should be noted that application design, application utility, and user interface graphics have the strongest impact on loyalty related to the use of mobile social media applications.

The aforementioned findings indicate that current studies on the assessment of mobile quality are mainly focused on exploring its usability dimensions whereas measuring hedonic facets of mobile quality is rather rare. In addition, there is a lack of studies which consider both pragmatic and hedonic attributes when evaluating mobile quality. All the set forth motivates us to introduce a framework that would enable assessment of all relevant attributes related to usability and user experience of social web applications and in the same time take into account particularities of their use on mobile devices. Details on the proposed framework are provided in the following section.

3 Research Framework

As a follow up to the initial set of quality attributes proposed by Orehovački [27], evaluation methodology introduced in [24], their validation on the representative sample of various Web 2.0 applications when they have been used in their native environment [26, 31–33] as well as games [28], and their refinements [25, 29], a novel evaluation framework adapted to the context of using social web application on mobile devices was proposed. The adapted version of the conceptual model is comprised of five quality categories which are further decomposed into 37 quality attributes.

System quality refers to attributes that measure the extent to which social web application: provides various navigation mechanisms (navigability); has uniform interface structure, design, and terminology (consistency); is similar to previously used applications (familiarity); can be customized to meet users’ needs (customizability); has implemented mechanisms that protect created artefacts from unauthorized use (security); operates properly with different types of devices and among different environments (compatibility); can exchange files with other applications and use files that were exchanged (interoperability); can adapt to the environment in which it is used (responsiveness); is usable on mobile devices (mobile-friendliness).

Service quality relates to attributes aimed for evaluating the degree to which social web application: provides various forms of help to users (helpfulness); is available every time users need it (availability); facilitates management of created artefacts (artefacts management); contains mechanisms that prevent errors to emerge (error prevention); is dependable, stable, and bug-free (reliability); can quickly recover from errors and operational interruptions (recoverability); notifies users with appropriate and useful messages (feedback); supports teamwork and enables different types of communication among users (interactiveness).

Performance refers to attributes that measure the extent to which the use of social web application: enables users to execute tasks accurately and completely (effectiveness); enables users to quickly perform tasks (efficiency); responds promptly to users’ actions (response time); is capable to operate under an increased or expanding workload (scalability); is usable within and beyond initially intended contexts of use (context coverage).

Effort relates to attributes dealing with the evaluation of the degree to which: the interaction with social web application consumes small amount of physical and mental energy (minimal workload); social web application is usable to people with the widest range of characteristics and capabilities (accessibility); users have full freedom in executing tasks by means of the social web application (controllability); is simple to operate the social web application (ease of use); is easy to become proficient in interacting with the social web application (learnability); is simple to memorize how the social web is used (memorability); the interaction with social web application is unambiguous (understandability).

User experience refers to attributes meant for measuring the extent to which: the social web application has visually appealing user interface (aesthetics); the social web application is beneficial in the context of tasks execution (usefulness); the interaction with the social web application holds the users’ attention and stimulates their imagination (playfulness); users have positive perception about the use of social web application (attitude towards use); the social web application has met users’ expectations (satisfaction); the social web application arouses users’ emotional responses (pleasure); the social web application is distinctive among applications with the same purpose (uniqueness); users have the intention to continue to use the social web application and recommend it to others (loyalty).

4 Results

Participants.

A total of 162 respondents (53.70 % male, 46.30 % female), aged 20.73 years (SD = 3.879) on average, participated in the study. At the time study took place, majority of the sample (74.07 %) were enrolled to one of the undergraduate study programs at Polytechnic of Rijeka (POLYRI) whereas remaining 25.93 % were third-year undergraduate students of Informatics at the Department of Information and Communication Technologies, Juraj Dobrila University of Pula (UNIPU). Most of them (77.78 %) were full-time students. All of them had been loyal consumers of mobile Internet on a daily basis. More specifically, 82.72 % of students had been accessing the Internet via their smartphones more than three times a day while 59.26 % of them have been spending more than ten hours a week on using the Internet via their smartphones. In most cases (88.89 %), students had been employing their smartphones for accessing social networking sites like Facebook which was followed by other activities such as making and receiving calls (80.86 %), Web browsing (71.60 %), synchronous messaging and exchange of multimedia content by means of services such as Viber (69.14 %), listening to music (66.05 %), sending SMS messages (66.05 %), taking photos (55.56 %), mobile learning (35.80 %), playing installed games (29.63 %), mobile banking (25.93 %), and taking and sharing selfies (19.14 %). When the use of mobile Internet was considered, the students reported they have been employing it most frequently (88.89 %) for social networking and subsequently for the purpose of synchronous messaging and exchange of multimedia content (73.46 %), Web browsing (71.60 %), watching videos on services such as YouTube (67.90 %), accessing the educational content published on learning management system such as Moodle (56.79 %), reading news on web portals (51.23 %), mobile learning (37.65 %), accessing educational content located outside the learning management system (31.48 %), mobile banking (24.07 %), online shopping (19.75 %), and playing online games (17.90 %).

Procedure.

The study was conducted in controlled lab conditions and was composed of two main parts: (1) interaction with two social web applications for collaborative writing on participants’ mobile devices and (2) evaluation of their perceived mobile quality by means of the post-use questionnaire. Upon arriving to the lab, the participants were welcomed and briefly informed about the quality evaluation study. At the beginning of the scenario performance session, each participant received the form containing a list of 46 representative steps of interaction. Participants were asked to conduct all scenario steps twice – first with Google Docs and then using the Zoho Docs (both depicted in Fig. 1). After completing all the scenario steps with both social web applications, the participants were asked to fill out the post-use questionnaire. At the end of the study, respondents were debriefed, and thanked for their participation. The duration of the study was three hours.

Fig. 1.
figure 1

Screenshots of mobile versions of evaluated social web applications for collaborative writing (left: Zoho Docs, right: Google Docs).

Full size image

Apparatus.

The study adopted a within-subjects design comparing two social web applications. Majority of participants (80.86 %) completed scenario of interaction with social web applications on smartphones that run Android operating system whereas 79.01 % of them employed Google Chrome web browser to carry out particular steps of the scenario. Data was gathered with the use of the post-use questionnaire which was administrated online by means of the KwikSurveysFootnote 1 questionnaire builder. The questionnaire comprised 12 items related to participants’ demography and 137 items meant for measuring pragmatic and hedonic dimensions of mobile quality. Responses to the questionnaire items were modulated on a four point Likert scale (1 – strongly agree, 4 – strongly disagree). The sum of responses to items assigned to corresponding attribute was used as a composite measure which reflects particular facet of mobile quality. Differences between Google Docs and Zoho Docs were examined with Wilcoxon Signed-Rank Tests. The rationale behind the choice to employ this nonparametric equivalent to the dependent t-test relies on the outcomes of Shapiro-Wilk Tests which revealed that at least one of the variables in a pairwise comparison violates the assumption of normality in data (p < .05). In that respect, all the reported results are expressed as the median values. The effect size (r) was estimated as a ratio of Z-value and the square root of number of observations. According to Cohen [5], the values of .10, .30, and .50 indicate small, medium, and large effect size, respectively.

Findings.

The analysis of collected data revealed that mobile quality of Google Docs is significantly higher (Z = −9.756, p = .000, r = −.54) than those of Zoho Docs. More specifically, it appeared that at the level of quality attributes the aforementioned applications do not differ significantly only in terms of the degree to which they are usable every time users want to interact with them (Z = −1.171, p = .242) nor to the extent to which they facilitate management of created artefacts (Z = −1.580, p = .114). It was also found that the degree to which Zoho Docs has met requirements of eight quality attributes (recoverability, loyalty, reliability, attitude toward use, response time, customizability, responsiveness, and pleasure) is medium worse than those achieved by Google Docs. In the case of remaining 27 quality attributes, Google Docs proved to have slightly better scores than Zoho Docs. When the mobile quality of evaluated social web applications is considered at the level of categories, it appears that they differ mainly by the extent to which they met requirements related to system quality (Z = −11.039, p = .000, r = −.61). The set forth is mostly caused by determined differences between Google Docs and Zoho Docs in the extent to which users can personalize them (Z = −5.679, p = .000, r = −.32) as well as to the degree to which they are able to adapt their features to the specificities of the device on which they are employed (Z = −5.404, p = .000, r = −.30). On the other hand, it was found that two social web applications differ at least with the respect to the amount of effort users have to employ in order to complete tasks by means of their interface functionalities (Z = −5.724, p = .000, r = −.32). This is mainly caused by the fact that there is small difference (Z = −2.453, p = .000, r = −.14) in the quantity of physical and mental load users have to invest to complete tasks with evaluated social web applications. Results of data analysis are summarized in Table 1.

Table 1. Outcomes of data analysis
Full size table

5 Conclusions

The aim of this paper was to determine which attributes are relevant for evaluation of quality in the context of social web applications for collaborative writing when they are employed on mobile devices. For that purpose, an empirical study was carried out during which the participants performed scenario-based assignments with two applications (Google Docs and Zoho Docs) and subsequently evaluated their perceived mobile quality by means of the post-use questionnaire.

According to the results of data analysis, when evaluation of mobile quality in the context of social web applications designed for collaborative writing is concerned, one should first pay attention to measuring all relevant dimensions of system quality followed by assessing facets of service quality, user experience, performance, and eventually effort. When study results are taken into account at the level of mobile quality attributes, they can be, with respect to their relevance, classified into four categories: (1) essential attributes (\( r > \left| { - .29} \right| \)) whose requirements social web application has to meet because otherwise its overall perceived mobile quality will be significantly decreased; (2) sufficient attributes (\( \left| { - .20} \right| < r < \left| { - .29} \right| \)) which are also very important but failing to meet their requirements will be penalized less severe than in the case of essential attributes; (3) desired attributes (\( r < \left| { - .20} \right| \)) whose relevance is significantly lower than those of previous two types of quality attributes but social web application still needs to comply with their requirements to some extent; and (4) not relevant (\( r = N/A \)) which does not have to be taken into account when measuring mobile quality of social web applications for collaborative writing. All the aforementioned indicates that proposed mobile quality model and corresponding post-use questionnaire are capable to determine significant differences between very similar representatives of social web applications for collaborative writing.

As in the case of most empirical studies, this one also has its limitations. The first one is related to homogeneity of participants. Although students are representative users of social web applications for collaborative writing because they employ them for educational purposes, heterogeneous group of users could have perceived particular dimensions of mobile quality in a different manner. The second limitation deals with the interpretation of reported findings because they are generalizable only to social web applications for collaborative writing. In order to explore the robustness of presented results, further studies should be carried out. In that respect, our future work will be focused on exploring the robustness of reported findings on a various types of mobile applications evaluated by heterogeneous sample of users.