1 Introduction

Estonia is living in the breaking times. In spring of 2011 the World Bank announced that Estonia is now high income country (World Bank, 2014). This is subject of discussion but based on that decision in turn Microsoft decided to increase prices of software licences for Estonia more than 20 and up to 60 times depending on which licensing scheme to choose and what is the former situation of licences in specific institution. For time to time Microsoft is doing special offers but this is not the sustainable basis on which government can rely. The first deadline of price increase was 30th June 2014, which has been extended now for 30th June 2017. This fact started discussion and activities in Estonian society. First step was free software pilot project organized by Tallinn City Municipality Education Board (EPL, 2014). This project involved five educational institutions: 3 schools and 2 kindergartens from Tallinn city. The project ended successfully in April 2014 and continues currently in next phases were already some more schools are involved.

During the project for time to time people has been claimed that GNU/Linux distribution Lubuntu 12.04 LTS and also 14.04 LTS user interface (LXDE desktop environment) lacks of usability compared with previous successor Microsoft Windows from versions of XP till 8. In that reason we conducted a research of different Linux desktop environments. We formulated two main research questions: “does the Linux user interface have poor user experience” and “which desktop environment would be most suitable for educational institutions”.

Users dissatisfaction was a little surprise as the LXDE desktop environment was chosen based on its similarity of famous MS Windows XP user interface. Based on our study the LXDE got the worst user rating. This also explains the user dissatisfaction that were experienced during the free software project.

2 Literature Review

There are many articles about free software. One of the most comprehensive look is book “Handbook of Research on Open Source Software: Technological, Economic, and Social Perspectives” in 767 pages (St. Amant et al., 2007). This book has 110 contributors from different point of views over all the world. It introduces the philosophy, challenges, innovation, social, developing and so many other aspects of free software. In chapter 3 there are descriptions of how free software can be much better improved due to its open sourcecode. Main strengths are freedom to use, which gives also reliability even if there are some difficulties as well. These aspects do not prevent the use of software.

Reliability for users usually means that applications works as expected - this means the application does not crash every second or not cause data loss (Garvin, 1984). In our study overall reliability was good - only very few times virtual machines crashed during shut down.

In security user need to know the application trustworthy that it does not contain malware or network features are secured (Hoepman et al., 2007). From user perspective the openness is good for security, especially if the developer community is active.

On the efficiency side for user it means that the application has clear, easy to follow user interface (Glott et al., 2010). Also features should be documented so users can search and find them. Functions should meet user needs and the application must be responsive on user actions.

Also open-source software would be much more interoperable thaln closed source one (Money et al., 2012). Interoperability itself has a crucial importance, especially when free office suite should open proprietary file formats. This also helps to avoid vendor lock-in when producing documents in open formats and spreading them with suggestion download also free office suite like LibreOffice and use it.

There are not many studies of free software and aesthetics based on ACM, IEEE, Google Scholar databases. Leach et al. mostly describes morality and aesthetics in free software design (Leach et al., 2009). It is a quite philosophical article and even says, that free and open-source software development and community life (as “social machine”) is like we should live in 21th century.

3 Evaluation of UX

The goal of evaluation is to acknowledge competitiveness of free and open source software (FOSS). Evaluation of UX can be used to prove FOSS to be pragmatically usable and satisfying, but also attractive and pleasing.

Pragmatic qualities of UX denote, how users perceive the technology’s ability to help them in completing their task and reaching the goal. The way, how pragmatics is experienced, may be expressed with opposing word pairs: e.g. confusing-structured, impractical-practical, unpredictable-predictable, complicated-simple (Hassenzahl, 2010).

Besides pragmatic qualities are also user’s feelings that play important role to engage users. These are the hedonic qualities, described by the emotional attributes (e.g., ‘exciting’, ‘impressive’, ‘presentable’), emphasizing psychological well-being through non-instrumental, self-oriented product qualities (Diefenbach, 2013). First impression, for example has crucial role in user’s decision making: affect, caused by first impression happens so quickly that pragmatic usage can not even happen (Lindgaard, 2006). Hedonic qualities of UX can be expressed as word pairs: dull-captivating, tacky-stylish, cheap-premium, unimaginativ-creative, good-bad and beautiful-ugly.

Aesthetics of interaction has significant role while modifying hedonic quality of interactive product. Considering the effect, it has on engaging the users, current study will use aesthetic dimensions for assessing the UX of FOSS besides the traditional, pragmatic usability study.

4 Method

4.1 Evaluating Usability

Usability, in terms of UX, is a parameter of interactive product, that describes user’s ability to complete intended task. It integrates both - the features of interface and users ability to use the interface. The value of usability expresses how effective and efficient is the interface in completing the intended task. Usability data is pragmatic, quantitative and it can be objectively assessed/analysed.

Usability evaluation includes three general components: time to complete the task, number of errors during the task completion and ability to complete the task (whether the task was completed or not). Usability data can be collected via observation. The procedures for observation are: taking time, counting errors and keeping notes. Supportive techniques can be video and audio recording, screen recording, key/mouse logging and eye tracking. Some of the data, collected during observation can be used for evaluating hedonic UX.

System Usability Scale (Brooke, 1996) is quick and dirty questionnaire that has been used successfully since 1996. It can be used for evaluating pragmatic satisfaction. It comprises of 10 questions on 5p Likert scale (agree-disagree), addressing the user’s perceptions about using the product. Completing the questionnaire takes max 5 min and the questions are easily understood.

4.2 Evaluating Hedonic Qualities

According to suggested definition (Djajadiningrat, 2004), the aesthetic interaction refers to “things that are beautiful in use” and comprises of two components, neither of which should be addressed separately. These components are beauty of appearance and beauty of action. Traditional methods of evaluating perceived aesthetics collect user reported data about beauty of interface. According to definition, interaction aesthetics comprises of two types: beauty of appearance and beauty of use.

Simplest way of collecting data is single question like: do You find the interface appealing? The answer provides quantitative data where the scale varies from “beautiful” to “ugly”. Such a question is suitable for assessing user’s feeling at any moment throughout the study or retrospectively after the study. The answer does not define the type of stimuli and collected data allows to determine general aesthetic value, perceived and reported by user.

Hedonic qualities were evaluated with questionnaire, comprising of 18 questions, of which 5 questions were open ended and 6 questions were mandatory. One open ended question out of 5 was mandatory. Questionnaire included one question about first impression, one question about credibility, two questions about general aesthetics, two questions about style, five questions about visual aesthetics, five questions about aesthetics of action/dynamics, two questions about sound.

4.3 The Procedure

The procedure describes the way we used four different operating systems based on Ubuntu and running them in one desktop computer. The hardware had 8 GB of operating memory, 500 GB of hard drive and quad-core 2,66 GHz Intel processor.

We have chosen the most used Linux distribution Ubuntu and its flavours: Kubuntu with KDE -, Lubuntu with LXDE -, Ubuntu Mate with Mate - and Xubuntu with XFCE desktop environment (Ryan, 2010). During four months in the beginning of 2014 the Tallinn City Government in Estonia performed a successful pilot project of free software using Lubuntu Linux as operating system and LibreOffice as office suite (Tallinn City Municipality ICT information, 2014). We also tested other flavours of Ubuntu to understand which desktop environment would suits better for everyday usage.

Testing environment were built in top of Oracle’s VirtualBox virtualization software using the built-in screencasting feature to record user activities during certain tasks. We created four different virtual machines with latest available versions in testing period of February 2015:

  • Kubuntu 14.04.1 LTS 32-bit

  • Lubuntu 14.04.1 LTS 32-bit

  • Ubuntu Mate 14.04.1 LTS 32-bit

  • Xubuntu 14.04.1 LTS 32-bit

We prepared virtual machines as much similar situation as it would be in normal life - made software updates for whole operating system, installed latest versions of used software (newest versions of LibreOffice, Firefox, Thunderbird, VLC Media Player, Adobe Flash plugin, Java plugin, PDF-printer and also some other printers installed). Also VirtualBox Guest Additions were installed for smooth perfomance of virtual machines - all that software were downloaded from Oracle servers using Ubuntu repository and also website for VirtualBox extension pack. Actually virtualization gives pretty much real feeling and the most significant difference users notice is that operating system is working in program window and not in full screen.

Virtual machine had 2 GB of operating memory, 8 GB of hard drive. Most of these desktop environments require less but for more convenient and seamless user experience we used a bit more operating memory.

Testing were started by running each virtual machine separately and filling the questionnaire. Virtual machines were configured so that also screen recording of virtual machine started. The first question was required to answer prior using and rest of after filling certain tasks on the virtual machine. During the virtual machine run all activities were recorded into video file for later analysis.

Tasks were separated into three parts: operating system basic functions, file manager tasks and office suite tasks. In parallel participants filled questionnaire, which was in two parts: SUS (System Usability Scale) and hedonics. Also users compared tested operating system and office suite with former used ones - first impression and how it feels with already used systems.

Participants were each one by own computer and solving tasks in own pace (Fig. 1).

Fig. 1.
figure 1figure 1

Participants in tests

Full size image

5 Results

During testing approximately 24 h video recordings were collected from 46 different computers. There was quite comprehensive challenge to collect all data from different computers and analyze it.

Testing went almost smoothly - only few participants had problems. These problems were related with virtualization software VirtualBox, which very rare cases crashed during virtual machine shut down. Luckily it did not affect ability to run virtual machine again.

Participants had to solve three parts of tasks: operating system -, file manager - and office suite basic tasks. With each virtual machine were same tasks. Last test was office suite with last virtual machine. All results are summarized and collected into Tables 1 and 2.

Table 1. Video recording statistics
Full size table
Table 2. Questionnaire statistics
Full size table

The tasks were as follows:

  • operating system:

    • change wallpaper

    • make shortcuts to Firefox, LibreOffice, Thunderbird,VLC Media Player

    • set default printer as PDF

    • change default program of the given file type: set VLC as default MP4

    • change mouse working scheme as single click

  • file manager:

    • find a file by name, open it and then close again

    • navigate to location by path

    • create a directory with given name

    • copy the given file into previously created directory

    • rename the copied file with given name

  • office suite:

    • open previously renamed file and save in another file format

    • change whole document to default style

    • change first three paragraph titles as heading 1

    • find a given phrase from text and change font attributes

    • change page layout

    • print to default printer and open created PDF-file for a while and then close it

In video recordings (Table 1, Fig. 2) there were measured completeness of tasks and spent time. Surprisingly the LibreOffice had quite high completeness and relatively small time footprint.

Fig. 2.
figure 2figure 2

Virtual machine tasks completeness

Full size image

Comparing different desktop environments tasks completeness the XFCE-based Xubuntu 14.04.1 LTS got the best results. Then follows Mate desktop, KDE and last is LXDE.

At the same time accomplishing tasks in Mate desktop were a bit faster than XFCE-based Xubuntu (Table 1, Fig. 3). Also based on completeness the LXDE-based Lubuntu took less time than KDE-based Kubuntu.

Fig. 3.
figure 3figure 3

Virtual machine task average time

Full size image

So the completeness and spent time are different and here the Mate desktop seems to be fastest and XFCE desktop seems to be easiest to use.

At hedonics side overall results are as follows: the best one is XFCE-based Xubuntu, then not much less comes Mate desktop based Ubuntu (Table 2, Fig. 4). A slightly more difference are with rest of two desktop environments: KDE-based Kubuntu is on third place and LXDE-based Lubuntu is on the last place. Also SUS (System Usability Scale) results are in the same order (Table 2, Fig. 5).

Fig. 4.
figure 4figure 4

Hedonics statistics (scale 0…1)

Full size image
Fig. 5.
figure 5figure 5

SUS statistics (scale 0…100)

Full size image

5.1 Usability and SUS Analysis

Participants said that XFCE has the most clear user interface but also more untranslated menus. XFCE seemed also more modern than others. Also expected behaviour were in XFCE the best. Even users also appreciated KDE it seemed too many opportunities for most of users. XFCE also reminded a bit Mac OS for some users. As we see from video recording results also XFCE got the best results - the highest completeness rate and but not the fastest time to complete tasks. Several users said that they would prefer in future with XFCE-based Xubuntu instead of MS Windows.

Mostly LXDE lacks of usability were common feedback by participants. At the same time LXDE has been found as most simplistic and logical but aged desktop environment. Most complicated were to find different tasks from menus and also some participants did not find the search feature. Also missing graphical sound mixer were one of the claims. This all reflects also in results of worst completeness but almost same time spent for tasks. Several users said that they would never replace MS Windows with LXDE-based Lubuntu.

The Mate desktop were only slightly less usable than XFCE by completeness and fastest by spent time for tasks. Some users found main menu opening at top unfamiliar. At the same time some users appreciated two panels - one for applications and one for taskbar. Overal look and feel were modern and usable by testers. Also most testers found that they could use Mate desktop instead of MS Windows.

The KDE desktop seemed nice and modern but also a little bit overbloated with different bells and whistles eagerly consuming computer resources. Some testers were disturbed of transparency used in most of windows when moved. Also quite big similarity with MS Windows were mentioned. Mac OS X were mentioned by some users as to similar with KDE. Overall feedback was that some people would even replace their existing operating system with KDE-based Kubuntu.

6 Conclusion

In current study we tested four different graphical Linux desktop environments usability and aesthetics based on appropriate Ubuntu versions. Participants were tested using previously prepared virtual machines and by analysing screen recordings made during tests. Also questionnaire were prepared to collect testers feedback based on System Usability Score (SUS) and usability (hedonic) questions.

Overall results did show, that XFCE-based Xubuntu were performed most well. It had the best usability and SUS results but not the best in time completion of tasks. Most shallow reason of good results was similarity with currently used system.

The second best result was in Mate desktop based Ubuntu Mate, which had the best result in time of task completion and quite close results in usability and SUS.

The third best result were performed by KDE-based Kubuntu. Testers spent most time to complete tasks on Kubuntu but overall rating was even quite good.

The fourth and last result were LXDE-based Lubuntu, which got worst overall feedback from testers.

In Tallinn free software pilot project currently LXDE-based Lubuntu were used. But soon it will be switched to Mate desktop based Ubuntu Mate. Also our study showed best perfomance in completing given tasks and very close usability and SUS test results.

Also we realized, that current systems (mostly Microsoft Windows) has influenced participant’s perception of how operating system and office suite should work and look. But still testers found that different Linux desktops were quite usable and several said that they would start using Linux in near future.