1 Introduction

Museums and cultural institutions have invested and are investing a lot to introduce cultural heritage in the digital era. Number of digital objects available in Europeana [1] has increased significantly. Currently, it provides access to over 33 million digitised objects, having already reached 30 million objects in November 2013 [2]. This means that, although around 88 % of Europe’s cultural collections (in average) are still not digitised, the trend is to increase these numbers [3]. Digitisation plans are increasing at European level, and a lot of progress has been done in the last years to raise the amount of digitised cultural material as well as with regards to online accessibility of cultural material, including more cross-border collaboration and public-private partnerships [2].

With the aim to re-use the available digitised content and provide it to the user through an adaptive and personalised experience, it has been developed COOLTURA. It is the outcome of the European funded project TAG CLOUD [4] that aims to explore the use of cloud-based technologies that lead to adaptability and personalisation to increase engagement with cultural heritage. COOLTURA has been developed as a platform and an application following a User-Centered Design methodology [5, 6]. COOLTURA Platform is an open data oriented platform which holds the necessary intelligence, through a set of algorithms, to provide a set of scalable services for cultural engagement. On the other hand, COOLTURA App access the cultural content processed in the COOLTURA Platform and provides the user with a unique personalised cultural experience. This is done through the visualisation of recommended cultural content, itineraries and routes, the exploration of cultural objects through augmented reality, storytelling and games, and the interaction with cultural objects through the usage of QR codes, NFC tags and/or Bluetooth beacons. It also allows the user also to share the cultural experience through social networks.

2 Background

The Cité des Sciences et de l’Industrie (CSI), a “universcience” site, is the biggest science museum in Europe, welcoming 3 million visitors per year [7]. In 1981 the architect Adrien Fainsilber decided to open its building to light, a “source of energy of the living world”. It is located inside The Park de la Villette, which is the third-largest park in Paris, 35.5 hectares in area, located at the north-eastern edge of the 19th arrondissement.

Through the development of science and technology, the Cité des Sciences et de l’Industrie provides the necessary learning tools to better understand the world in constant change around us. Each exhibition space is open to all age groups, from junior to senior, from tourists to professionals/researchers or to the simply curious.

On the other hand, the Integrative Usage Lab (IUL LUTIN) is a Living Lab located at CSI in Paris. IUL-LUTIN has primarily been designed as a Living Lab for usability: a research and development lab that provides facilities for companies who wish to evaluate the usability of new products, services and digital media. Nowadays, it is becoming a place for experiencing scientific mediation and learning within a Living Lab methodology, taking into account ethical issues [8, 9], and with a special focus on digital technologies for museums, mainly because LUTIN is a museum structure for museum validation of technologies dedicated to visitors.

In the past, the Cité des Sciences et de l’Industrie (CSI) and IUL-LUTIN carried out several studies and innovative experiments regarding different aspects, such as websites of museums [10], cultural heritage for children [11], Open Science Resources [12], augmented reality [13, 14], learning [15] and tours [16]. The main objective of the most recent experiments was to enrich the visitor experience [17]. Between real and virtual (augmented reality [18], documentary records [19], audio annotation, associated metadata, social media, etc.), the visitor may live a unique experience by combining both, real and virtual environments.

This manuscript describes the Living Lab experiment carried out in the CSI together with IUL-LUTIN to evaluate the TAG CLOUD concept through the experimentation of COOLTURA App. The experiment has been carried out within the framework of the eCultValue project [20], co-funded by the FP7 programme of the European Commission. This project aims to support and encourage the use of new technologies that have the potential to revolutionize new ways to access cultural heritage and experiences offered by cultural resources in real and virtual environments or a mix of both. Concretely, eCultValue is looking at technologies arriving from EU funded projects, promoting these technologies to stakeholders who will apply them and relate technologies to showcase scenarios for easier up-take.

The eCultValue project launched an Open Call for Living Labs to run experiments for the concept validation at European museums of technologies for cultural heritage coming from European projects. COOLTURA, the TAG CLOUD App, was selected to be tested in the validation experiment performed by the Integrative Usage Lab (IUL LUTIN) and CSI in October and November 2014. The COOLTURA application provides a new way to enjoy culture by combining visualisation, exploration and interaction with cultural heritage artefacts. In the context of CSI and IUL-LUTIN Cultural heritage past projects, TAG CLOUD /COOLTURA appears to be the bridge that is conveying an informational richness to visitors: TAG CLOUD /COOLTURA may be a real experience for the Park la Villette visitors. This rich cultural environment in events and entertainment offers fertile ground to test this concept. The experiment focused mostly on the Cité des Sciences et de l’Industrie (city of science and industry), the Géode (geode), the Argaunaute (Argonaut) and the Bibliothèque des Sciences et de l’Industrie (BSI – library of sciences and industry). While using mobile devices and COOLTURA, visitors have the opportunity to “augment” their visit with additional information, videos, pictures and social media. They also have the opportunity to be located and discover other activities around them and know, for example, how to book a ticket for an exhibition (Fig. 1).

Fig. 1.
figure 1

Cooltura in the Cité des Sciences et de l’Industrie, Paris, France

Full size image

The aim for this experiment is based on the analysis of the usage of ICT tools (such as mobiles, PDAs, Augmented Reality Displays, etc.) in cultural institutions. This kind of devices is more and more used for guiding the user through a museum while providing information about the visit and objects in the museum. These devices are often used to integrate or to provide additional information to what is already displayed in the current context. For this reason, when using a mobile device, the user has to choose how to share attention between the intended object (the object of interest), contextual information in the real scene, and the additional information to get from the mobile device. The use of ICT tools for personalising the visit is likely to affect the way visitors explore the museum and the way they learn from the museum contents. How these technologies can influence the visitor’s habits and change the museum experience is far from being understood and clearly assessed.

Thus, the main objectives of this experiment consisted in assessing and validating the use of COOLTURA (an App for smartphone or tablet) in real-life settings (museum). For this it was aimed to measure the user acceptance of the proposed technologies, including the effects on the visitors’ habits of information searching and attention strategies as well as overall comprehension and learning.

3 Methodology

To prepare and carry on the experiment it was needed, first, to define and design the experiment, second, to prepare the environment for the experiment, and finally to perform it at the Cité des Sciences et de l’Industrie.

3.1 Design of Concept Validation Experiment

As mentioned in the previous section, the main objectives of the experiment consisted in assessing and validating the use of COOLTURA App in real-life settings, as in the CSI /IUL-LUTIN. Thus, it was aimed to be performed in real-life conditions with regular visitors of the museum.

In order to measure the effects of using an application, as COOLTURA, in their visit to a museum and how it affects on the visitors’ habits of information searching and attention, the LUTIN Mobile Platform was used for observation and evaluation of Users Behaviour, by using eye tracking, camera goggle, audio video recorders and micro cameras. This allowed obtaining quantitative and qualitative data to refine the solutions and services experienced. In addition, questionnaires were used.

The experiment was planned to be carried out in three phases, described below:

  • Phase 1: Welcoming

  • Welcoming at the Cité des Sciences et de l’Industrie;

  • Presentation of the COOLTURA application, as an interactive medium to explore several sites (Geode, BSI, Argonaut);

  • Exploration of the application by visitors: participants were encouraged to freely explore the application.

  • Phase 2: Preparation

  • Completing a questionnaire on user’s profile (regarding usage of smartphone and museum habits);

  • Installation and calibration of equipment (camera, mobile eye tracker).

  • Phase 3: Experiment performance

  • The visitor must search for the three sites (Geode, BSI, Argonaut);

  • Scan the QR code;

  • Discover the site information. The test stops when the participant/visitor finds all of the information on the site while visiting the museum.

During phase 3, the experiment performance, supervisors observed visitors using the TAG CLOUD services (i.e. COOLTURA App). In addition, participants were asked to provide their appreciation, ideas about innovation, criticisms and recommendations while performing the experiment, following a Think Aloud Protocol (TAP) [21]. After the experiment, some questions were asked to participants in order to complement the information gathered along the experiment performance.

Participants

Participants were randomly selected from regular visitors of the museum (i.e. Cité des Sciences et de l’Industrie) with the following criteria for user’s inclusion:

  1. 1.

    To be there for the museum scientific and technical objects, not knowing the places to visit (that were mainly outdoor objects of the museum);

  2. 2.

    To use smartphone or tablet.

By fulfilling these criteria, participants were asked at random to participate in the study that was said to be about a new type of experience while visiting the museum and with the use of a mobile device (i.e. smartphone or tablet).

Materials and tools for test support

A questionnaire was prepared to gather the following information from participants’ profile, which was given during the phase 2 of the experiment:

  • Age, gender, profession.

  • Use of technology.

  • Use of Smartphones, specifically.

  • Frequency of visits to museums.

  • Use of museum guides (printed, human guide, audio, apps).

In addition, to perform the experiment in phase 3, the following materials were planned to be used:

  • The “COOLTURA” App.

  • The “Barcode Scanner” App.

  • Smartphone Samsung Galaxy S5 tablet 5.1 inch (12.7 centimetres) screen diagonal.

  • Three QR-codes (for the three sites: Geode, BSI, Argonaut).

  • The recording and measurement equipment:

  • Eye-Tracking Glasses (recording of fixations and eye movements)

  • Camera Scene /GoPro (user behaviour)

  • Questionnaire and interview procedure (profile, collection of opinions and of feelings of the user)

With the measurement equipment of the LUTIN Mobile Platform, it was aimed to measure the following variables:

  • Interaction:

  • a. With menus,

  • b. With navigation,

  • c. Perception of interactive elements.

  • Ergonomics:

  • a. Eye fixation on text,

  • b. Eye fixation on icons.

  • Feelings:

  • a. While using COOLTURA,

  • b. After using COOLTURA.

Questions driving the research experiment

As mentioned before, the main aim is based on the analysis of the usage of ICT tools in cultural institutions. Thus, during the experiment, participants were asked to provide their impressions of the usage of the App, as well as their ideas and recommendations to improve it. The experiment was driven by the following questions, which were aimed to be answered by using observation and TAP:

  1. 1.

    Are LL participants able to identify the elements of interaction?

  2. 2.

    Are they able to understand their function?

  3. 3.

    Understand how to interact with?

  4. 4.

    Users do they encounter difficulties in the menu navigation?

  5. 5.

    Are the answers offered by the application in line with user expectations? Or are they rather confusing or unpleasant?

  6. 6.

    What do they find to be particularly satisfactory or to be very disappointing? Why are the functions they would use the most? The least?

  7. 7.

    Under what conditions and in what circumstances they would consider using COOLTURA?

3.2 Preparation of the Experiment

Before carry on the experiment, it was needed to prepare the content from the Cité des Sciences et de l’Industrie to be included in the COOLTURA App. The first step was to implement the content (videos, images and texts) about outdoor objects (the three included in the experiment: Geode, BSI, Argonaut) of the museum in Android 4.0 for smartphones and tablets.

Once COOLTURA App was ready with the CSI content, training on the usage of the application was given by the TAG CLOUD team to the CSI and IUL-LUTIN team involved in the experiment. Then, an internal testing of the application was carried out by the CSI and IUL-LUTIN team, where all functions were checked as well as the how-it-works.

Next step was to prepare the CSI environment for the experiment, and for this, QR codes were placed in the three spaces included in the experiment in order to interact with the objects (Fig. 2).

Fig. 2.
figure 2

QR code in one of the three sites at CSI (Geode, BSI, Argonaut)

Full size image

After this, the LUTIN Mobile Platform was prepared to be used, and questionnaires were prepared and placed available online.

Finally, once COOLTURA App, the CSI environment and all materials and tools to be used were ready for the experiment, regular CSI visitors were recruited as participants, according to the criteria mentioned in the previous section.

With these criteria, regular visitors of CSI (n = 12; mean age: 31, 7 y) were recruited as participants, in addition of usability experts (n = 5) [22].

3.3 Experiment Performance

The purpose of the Living Lab study of COOLTURA was the understanding how museum visitors are using the application to visit three sites in the City of Science and Industry (the Geode, Scientific Library and the Argonaut) by evaluating the application in its ability to disseminate content: Do visitors read the texts? What is their perception of QR codes? Is their navigation satisfactory? What’s about their user experience?

To do so, the dimensions of the analysis were:

  • Their path within the application.

  • Their favourite functions and their neglected functions.

  • Their understanding of these functions.

The data gathered was related to:

  • Project Architecture.

  • Navigation /ergonomics.

  • Features.

  • The proposed interactions with the museum.

  • Strengths /weaknesses (advantages & disadvantages).

Finally, participants were using the COOLTURA App with (n = 5) or without eye tracking devices, having then open questions (n = 6) or on line questionnaires (n = 6).

The running of the experiment was done as expected and as described in the above sections.

The use of COOLTURA by the regular visitors was done in real-life settings (CSI), having the task of discovering information about the museum objects while interacting with the App. Regular visitors were regular visitors for the CSI services: Library, Fablab, Living lab (n = 6), but also for exhibitions (n = 6), young adults (n = 8), from both gender: males (n = 9), females (n = 3), as well as with family (n = 2).

Usability experts were CSI (n = 1) and LUTIN members (n = 4) that agree to expertise the App. The use of COOLTURA by usability experts was done as they please according to their rigorous way of evaluating the ergonomy of devices [23, 24].

4 Results

4.1 Results from Visitors

Main results gathered from visitors were that:

  1. 1.

    They all appreciate the graphical design of the interface: simple, enjoyable, readable and nice.

  2. 2.

    75 % do appreciate navigating with COOLTURA.

  3. 3.

    All of them find easy to use QR code.

  4. 4.

    25 % find difficult to look for QR code while visiting the museum.

  5. 5.

    Half of them would like to have the App for their next visit.

  6. 6.

    Half of them would pay the App if included in a ticket price.

Participants were wearing eye-tracking glasses while visiting the museum, searching for QR code to flash in order to get the content. The eye-tracking device allowed understanding the process of using COOLTURA:

  • (Stage 1) searching for QR code in the environment,

  • (Stage 2) flashing

  • (Stage 3) getting the content in the content of the museum object.

There were results according to each of the 3 stages of the process.

Stage 1. The participants knew that there were QR codes for some of the museum objects and were searching around it where they could find the corresponding QR code. The sight scan path of the participants show that they were looking at places where the QR code could be located in a goal-directed search, but not paying attention to other things. Although interesting, this was time consuming, attention consuming with sometimes some mental workload.

This was the most problematic phase according to eye movements recording. Participants that were looking at their own eyes movements recording were complaining of this difficult task. They recommended that instead of searching for a QR code, it would be more convenient to discover it: “When you walk and see a QR code then you flash it!” and QR code could have a specific indication for identification in the environment: a large coloured pictogram.

Eye-tracking participants found that installing QR code and pictograms in the environment of the museum was of poor aesthetic. They reminded having this feeling in the real situation while they were viewing their own eye-tracking recording. Some of them (3) were asking why QR code is needed since the museum objects are geo-located: “When you walk, the App (COOLTURA) could inform you that it has some information about closed objects. Thus you at the screen and you decide if you are interested or not”. One participant said that “if there are many museum objects, there could have some filters made from your personal interest” (Fig. 3).

Fig. 3.
figure 3

Stage 1 of the process of using COOLTURA: searching

Full size image

Stage 2. When participants found a QR code, participants had to flash it. In fact they always were in the last search-find state. Recording of sight scan path shows that it took some time to go to the flash command and sometimes with errors.

As other participants, they recommend not having to search for the QR code command on the COOLTURA interface (Fig. 4).

Fig. 4.
figure 4

Stage 2 of the process of using COOLTURA: flashing

Full size image

Stage 3. Eye-tracking recording indicates two types of strategy. Some content texts were read wholly before exploring and discovering the target objects while with some other contents were partly read before exploring and discovering the target objects (Fig. 5).

Fig. 5.
figure 5

Stage 3 of the process of using COOLTURA: getting the content

Full size image

While viewing their sight scan path recording, participants said that there could have specific text for specific parts of the museum object. Texts should be more precise: “we don’t need general information we could find everywhere but the information we need”. They also said that pictures could be displayed according to the viewing point of view and with some augmented reality information for naming parts of the museum objects (Fig. 6).

Fig. 6.
figure 6

Usage of COOLTURA without (left) and with (right) eye tracking

Full size image

4.2 Results from Experts

Experts found that the structure of the App was a little too complex and could be made simpler. The fact that the QR code function for instance was not attached to the main screen (it was provided in a slide menu) was found uncomfortable. They all recommended having all of the functions always accessible.

Usability experts developed a map of the COOLTURA interaction design showing how one might navigate when using it, according to their expertise.

5 Conclusions

The experiment was of high interest, on one hand, for both the CSI and for LUTIN, to know better about the implementation of mobile applications for the museum. On the other hand, this evaluation was very interesting from the point of view of TAG CLOUD because it provided TAG CLOUD with more information about the evaluation of COOLTURA, from a complementary perspective than the one proposed within the project. This experiment gave information about the User Acceptance and User Experience as well as evaluating the Users’ Behaviour measured with different techniques such as eye tracking, camera goggle, audio-video recorders and micro cameras; very useful to enhance the COOLTURA prototype for improving the experience of using it.

After the experiment results, it can be concluded that user experience of using COOLTURA, although generally was positively appreciated, should be improved and made simpler, more intuitive and with more quick access to key elements or functionalities.

Another conclusion is that people generally liked to interact with the environment, but doing it by placing QR codes was found not convenient for outdoor spaces, where GPS location can be used. In addition, quarter of the participants found difficult to find the QR codes. Thus, in case a sensor or code (as a QR code in this case) has to be placed to be searched and scanned to interact with the physical object, it should be easy to identify.

With this experiment, it was also demonstrated that while searching for a code to interact with the environment makes visitors to not pay attention to other objects around. If instead of asking for searching QR codes, the aim is to let the user discover the environment, then the process will be less time consuming, and probably visitors will experience more the whole cultural environment.