Virtual Assistants for the Cultural Heritage Domain
Virtual assistants and similar software tools are gaining importance among phone and computer users. The most well-known assistants (Siri, Cortana, Google, etc.) provide general information to users and cannot be adapted to specific needs. Custom implementations usually cover specific domains and are specialized to provide a comprehensive set of information or functionalities from that area. Currently, there is a lack of general applications for cultural heritage, since most implementations are specialized virtual guides for museums or exhibitions. We present an overview of virtual assistants that we developed and are applied to the area of cultural and natural heritage, where we try to improve the current shortage of tools. The presented prototypes provide various functionalities to users and can help them to discover, learn and plan visits to cultural sights.
KeywordsVirtual assistants Natural language processing Web services
In recent years, virtual assistants (VA) have become a popular tool that helps users find information faster and in a more accurate manner. They are web-based services that serve as a human-computer interface to deliver information from a specific domain that the user is interested in or to interact with an information system. Companies like Google, Microsoft or Apple developed general assistants for mobile devices that help users find restaurants or shops close to their location, manage their phone or provide computer-aided support for software packages.
The main advantages of virtual assistants are their ability to understand, to a certain extent, natural language and present the answers in an enriched format (additional links, multimedia presentation, etc.). This way, users can provide questions and queries in natural language, which makes virtual assistants well-suited for less skilled or handicapped users. In addition, virtual assistants can implement speech-to-text and/or text-to-speech systems to further improve the user experience.
Nowadays, assistants are being developed for a wide variety of domains and applications and they thrive due to the increasing publicly available data on the internet, both structured and in free-form. In addition, web based services that offer API access can be integrated into the assistants to provide a comprehensive user experience (for example, booking of flights, transportations, events, tours, medical visits, hairdressers, etc.). In addition, modern third party services that provide advanced functionalities that can also be managed through APIs (for example, messaging, text mining, recommendations, path planning, etc.) can also be exploited by assistants.
Applications of virtual assistants in the cultural heritage domain are not as widespread as one would expect. There is a notable lack of applications for the interested public, providing general information and covering non-specific domains, since most applications are custom solutions intended as virtual guides for museums and exhibitions. The introduction of more generic applications and virtual assistants would help popularize and spread the appreciation for cultural heritage also among youngsters.
In this paper we provide an overview of the various virtual assistants developed at our institutions for the domain of cultural heritage and sightseeing. Virtual assistants integrate several advanced functionalities. Text mining and Natural Language Understanding (NLU) to identify the intent of the users and extract important entities from text. Next, they implement API calls to third party services in order to provide trip planning functionalities and data fusion. All presented assistants are available online and can be modified and thus applied to new domains or different languages.
2 Related Work
Virtual assistant in the domain of cultural heritage have been already applied to virtual visits of cultural sites. They are deployed in real or virtual environments as virtual guides to engage the visitors and deliver a comprehensive learning experience . Other implementations also provide mechanisms, such as multimedia, storytelling, and enriched content to entertain the user during the learning process [5, 6]. All these agents must be flexible, responsive, and provide natural interaction in order to be seen as a social partner by visitors.
Several applications were developed that deliver a personalized museum guide. Mathias et al.  proposed a new method for personalized museum tour recommendations. Their research tackles the problem of optimizing museum visits according to visitor’s preferences and artwork importance. Huang et al.  developed a museum guide system that uses association rule mining to discover recommendations from both collective and individual visiting behaviors. Pechenizkiy and Calders  presented a simple user-focused framework for personalizing museum tours that is focused on efficient learning since the system should be able to quickly provide relevant suggestions only after a small set of user preferences.
Other general applications of virtual assistants are in the area of education for instructions and learning , delivering information from large data sets like encyclopedias in libraries , for informational purposes in e-commerce , merging heterogeneous data using multi-agent conversational agents , tourism , and several others.
3 Overview of Deployed Virtual Assistants
3.1 Platform for VA Creation
We developed a cloud-based platform Asistent [2, 7] for the creation of virtual assistants and during a national project we created and integrated several VAs for web pages of municipalities in Slovenia. The developed web platform supports the creation, management and integration of virtual assistants to an arbitrary web page. It was developed as a Software as a Service (SaaS), where clients access the service via a web browser, which eliminates the need for software and hardware maintenance. Since the software runs in the cloud all future upgrades are applied centrally and are available to all clients instantaneously.
The knowledge base of the deployed VAs for the municipalities was constructed manually and contains the information that municipalities deemed the most important. It covers the general information regarding the municipality, their acts and regulations, instructions and news for residents, natural and cultural heritage and touristic information.
3.2 Virtual Museum Guide
Museums offer to its visitors a wide selection of artwork at their premises. The vast amount of exhibitions is often overwhelming for the visitors and most of them might not be of interest to them. This might cause to miss artwork or entire exhibitions that are best suited to their personal preferences. The growth of the Semantic Web in recent years and the availability of more and more content in the digital and linked data format available on the internet supports the creation and reuse of services of a higher added value. One of such is the virtual assistant for museums that offers virtual tours of institutions covered in the Google Arts & Culture project (Google Street View). At the time of writing this paper, more than 3400 museums and sights are included in the Google Arts & Culture project. To deliver a simple proof-of-concept we mapped into the virtual assistant content from three museums (British museum, Art Institute of Chicago, and National Gallery of Australia).
The application provides a natural language interface using a virtual assistant created through the Asistent platform. The users can input questions or requests and the VA takes care of identifying the correct answer and displays the corresponding feed from Google Street View. In addition, a simple recommender system is integrated in the web service that provides suggestions regarding exhibits targeted for a specific user. Since the recommender does not get implicit rating for items from users, we had to design a recommendation system that learns user preferences based on implicit data (queries, time spent on each answer, motion, etc.) and item features. The algorithm implements a content-based filtering mechanism that is well-suited for this kind of domains. To generate a recommendation, the recommender does not use data from other users but the similarity between items in the dataset and the history of previously searched items to predict ratings for user-item pairs not present in the dataset. We showed that in the limited domain, with a small number of learning examples and user-item pairs the recommender works well, even with new users.
The virtual assistant can deliver information about specific exhibit rooms in the museum (for example, Egypt room), groups of similar objects (for example, Chinese weapons), or important exhibits (for example, the Rosetta stone). The virtual assistant contains the most relevant items and groups from the three museums, but it does not provide a comprehensive overview of the museum, since the mapping and creation of the knowledge base was performed manually, which is relatively time consuming.
3.3 Slovenian Cultural Heritage Database
The developed virtual assistant was created with the aim to deliver a wide range of information to users regarding cultural, natural heritage, and visiting sites in Slovenia. Due to the large amount of data and information that needs to be provided, the mechanism for content delivery was designed in such way to remove the need for manually inputting and editing database entries. This is a crucial step to enable a comprehensive delivery of content for all sights in a specific country. We implemented a web crawler that parsed several web pages related to cultural heritage and touristic information in Slovenia (for example, www.slovenia.info). The crawler was able to classify the type of information that a page has and the type of the attraction (castle, lake, museum, etc.), extract metadata (location, region, descriptions, photos, related web page), and store the obtained information in a structured way. In addition, the crawler tried to obtain additional content also from external links that were included in the provided web pages. All the extracted content was stored in a relational database and the text in the main fields were indexed using Whoosh , a python library for indexing free-form or structured text and quickly find matching results based on simple or complex search criteria. Since the text in the database fields is in free-form, Whoosh provides an ideal solution to search within the text and at the same time obtain a relevance score that can be used to sort the search results.
The entire search mechanism in the virtual assistant is composed of four parts. First, the NLU module is used to obtain the intent of the user. The intent can be regarded as the type of the question. The user might want to search for a specific sight or obtain a list or related items. Second, entities are extracted from the query. They are linked to the information extracted with the web crawler. For example, the type of the sight (museum, castle, lake, churches, etc.) and the region or location where the sight is. Next, the obtained values for entities are matched against the records in the database and a set of results is obtained based on the search criteria. Finally, the answer to be displayed in the virtual assistant is generated. A short description is provided if the user searched for a specific sight, or a list of matched items if the query return multiple hits.
Show me some lakes in the Dolenjska region.
List the castles near Kranj.
Show me some info regarding the Ljubljana castle.
Museums in Ljubljana.
In Fig. 3 the user searched for a specific sight (the Predjama castle). The assistant identified the adequate entity and obtained the stored entry in the indexed database, based on the title field of the sight. A short description is provided in the text area of the virtual assistant and the related web page is displayed in the background.
The developed assistant and the search mechanism can be applied to different domain and languages.
3.4 Virtual Assistant for Sightseeing
Tourism is a domain that can support the introduction of virtual assistants and they can provide significant added value to tourists. Companies that organize trips, tourist centers in cities and others can use the functionalities of a virtual assistant as additional services that they offer. Virtual assistants can deliver information from an area or regarding specific sights, provide content in a segmented, appealing, and entertaining format, they can help users to book tickets for events, sights, and urban transportation, create personalized city tours, and more.
We are currently developing a comprehensive platform that provides all the listed functionalities to enable smart tourism. The system uses several external services to improve the tourist’s user experience during trip planning and provides guidance amid the actual trip. The platform integrates: (i) Rocket.Chat, a chat platform where users can communicate; (ii) Interface with Messenger; (iii) eTurist , a tour planner based on a recommendation system; (iv) Asistent, a rule based question-answering toolkit; (v) Dialogflow, a natural language understanding toolkit.
The prototype version of the platform supports various kinds of interactions and queries during the communication with the user. The classification of user intents is performed by the specialized virtual assistant that can understand queries and commands from the following categories: general information about a sight, sights near a specific place, similar sights near a specific place or region, adding sights to the preference list, generate path to a sight, recommendation of sights. In addition, the virtual assistant is able to recognize an important word or sequence of words in the query. These words are from the following categories: sight name, sight type, place, and region.
Based on the recognized inputs from the user, the NLU module constructs the appropriate commands, methods, and API calls. The webhook functionality then retrieves the needed data from third party services, if required.
This paper presents an overview of the various virtual assistants developed at our institutions for the domain of cultural heritage and sightseeing. We presented a service for the creation of virtual assistants, an advanced tool for virtual interfaces and three prototypes of virtual assistants providing functionalities and content for different domains. First, a web based virtual guide for museums that delivers information about specific items, groups of exhibits, and rooms inside the British museum, Art Institute of Chicago, and National Gallery of Australia. The assistant shows a short description for each query and displays the pertinent feed inside the museums from Google Street View. Second, a virtual assistant for delivering information about sights, cultural and natural heritage in Slovenia. The assistant implements an intelligent mechanism for the identification of intent and extraction of entities from the user query. This way it is possible to deliver specific and correct information to the user. Finally, we presented a comprehensive platform for smart tourism and sightseeing. The platform integrates chat functionalities in several interfaces, a NLU module, a virtual assistant to understand the requests of the user and deliver enriched answers, and is able to use several third party services to obtain specific information and create personalized trips.
All services and prototypes are available online and can be applied to other, different domains and languages.
- 1.Bickmore, T., Pfeifer, L., Schulman, D.: Relational agents improve engagement and learning in science museum visitors. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds.) IVA 2011. LNCS (LNAI), vol. 6895, pp. 55–67. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23974-8_7CrossRefGoogle Scholar
- 2.Mathias, M., Moussa, A., Zhou, F., Torres-Moreno, et al.: Optimisation using natural language processing: personalized tour recommendation for museums. In: Proceedings of the 2014 Federated Conference on Computer Science and Information Systems, pp. 439–446 (2014)Google Scholar
- 3.Huang, M.Y., Liu, C.H., Lee, C.Y., Huang, Y.M.: Designing a personalized guide recommendation system to mitigate information overload in museum learning. Educ. Technol. Soc. 15(4), 150–166 (2011)Google Scholar
- 4.Pechenizkiy, M., Calders, T.: A framework for guiding the museum tours personalization. In: Proceedings UM 2007 International Workshop on Personalization Enhanced Access to Cultural Heritage (CHIP) (2007)Google Scholar
- 5.Cimiano, P., Kopp, S.: Accessing the web of data through embodied virtual characters. Semant. Web 1(1), 83–88 (2010)Google Scholar
- 6.Doswell, J.T.: PECA: pedagogical embodied conversational agents in mixed reality learning environments. In: AIED, p. 957 (2005)Google Scholar
- 7.Kuznar, D., Tavcar, A., Zupancic, J., Duguleana, M.: Virtual assistant platform. Informatica 40(3), 285–290 (2016)Google Scholar
- 10.Xu, B., Pan, Z.G., Yang, H.W.: Agent-based model for intelligent shopping assistant and its application. In: The first Conference on Affective Computing and Intelligent Interaction, Beijing, Citeseer, pp. 306–311 (2003)Google Scholar
- 13.Project Asistent - virtual assistant for municipalities. http://www.projekt-asistent.si/. Accessed 26 Apr 2018
- 14.The Whoosh Library. http://whoosh.readthedocs.io/en/latest/intro.html. Accessed 26 Apr 2018
- 15.Cvetkovic, B., et al.: e-turist: an intelligent personalised trip guide. Informatica 40(4), 447 (2016)Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.