Keywords

1 Introduction

The user experience of a product is the lived and felt interaction with that product, be it a web site, a tangible object or an administrative process [1]. These days, it is not enough that new products and services offer clear gains in terms of efficiency or effectiveness, they also need to provide a memorable experience. This argument can also be applied to introducing new services such as sentiment analyses in support of workplace learning. Adoption of new technologies cannot be reduced to an argument about functional benefits or user interface aesthetics. User experiences such as engagement, absorption, education or joy become ever more important if users are expected to stick with a new technology [2].

A second key concept of this paper is sentiment analysis, generally understood as a natural language processing task with the aim to extract feelings, affects, emotions or opinions in a text [3] opposed to the extraction of facts, e.g. characteristics such as weights or prices. Outputs generated by sentiment analyses can be very diverse including simple dichotomies (positive versus negative expressions), scoring approaches from −5 (very negative) to +5 (very positive) [4], or categorical approaches such as distinguishing basic emotions such as joy, trust or anger [5]. Although sentiment is already widely applied for analysing texts included in product or movie reviews, political discourse analysis or spam detection, sentiment analysis is in its infancy so that completeness, accuracy and predictive powers of analytical results are far from prefect and also the degree of automation or human intervention needed differs depending on the analytical methods used [6].

Sentiment analyses can be presented in a variety of formats, with differing levels of details and explanations. In this article the primarily intended user group are course tutors facilitating online course for adult learners in large public employment service centres. The underlying assumption is that tutors’ flow experiences, when exploring increasing levels of analytical details, is impacted by conceptual decisions such as the unit of analysis for sentiment detection (e.g. words, n-grams of two or more words, sentences or larger text units) or the sequence of analytical results (e.g. hot-spots in discussions and the respective context in terms of surrounding texts as well as how pervasive a sentiment is in relation to a group of learners). Also the analysis itself can follow a number of different approaches, which are often grouped into dictionary based approaches and methods based on machine learning, which itself can deploy supervised or unsupervised algorithms, depending on whether the system can be trained or not [7].

Moreover, user experience is not subject to systems design alone; e.g. the possibility that sentiment analyses could be misused for performance evaluations rather than for supporting course management is a serious concern for learners and tutors alike. Knowing about what data is used, for what purpose and by whom becomes part of the user experience.

2 State of the Art and Research Objectives

Sentiment analysis for training and education purposes is still very much discussed as part of the learning analytics debate. Learning analytics is analysis of data generated from and about the activities of learners with the aim to better understand learners and optimize their learning experiences [8]. Learning analytics can, for example, use data from social networks, learners’ textual contributions, their activities within virtual learning environments and their navigational paths. These learner centric data can then be combined with tutor and lecturer inputs and other circumstantial data such as group sizes, frequency of off-line activities or learner profiles of course participants. Increasingly the explosion of available data as well as their innumerable combinations raises concerns about learning analytics’ pedagogical values [9]. After all, training and education are also economic activities, hence using learning analytics to optimize the use of scarce resources, is an objective that has an increasing appeal to providers of education and training. The dual purposes of learning analytics are also reflected in related products where dashboards show developments over time, allowing for comparisons of courses based on their performance metrics, learner drop-outs or materials used. Adopting a business logic, the upward trending graph is seen as a sign of success, neglecting the need to integrate the computational aspects of data analytics with the methodological aspects of learning [10]. A good example where this integration has happened is [11], where clustering of learners based on their activities has been related to learner motivation such as Herzberg’s theory.

There is a wide range of possible learning analytics technologies that can be used as listed in [9], including:

  • Predictive modeling, based on mathematical techniques such as factor analysis and logistic regression;

  • Social network analysis, identifying interaction patterns such as subgroups within a larger course or courses which are more teacher/tutor centric;

  • Analysis of usage data, often including the tracking of clicks or time spend on certain learning activities;

  • Text analysis, e.g. identifying the conceptual richness of a learner’s contributions or, as done in this paper, estimating the sentiments and emotions within online discussions.

An area where learning analytics shows a lot of potential are MOOCs (Massive Open Online Courses). On the one side, student numbers in the hundreds and thousands generate large and rich data to make meaningful use of algorithms depending on the size of datasets. On the other side, depending on the number of participants, tutors can not read every posting and monitor the activities of every learner in order to anticipate a conflict or identifying the learners who would need additional support [11]. However, there is a danger that we measure and use what is there rather than what we need. Or put differently, given that we concentrate on learners’ behavioral data, there is a risk that we go back to behaviorism as the dominating conceptualization of learning [8].

Hence this paper moves towards including sentiments within learners’ expressions as an estimate for a group’s wellbeing, e.g. a group’s ability to accommodating a variety of opinions and empowering all members of a group to participate in collaborative learning activities [cf. 12]. A first step in that direction is the provision of related analytics to tutors and trainers. For this information we generated a number of visualizations (proof-of-concept prototypes), which were presented in a connected way, meaning that a given visualization (e.g. displaying an overly negative tone in week 2 of an online course) could trigger another visualization (e.g. showing the keywords of the debate, related co-word analyses or relevant text snippets). Eventually, we aim to shed light on the added values of sentiment analyses to the facilitation of online workplace trainings, whereas we wanted to avoid a purely cost - benefit driven evaluation. We aim to include the user experience as a method to extract the added values of sentiment analyses as seen by practitioners, placing the technology in the context of their daily workflows [13].

Our expectations are that trainers benefit from better understanding the emotional ups and downs during their courses. Learning for career and labour market transitions can be seen as occurring across four domains: relational development; cognitive development; practical development; emotional development [14]. Thus understanding which emotions are expressed during the learning process can improve trainers’ options to facilitate the development of personal qualities, not only in dealing with others, but in dealing with one’s own emotional and practical development.

3 Method

Our method included two steps: (a) the generation of proof-of-concept prototypes based on the data of an actual workplace-training course and (b) exploratory interviews with tutors and human development experts.

3.1 Proof-of-Concept Prototypes and Scenarios

The purpose of the proof-of-concept prototypes was to demonstrate possible applications and visualization of data analyses, in order to specify future requirements, tell a story about the expected benefits and, generally, have an open dialogue about the conditions under which learning analytics would be useful or maybe even be harmful [cf. 15].

In this paper we use a specific branch of sentiment analysis, based on dictionaries, which were developed to extract the most likely polarity (pos. or neg.) of a sequence of words (sentences, postings, tweets etc.). The full gamut of techniques is shown in the figure below (Fig. 1).

Fig. 1.
figure 1

Sentiment classification techniques [7]

Full size image

Our sentiment analyses were based on text materials (n = 1,1170 postings) from 68 participants in an online workplace training course. The participants of this course were all employees of public employment services (Work Coaches) and aimed to learn about the challenges of a fast changing labour market. Learning objectives included developing a set of skills and competences in areas like coaching or providing labour market information, as well as the use of digital tools.

3.2 A Dialogical Evaluation of Prototypes

Bussolon [16] highlights the fact that despite 30 years of UX design, few publications explicate their understanding of ‘experiences’. What follows are often UX definitions containing a loose collection of concepts rather than a systematic organisation of what influences ‘experiences’. Research on user experience highlights the need for an interpretative, iterative and, most importantly, dialogical approach to understanding and evaluating user experiences [1].

Or as argued in [17], each technology includes both, a response to a problem and a hypothesis about how this problem will be solved. Hence the dialogue between user and designer can enable

  1. (a)

    the subtlety needed to capture even minor divergences from what was intended by the design and

  2. (b)

    the empathy needed to see users as the full persons they are, with needs going beyond efficiency and effectiveness [18].

The work with prototypes builds a bridge between the developers’ design alternatives of a future system or feature and the users’ needs by offering an intermediate presentation which is technically feasible and affords practical interpretation [19]. In our case we used low-fidelity prototypes [20, 21] in the form of paper-mock ups, as the hand-made appearance should focus the attention of the interviewees on the content and the functionalities of the proposed sentiment analysis rather than the appearance. We presented these prototypes along narrative descriptions derived from the data of the specific online workplace training course to facilitators exploring the goals they would like to reach when consulting the sentiment analysis [22].

We conducted five exploratory interviews, presenting the scenarios and prototypes. The interviewees were all facilitators of workplace learning interventions. Two interviewees were facilitators in the course that provided the pilot data for the sentiment analysis tool design and was structured along topics such as cultural change, impact of the digital, coaching, labour market information and sectorial knowledge, reflection on experience and learning. One of these course facilitators is part of the learning and development team within the organization, the other one is working for an academic institution advising the public organization on human development aspects. The third interviewee facilitated previously a very similar course in the same organization and was involved in the development of the new course.

Given the fact that the data in the low-fi prototypes were based on the real comments and discussions of this online course, the research team had the opportunity to anchor the feedback of those three interviewees in their concrete experiences with the course and the sentiments they witnessed from course participants.

The other two interviewees are employees and learning providers of a public employment service in another country. While they facilitated an online course, which had another training focus, both facilitators are familiar with the course setting providing the testing data for the sentiment analysis. The feedback from these two provided to enrich feedback with experiences from another context.

4 Conceptual Framework

The paper is based on a modular process for designing the user experience of sentiment analyses in the context of supporting large scale online training courses. Figure 2 presents a chain of decisions included in the design of sentiment analyses. The details in the boxes are not meant to be complete depiction of all possibilities. For example, sentiments can be extracted from a number of information sources including social interaction graphs, log files of system usage or even facial expressions and body postures. The same logic applies to the number of use contexts, which can be expanded in accordance to the needs of a trainer.

Fig. 2.
figure 2

Decisions determining the user experience

Full size image

For now we envision the potential of sentiment analyses in supporting trainers’ understanding of

  • group dynamics [23],

  • the way certain topics can impact on learners’ sentiments [24], or

  • the role of logistics and administrative elements that can impact on learners’ sentiments too [25].

5 Discussion of Scenarios

Each interview started with a short presentation of the main rational for doing a sentiment analysis, followed by the scenario and the prototypes of different visualisions, such as shown in Figs. 3 and 4. Based on these visualizations for future interfaces and an interview guideline defined the main feedback questions of the interview.

Fig. 3.
figure 3

Number of postings containing words, which determined a positive of negative sentiment

Full size image
Fig. 4.
figure 4

Words and co-words: words that are in direct relation to another word

Full size image

5.1 Usefulness of ‘Sentiment Analysis’

The overall relevance of sentiment analysis was confirmed by the interviewees. In accordance with previous studies [25, 26], the facilitators reinforce that emotions are an important part of learning, independently whether they are considered positive or negative by a system, a tutor or a peer learner. As interviewee A. puts it “when you express emotions there are greater chances of learning”. The sentiment analysis was recognized as a tool that provides facilitators with a quick overview of what is going on in the course in terms of emotions and thus helps to identify those discussions where the presence and moderation of the facilitator might be required, or become aware of individual learners who might need support in their learning.

More than that, all interviewees related sentiment analysis to the overall course evaluation, for which they think it could be a very useful element. A sentiment analysis could help the tutors gain relevant insights into the emotional stages of the course and contribute to improving future courses.

When asked whether the sentiment analysis should also be visible for learners the interviewees expressed some concerns. While none of the tutors strongly objected to the possibility of providing the information not only to tutors but also to learners, tutors were concerned that the right interpretation of the data would be lacking, leading to some wrong conclusions or actions. The interviewees agreed that this needed very careful treatment and explanations of how to interpret the data. One suggestion was that the tutor would present an overview of sentiments and decide on what to show and what to take out from the data being displayed to all participants. One tutor was especially careful with very strong words, while another tutor found some value in showing negative sentiments in order to alert the learners and maybe trigger some changing behaviour within the course.

5.2 Unit of Analysis

The presented unit of analysis is on a word and co-word level as shown in Figs. 4 and 5. There was a common agreement amongst the interviewees that a word and co-word analysis can provide a good first overview for a tutor. It provides quick and manageable insights into current emotions of the course and is therefore a suitable unit of analysis

Fig. 5.
figure 5

Percentage of positive or negative words indicating positive and negative sentiments per learning step

Full size image

When it comes to the process of defining the value of words in terms of positive and negative meaning all interviewees expressed certain reservations towards the automatic classification by sentiment dictionaries. To get this feedback, we presented the results obtained from a dictionary-based classification algorithm. For example Fig. 3 shows words used most often in expressing positive sentiments (agree, support, confidence, etc.) or negative sentiments (difficult, blocked, lack), whereas Fig. 5 shows the percentage of positive or negative words during the six course weeks. To a trainer who knows his or her course, these single words might already mean something, however, the interviewees confirmed that a further link to an exemplary set of postings expressing a sentiment or a co-word analysis [27] to contextualize the sentiment analysis is needed.

All tutors agreed that a more detailed analysis of the context in which the words were used was perceived as being indispensable for facilitating the course, as none of the interviewed facilitators would solely rely on this analysis. Thus the first level of analysis needs to provide a quick access to the deeper and specific context in which the words are used, e.g. the posting within the course. Also, all interviewees stressed that they would go back to the learner in their analysis of emotions, read through the comments shared and get their own picture of emotions within the course. Automatically created sentiment analysis does not replace this need to dive into the context.

One interviewee perceived the word and co-word analysis as sufficient, not for tutoring the course, but for certain reporting tasks that are related to the evaluation and collection of formative feedback to the course

5.3 Valuable and Less Valuable Features

All interviewees could image the sentiment analysis tool offering some sentiment tracking features, or trend spotting. They also expressed their interest in an alert function as suggested by [28] in order to be made aware in real time about issues coming up in the course. Thus, when asked about whether they preferred a push or pull mechanism for obtaining results from the system, all favored a mix of both options.

In addition, one interviewee described a scenario where he could envision using the sentiment analysis in order to regulate the visibility of comments. As the visibility of comments tends to decrease with the number of participants, an emotion alert could be used to point to specific comments dealing with emotions, complementary to the commonly used “like” function.

Other display elements voiced during the interview are: a timeline view to see if emotions change of the course; or the recognition of patterns in use of negative/positive words.

Most importantly and shortly mentioned above, when it comes to displaying the words facilitators want to be able to see the words in context. The presented co-word analysis is perceived as being a good first step, but interviewees would also like to be able to dive directly in the context of the word and see where and how it has been used. The “individual learner view”, where a user could see the emotional posting behaviour of individual learners, is rated less important by all five facilitators. They rather stress the importance of seeing an emotion expressed in context rather than assigning it to an individual learner (Fig. 6).

Fig. 6.
figure 6

Tracing back the number of positive and negative words to individual learners

Full size image

5.4 Training the Future System

A common agreement amongst the facilitators seems to be the fact that they need tools for regulating emotions [25]. However, the efforts they would put into training and adjusting such a tool differs across participants, with the answers being mainly driven by their institutional role. The idea behind this training is that sentiments of a course are once tagged manually in order to create a basis, which is then used to train the system and thus improve the validity of identified emotions in a learning intervention.

Especially the informant from the learning and development department in a very large public organization does not see a lot of value for training a sentiment analysis tool. Sentiment analysis is not considered of high importance within his specific organization and thus he states that the organization would not be willing to invest resources in such an activity. In the interviewee’s opinion the training of the system would not outweigh the costs and therefore lack cost-effectiveness. This belief is expressed by a second interviewee who confirms that the specific public organization does not want to go into learning analytics as part of its approach to improve the experiences of learners. Rather there is a stronger focus on evaluation and that’s why sentiments analysis could be seen as a tool that supports formative and summative evaluation of online courses too.

6 Outlook

The first evaluation using low fidelity prototypes based on real data provided the research team with insights into usefulness and user experience of sentiments analysis in a work place learning context. It helped to prioritize features that would support facilitators of online courses in understanding the emotional dynamics amongst their professional training participants. The limitations of this work are in the small number of tutors giving feedback to a first set of proof-of-concept prototypes. However, important aspects have been addressed. As a next step we plan to work closely with a set of tutors to build further on the prototypes and develop an interface that allows the contextual view as requested by the interviewees. After several user-development loops we will try to implement a high fidelity prototype of the sentiment monitoring tool in a real course scenario and obtain further feedback, aiming to collecting detailed experiences from this usage with regard to applicability and impact.

New features will be integrated as low-fidelity prototype in this functional prototype for further conception and initial testing. First ideas are for instance using benchmarks, and observe how comparisons between groups and individuals affect personal motivation, e.g. comparisons with group averages can be counterproductive if an individual’s set goal or previous performance had been better than the average.

7 Conclusions

Based on a set of initial decisions guiding the generation and presentation of a sentiment analysis, the paper discussed a number of options how to present the process as well as the results of such an analysis in different ways. A dialogical evaluation format is used to discuss these options under pragmatic and emotional aspects. Having this type of evaluation at a prototyping stage enables designers to prioritize developments for a final, productive system where some of these options can be combined according to trainers’ preferences.

Overall, the qualitative feedback obtained with a limited number of trainers from the public sector confirmed the interest by tutors in the sentiments of their course participants. Sentiments analysis is understood as one way to support this need, especially when it comes to courses with high numbers of participants, where the human efforts of the tutor are not enough to follow all activities. Sentiment analysis can form part of a suite of learning analytic tools that serves the tutor as a monitoring tool. It is valued as a tool that provides a quick and manageable overview of sentiments in online courses in order to detect those situations, where the input from facilitators might be needed: to moderate discussions, manage group dynamics or support individual learners. It is understood as a tool that helps to quickly dive into the context where positive or negative words are used, but cannot replace the individual tutors’ interpretation of words and expressions in context.

Learning is messy and simplistic models won’t work [8], at least not when they stand alone. They can provide first orientation but cannot replace the complex processes that facilitators apply to interpret and regulate emotions in online courses. The simplistic and automated analysis of sentiments in learning can be used as an approach to provide an overview of dynamics and patterns of emotions, as a contribution to the evaluation and further improving of training interventions. In times where resources are scarce, workplace-learning interventions are driven by the need for efficiency and investments in this type of learning analytics are small. Thus this approach has to bridge the gap between the complexity of emotions in learning and the need for easy to use and efficient support tools.

Robust learning analytics requires investments which depend on regulatory frameworks that see added value in learning analytics [29] and thus require for the inclusion of management. On the other hand, the inclusion of management might bring the risk that sentiment analysis is used not to enrich the learning experience but to assess online training interventions. The latter could lead to ‘teaching for the test versus teaching to improve understanding’ [10], which requires organizations to develop cultures and policies around the appropriate use of sentiment analyses.