Keywords

1 Introduction

Recruiting the right type of employees such as salesperson who matches the organizational needs has a critical impact on the performance and sustainability of the sales force, sales manager, and the organization as a whole [1, 2]. High turnover and poor retention of salespersons in organizations is commonly seen and has exposed the limited success of existing hiring processes [1]. Emotionally aware social robots have hardly been used in human resource management. In this paper we report the use of emotionally aware robots to facilitate hiring and benchmarking of employees vis-a-vis existing employees based on their job related emotional and cultural or cognitive behaviour fitness.

In this paper we model a sales candidate’s interview with an Emotionally Intelligent Robotic System (EIRS). EIRS is based on integration of psychology based selling behaviour model [3], artificial intelligence, image processing and soft computing techniques. Most existing approaches to salesperson’s hiring rely primarily on the interview process and/or psychometric tests for selling behaviour evaluation [3]. Some of the limitations of the interview process include human factor and subjectivity, time constraints, and lack of uniformity [4].

Unlike general behavioural profiling systems like Myers-Briggs profile based on psychometric techniques (which rely on indirect questions related to a candidate’s motivation, temperament, etc.) [5], EIRS is based on direct questions related to selling behaviour which are well understood by sales managers and sales candidates. Additionally, the indirect methods (e.g., Myer-Briggs) based on existing psychometric techniques for selling behaviour profiling has encountered with resistance from candidates as well as line/sales managers. Any measure of candidate’s emotions (using psychometric technique driven emotional intelligence tests [6]) is again derived indirectly from the questions and is compared to an absolute value or number thus limiting their applicability in terms of developing organization specific benchmarks. In this paper we are proposing a direct and independent measure of a candidate’s emotional state using physiological indicators like facial expressions.

A set of procedures are undertaken to correlate a sales candidate’s emotional profile with their selling behaviour profile (based on their cognitive responses to selling behaviour questions) computed by the EIRS. The novelty and significance of the contribution include: (i) First novel robotic system for conducting job interviews - sustainable management of employees; (ii) Embodiment of interview in emotionally aware robot results in natural social interaction; (iii) Improving information quality for decision making in real time by including verbal and non-verbal data; (iv) Customisation of interviews based on correlation of verbal and non-verbal responses; (v) Cost effective candidate filtering for large and small organisations; and (vi) Innovative fundamental shift towards robot based social innovation and design of sustainable organisations of the future.

The structure of this paper is followed by the system architecture section. Section 3 demonstrates the research outcomes, and the last section concludes this paper.

2 System Architecture

This section describes the architecture (Fig. 1) of EIRS system and technical background utilised in each system component.

Fig. 1.
figure 1

System architecture

Full size image

2.1 Interview System

The interview system is used for conducting job interview consisting of three main components: text-to-speech engine, speech recognition and dialogue system. The text-to-speech engine is responsible for vocalising the text questions to the candidate, while the speech recognition capture the audio responses of the candidate and convert them into text.

The dialogue system is responsible for managing the job interview dialogue with the candidate. Besides the human voice and speech recognition and emotion tracking capacities, the robot is interacting (interviewing) the candidate in a very interacting manner with rich human-like characteristics like gestures and emotive facial expressions.

During conducting the interview, video capturing and cognitive responses of the candidate will be input in the emotional profiling and cognitive profiling components for profiling and benchmarking.

2.2 Cognitive Behaviour Modelling

The behavioural model developed by Buzzotte, Lefton & Sherberg [7] has been used for predicting selling behaviour profiles of salespersons. The two dimensional behavioural model [3, p. 171] used for predicting four behavioural categories, namely, Dominant-Warm (DW), Dominant-Hostile (DH), Submissive-Warm (SW) and Submissive-Hostile (SH) is shown in Fig. 2. This model has been used based upon interactions with senior managers in the sales and human resources arena in the consumer and manufacturing industries in Australia [8].

Fig. 2.
figure 2

Salesperson behaviour profile [3, p. 171]

Full size image

For analysing the selling behaviour profile of a salesperson 17 areas related to the model have been identified for evaluation of a sales candidate behaviour profile as selling as a profession of assertiveness, decisiveness, prospecting, product, customers, competition, success and failure, boss, peers, rules and regulations, expenses and reports, training, job satisfaction, view about people, relationship with non-selling departments, and general attitudes [8]. These areas have been identified after several brainstorming discussions with sales managers (domain experts) and knowledge available in the literature [9]. Weights have been assigned to 17 areas on a scale of 1 to 10 using AHP (Analytical Hierarchy Process) technique [10]. The selling behaviour attributes associated with the 17 areas are designed in the form of a questionnaire. The questionnaire consists of 76 questions with at least four questions corresponding to each area. The psychological inputs or answers provided by a sales candidate are used for determining the selling behavioural profile of the candidate. EIRS combines deep knowledge in the form of a selling behavioural model (and 76 questions related to it) and shallow knowledge in the form of behaviour categorization heuristics representing knowledge and experience of the sales managers (an Expert System).

Fig. 3.
figure 3

Emotional profiling

Full size image

2.3 Emotional Modelling

Emotion Tracking

The subtle variations in facial expressions are usefully modelled by tracking the eye shape and movement, eyebrow movement, and cheek and lip movement. The facial action units associated with eyes, eyebrows, cheek and lips have been used in this research as shown in Fig. 4.

Fig. 4.
figure 4

Tracking feature points

Full size image

In order to continuously monitor the changes between emotional states, the movement of 16 facial points (Fig. 4) are tracked and classified by LVQ (Linear Vector Quantization) [11] supervised classification algorithm into positive and negative emotional sates, modelled by Affect space model [12] (Fig. 5). The reason for selecting LVQ is that it can give a nonlinear separation of the sample space. This property of the LVQ algorithm becomes useful when dealing with complex class domains where a linear decision border is not sufficient (Fig. 3).

Fig. 5.
figure 5

Affect space model with positive and negative emotional

Full size image

Emotion Interpretation

The emotional responses of a sales candidate are interpreted by four emotion variables (intensity, duration, frequency and shape). Emotion intensity measures the strength of affect [13], and can be defined as the relative degree of displacement away from a neutral expression, of the pattern of muscle movements involved in emotional expressions of a given sort [14]. In our case, this degree of displacement corresponds to the displacement of tracking feature points. The displacements of these tracking points are used as input vector for Fuzzy inference to infer the emotion intensity. In this work, the intensity is divided by 3 degree (high, med and low). We select the sigmoidal function \( f(x) = \frac{1}{{1 + {e^{ - a(x - c)}}}} \) as the membership function for intensity (Fig. 6).

Fig. 6.
figure 6

Intensity membership function

Full size image

The duration of emotion measures how long the emotion lasts and is recorded as the time when the emotion occurs (change from neutral to + iv/− iv) to the emotion disappears (change from + iv/− iv to neutral). To capture the duration of emotion, the locations of the tracking feature points at the frame before the emotion occurs (corresponding to neutral state) are stored as reference locations to measure feature movement for emotion tracking until the emotion ends (back to neutral state). The emotion frequency defines the number of emotion changes over a period of time [13]. Higher frequency of positive valence shows of acceptance and acknowledgement as well as readiness to take remedial action.

Emotion Profiling

The system attempts to make use of the candidate’s emotional state to determine the correlation or commitment the candidate has to the entered response. Rather than attempting to determine the absolute emotional state of a person or sales candidate i.e., exactly where their emotional state lies in the affect space model, we have modelled a change in emotional state in either a positive direction or a negative direction. It is proposed that a positive change in emotional state of the candidate that coincides with the answering of a question indicates a candidate’s higher commitment to the answer given. Conversely a negative emotional state change indicates a reduced commitment of the candidate to the answer given. The psychologists point out that facial expression alone may not be an accurate indicator of the emotional state of a person but changes in facial expression may indicate a change in emotional state [11, 15]. In our case, the transient or temporal changes in facial expressions and emotional states as the sales candidate answers different questions are modelled.

2.4 Correlations and Benchmarking

Figure 7 is a graph of the candidate’s answers to questions in the area of success and failure, affirmative and negative for each of the behaviour categories. The graph also shows his emotional state as detected by this system on the same axis. The continuous line represents answers and the dashed line is emotional state. There is a divergence with the question relating to the dominant-warm, (DW) behaviour category. This correlation indicates a conflict with the answer given by the candidates in the indicated area/behaviour category. This should translate, in practice, to a follow up in an interview if this system were to be used in a hiring situation.

Fig. 7.
figure 7

Correlation of emotional profile and selling behaviour profile

Full size image

For behavioural benchmarking, an existing sales person’s behavioural category as predicted by the system is compared against sales manager’s categorization of the salesperson based on their experience of working with them.

3 Implementation and Results

The SRBS system has been implemented for conducting the job interview with salespersons. One of our social robots (i.e., Jack) was employed to conduct the interview with sales candidates (Fig. 8). During the interviews, the emotional responses of the candidates are tracked and profiled by the emotional component. Meanwhile, the cognitive responses are captured by the interview system are input into the cognitive profiling component.

Fig. 8.
figure 8

The robot is conducting job interview with a candidate (left), and his emotional responses are profiling (right).

Full size image
Fig. 9.
figure 9

Candidate result screen

Full size image

SRBS combines deep knowledge in the form of a selling behavioural model (and 76 questions related to it) and shallow knowledge in the form of behaviour categorization heuristics representing knowledge and experience of the sales managers (an Expert System). There are overall 400 rules in the system.

The pie chart in Fig. 9 represents the overall distribution of four category scores. That is, the upper right hand corner of the figure shows the area wise breakup of a candidate’s selling behaviour as related to the Dominant-Hostile (DH) category.

The field tests compared an existing salesperson’s behavioural category as predicted by the system against sales manager’s categorization of the salesperson based on their experience of working with them. In Fig. 10, we show a comparison of the candidate’s profile (one with low dominant hostile score) with the benchmark profile (one with high dominant hostile score) of a particular organization.

Fig. 10.
figure 10

Benchmarking based on cognitive responses

Full size image
Fig. 11.
figure 11

Benchmarking based on emotional responses

Full size image

The hiring manager is particularly interested in the orientation of the two profiles. That is, are the two profiles parallel or do they cross each other. They are less interested in the magnitude of difference between the two profiles (which if required can be deciphered from the Y coordinate dimension of the comparison of profile graph).

The system the benchmarking of emotional and cognitive behaviour profiles of the candidate against the according selected benchmarking profiles (as shown in Figs. 10 and 11). The cognitive and emotional profiles of all candidates are also compared (Fig. 12).

Fig. 12.
figure 12

Comparison of cognitive and emotion response between all candidates

Full size image

These benchmarking data together with the reports can be used for conducting customized interviews and probing the candidate in areas or competencies where the emotional profile and behaviour profile based on cognitive inputs are not aligned with each other or are fairly divergent from each other.

4 Conclusion

This paper have presented human-robot interaction modelling for recruitment (named EIRS) involving verbal and non-verbal data modelling and benchmarking for improving information quality as well as quality of decision making. EIRS supports employers to analyse data about their employees through: (i) organization specific benchmarks by using emotional and cognitive behaviour profiles of job applicants with those of existing team of employees; (ii) creating an organisation wide repository of job related emotional and cognitive behaviour profiles based on verbal and non-verbal data for emotionally intelligent management and deployment of employees in various roles and teams; (iii) customisation of face to face interviews based on high information quality and profiles to save time and resources; and (iv) employing social robot for natural social interaction between employees and information technology compared to existing screen based technologies, and (v) developing a range of other root enabled personalised services based on verbal and non-verbal data for improving employee motivation, training and commitment.