Affective Recognition Using EEG Signal in Human-Robot Interaction

Abstract

Human-robot interaction is a crucial field in human factor field and mechanical arm operation is a widely used form in human-robot interaction. However, the mistaken operations caused by the affect influction of operators are still one of the dominant reasons causing accidents. Because of the close link between affective state and human error, in this paper, we analyzed the EEG signal of five subjects operating mechanical arm and the track record of the mechanical arm movement. A combination label model including the subjective part and the objective part are proposed to reflect the real time affective state influction. Additionally, in subsequent recognition experiment, the results indicate that the affect is a state of mind that requires a relatively longer period of time to be effectively represented and the frequency domain features are significantly more important than time domain features in affective recognition process using EEG signal.

1 Introduction

Hitherto, mechanical arm, as a vital product in industry field, has been broadly used in medical, exploration, rescue field etc. However, the mistaken operation caused by human error, which should have been avoided, is still one of the dominant reasons causing accidents. As we all know, the performance of human has a close link to the cognitive state of human and to some degree affect is the main reason causing the change of the cognitive state. Therefore, one of the critical ways to avoid human error in mechanical arm operation is recognizing the affect during the operation process through detecting the physiological signal. Since R. W. Picard has defined $Affective\ Computing\ (AC)$ [1] in 1995, affective computing has been a critical field in human-computer interaction area. There are numerous physiological signals which could reflect the cognitive state of human, such as Blood Volume Pressure (BVP), Skin Conductance Response (SCR), Respiration (RESP), Electrocardiogram (ECG), Electromyogram (EMG), Electrocorticogram (ECoG), Electroencephalogram (EEG), Heart Rate (HR), Oxygen Saturation (SaO2) and Surface Temperature (ST) [2]. In all physiological signals, EEG is no doubt the most capable signal directly reflecting the brain activity. Therefore this paper chooses to use 32 dry electrodes EEG signal acquisition equipment to obtain EEG signal data during mechanical arm operation.

There are lots of models which have been proposed to describe the affect, such as six basic emotions model proposed by Ekman et al. [3], eight basic emotions model proposed by Plutchik [4] and the valence-arousal scale proposed by Russell [5]. For simplifying this problem, this paper chooses to use the valence in the valence-arousal model of Russell to evaluate the emotion of the subjects. And the valence reflecting the positive or negative aspect of the subjects is enough to describe the cognitive state of the subjects.

Besides the affective model, the way to obtain the ground truth of the subjects cognitive state is also crucial. Almost all research in Affective Computing field use the self-assessment scores to estimate the true cognitive state of the subjects. However, even the subjects themselves could hardly to retell the exact affective state in the mechanical arm operation and using one single scores to estimate the cognitive state during the entire operation process is obviously not reasonable in detail. So this paper proposes to use objective and real time indexes to represent the cognitive state of the subjects. In this paper, we record the track of the end point of the mechanical arm and extract the features of the track to represent the fluctuation of the cognitive state of the subjects. Meanwhile, we assume that the workload and the time pressure could stimulate the affective change of the subjects, so we give a basic score, which reflects the affective state the subjects should be, and add the weighted track features scores to the basic score to reflect the fluctuation of the affective state.

In our experiment, we designed three levels of operating tasks in different difficulty to stimulate the affective state change. And the level of difficulty is determined by the workload and the time pressure. To eliminate the influence of unfamiliarity, one minute of free exploration is added before these three tasks and to eliminate the interaction of different tasks, a 30 s reset time interval is added between the different tasks.

During the data process part, because of the low signal-to-noise ratio (SNR), the disturbance of EMG signal and the electromagnetic interference, the raw data have firstly been filter to the 1–64 Hz frequency band [6]. After normalization process, different scales sliding window are induced in extracting features. Because of the real-time label we obtain from the track mentioned above, it allows us to consider the data in single sliding window as one sample.

The feature extraction methods are detailed summarized in paper [6], we choose three time domain features and one frequency feature to represent the raw data according to the value of the weighted relative occurrence. And at the feature selection process, we apply Principal Component Analysis (PCA) to select the extracted features above. And at the classification design process, we apply Support Vector Machine (SVM), which is an effective classification discriminator, to predict the affective state.

Here is the reminder organization of this paper: Sect. 2 introduce the apparatus used in the experiment and the detail experiment protocol. Section 3 describes the data preprocess procedure, including EEG data preprocess and real time label data synthesis. Section 4 focuses on affective recognition methods, including multi-scale feature extraction, feature selection and classification design. Section 5 focus on the results of the experiment and the analysis of these results. Section 6 summarizes the conclusion of this paper.

2 Experiment Setup

2.1 Apparatus

There are 3 key equipments we use in our experiment: EEG Signal Acquisition Equipment, Mechanical Arm and Joystick. And there are 3 personal computers for communicating with EEG equipment, controling movement of mechanical arm through joystick and showing the end point of the mechanical arm.

EEG Signal Acquisition Equipment: The EEG Signal Acquisition Equipment we apply is the Cognionics HD-72 Dry EEG Headset [7] (see in Fig. 1).

Fig. 1.

Cognionics HD-72 Dry EEG Headset

Considering the difficulty of wearing EEG equipment and the comfort of the subjects, we choose 32 dry electrodes, according to the international 10–20 system, to obtain the EEG signal, which are showed in Fig. 2. The sampling rate is 500 Hz which is sufficient for obtaining EEG signal.

Fig. 2.

32 dry electrodes sensor location

Mechanical Arm: The mechanical arm we apply is the Dobot Magician mechanical arm [8] (see in Fig. 3), because it supports reprogramming according to the need of users and the control precision (0.2 mm) is adequate for our experiment.

Fig. 3.

Dobot Magician mechanical arm

Joystick: The joystick we use is the flying joystick named Extreme 3D Pro [9] produced by Logitech (see in Fig. 4). The perfect ergonomic design with a custom twist-handle rudder relies its one-handed control resulting in a smaller device footprint. There are six programmable buttons on the base. Each programmable button can be configured to execute simple single commands or intricate macros involving multiple keystrokes, mouse events, and more. In our experiment, we only operate the rocker in six basic direction movement, which are left-right direction, front-behind direction and left-right rotation.

Fig. 4.

Logitech Extreme 3D Pro joystick

In our experiment, we use Robot Operating System (ROS) to receive the control signal from joystick, release the control signal to mechanical arm, receive the position information of the end point of the mechanical arm and record the track information with real time mark point in a sampling rate of 10 Hz.¹ Meanwhile, real time EEG signal are recorded with mark information which could be used in subsequent time correcting process.

2.2 Experiment Protocol

Five healthy participants (two females, three males), aged between 23 and 25, participated in the experiment. Before all experiments, all participants were asked to have adequate sleepness.

Fig. 5.

The experiment environment

In our experiment, firstly, participants were asked to read the experiment procedures and notes and the experimenter would read out the procedures and notes for a second reminder. And the experimenter was also present there to answer any questions. At the beginning of each experiment, one minute of free exploration were given to remove the interference caused by unfamiliarity. And then the subjects were asked to operate the mechanical arm to touch the different color point on the desktop according to a certain order. We designed three levels of operating tasks in different difficulty (easy, medium and hard mode) according to the number and position of points and the time constraint. In easy mode, the subjects were asked to touch three colored points without time pressure. In medium mode, the subjects were asked to touch five colored points without time pressure. In hard mode, the subjects were asked to touch five colored points in 90 s. To eliminate the interplay between the three tasks, a 30 s break time for resetting was added after each task.

The experiment enviroment is showed in Fig. 5. The Fig. 5(a) is the overview of the entire experiment environment. The subject operation platform (see in Fig. 5(b)) is insulated from the mechanical arm platform and all the information helping the subjects to move the mechanical arm was from three camera set around the mechanical arm and on the end point of the mechanical arm (see in Fig. 5(c)). The screen interface presenting the information from cameras are showed in Fig. 5(d).

3 Data Preprocess Methods

In data science field, data preprocess is the key procedure before any data analysis procedure. The quality of data preprocess directly affects the final accuracy of recognition. In EEG experiment, because of the low signal noise rate (SNR) and various interference, filtering and amplifying EEG signal is a crucial step before EEG data analysis. And in our experiment, we need to do some label data preprocess to obtain more objective real time label of EEG data because of the use of sliding window which would be mentioned in Sect. 4 and the need of combining self-assessment with objective data.

Fig. 6.

Data preprocess wave charts

3.1 EEG Data Preprocess

For one subject experiment, the raw data are drawed in Fig. 6(a). And then according to the track mark and EEG signal mark, the EEG signal and the track were aligned in time. And we divided the entire EEG signal to easy mode, medium mode and hard mode procedure according to the mark information. The divided easy mode raw data, as an example, are showed in Fig. 6(b). Next, to remove the disturbance of EMG signal and the interference of various electromagnetic signal which are generally high frequecy signals, we use band filter to filter out 1–64 Hz signal, which are the dominant frequency band of EEG signal (see in Fig. 6(c)). Because the EEG signal has extra low SNR, the filtered signal at the beginning and end part is unstable. Therefore we selected to remove the first and last 5 s data (see in Fig. 6(d)). And for the computing convenience, the selected data are normalized to [$-1$,1] area according to all 32 channels signal by min-max normalization.

3.2 Real Time Label Data Synthesis

Label data includes three parts in total: the self-assessment of subjects, the difficulty level experimenters design and the performance the track reflects. The self-assessment is the subjective label data and the difficulty level and the performance are the objective data.

For the self-assessment, after the entire experiment were finished, each subjects were asked to fill a table about their affective state during operating different tasks. There were seven level the subjects could choose. “1” means the most positive affective state, “4” means neutral affective state and “7” means the most negative affective state. The greater the number, the worse the affective state. And then we used each score to weighted each task.

For the the difficulty level experimenters design, we simply gave the scores of difficulty according to the workload and the time pressure. In easy task, we asked the subjects to touch three points without time constraint, which means that we assumed the most positive affective state could be stimulated by this task. On the contrary, in hard task, we asked the subjects to touch five points with a ninety seconds constraint, which means that we assumed the most negative affective state could be stimulated by this task. We used “2.5”, “4” and “5.5” to represent the affective states that different difficulty levels could stimulate in our assumption.

For the performance, we recorded the tip trajectory of mechanical arm. As showed in Fig. 7, in easy mode, the subjects were asked to touch orange point, purple point and pink point in order from a random start point. We assumed that the subjects would feel positive when they moved the mechanical arm smoothly, so we counted the numbers of direction change, in selected sliding window mentioned in Sect. 4, as the estimation of the affective state of the subjects.

Fig. 7.

The tip trajectory of mechanical arm in easy mode

We use the formula below to calculate the final label.

$$\begin{aligned} L = [\ \frac{1}{2}L_{self-assessment} + \frac{1}{2}(L_{difficulty-level} + L_{direction-change}) + \frac{1}{2}\ ] \end{aligned}$$

(1)

where [x] means the largest integer not exceeding x. And in this equation $L_{self-assessment}$ means the self-assessment score, $L_{difficulty-level}$ means the difficulty level score, and $L_{direction-change}$ determined by equation below:

$$\begin{aligned} L_{direction-change} = 3\frac{N_{direction-change} - N_{min}}{N_{max} - N_{min}} - 1.5 \end{aligned}$$

(2)

where $L_{direction-change}$ means the direction change score, among them, $N_{direction-change}$ means the number of direction changes, $N_{min}$ means the minium of $N_{direction-change}$ and $N_{max}$ means the maxium of $N_{direction-change}$.

4 Affective Recognition Methods

Traditional pattern recognition process includes 4 steps: data preprocess, feature extraction, feature selection and classification design. In this paper, we use several classic methods in feature extraction, feature selection and classification design procedure to explore their performance in our experiment data.

4.1 Multi-scale Feature Extration

The multi-scale feature extraction procedure are divided in two steps: (1) sliding window selection and (2) feature extraction.

Sliding Window Selection

Because uncertain relation between time interval and affective state change, we chose every 2 s between 4s to 30 s as the length of sliding window and 0.2 s as the length of stride to extract multi-scale features.

Feature Extraction

According to the paper [6], we chose 18 dimensions time domain features, which are statistical features, higher order crossings (HOC) features and fractal dimension feature, and 53 dimensions frequency domain features, which are band power, bin power and the ratio of mean band powers $\beta /\alpha$.

Time Domain Features

In this paper, we selected 3 classic time domain features in recognition process: statistical feature, higher order crossing (HOC) features and fractal dimension (FD) feature.

There are seven statistical features representing the raw EEG time series. These are:

• (a) Mean: ${\mu }_{\varvec{\eta }} = \frac{1}{T}\sum _{t=1}^T \varvec{\eta }(t)$

• (b) Power: $P_{\varvec{\eta }} = \frac{1}{T}\sum _{-\infty }^{\infty } |{\varvec{\eta }}(t)|^2$

• (c) Standard deviation: $\sigma _{\varvec{\eta }} = \sqrt{ \frac{1}{T} \sum _{t=1}^{T-1} ({\varvec{\eta }}(t) - {\mu }_{\varvec{\eta }})^2 }$

• (d) 1st difference: $\delta _{\varvec{\eta }} = \frac{1}{T-1}\sum _{t=1}^{T-1}|{\varvec{\eta }}(t+1) - {\varvec{\eta }}(t)|$

• (e) Normalized 1st difference: $\bar{\delta _{\varvec{\eta }}} = \frac{ \delta _{\varvec{\eta }}}{\sigma _{\varvec{\eta }}}$

• (f) 2nd difference: $\gamma _{\varvec{\eta }} = \frac{1}{T-2}\sum _{t=1}^{T-2}|{\varvec{\eta }}(t+2) - {\varvec{\eta }}(t)|$

• (g) Normalized 2nd difference: $\bar{\gamma _{\varvec{\eta }}} = \frac{\gamma _{\varvec{\eta }}}{\sigma _{\varvec{\eta }}}$

To find more robust features and pattern of EEG, Petrantonakis and Hadjileontiadis proposed higher order crosssings (HOC) features [10]. Therefore we applied HOC to represent the raw EEG time series. The higher order series could be calculated by the formula below:

$$\begin{aligned} H_k\{{\varvec{\eta }}(t)\} = \nabla ^{k-1}{\varvec{\eta }}(t) \end{aligned}$$

(3)

where $\nabla$ means ${\varvec{\eta }}(t) - {\varvec{\eta }}(t-1)$. Therefore, the features of HOC could be caculated by counting the sign changes or equation below:

$$\begin{aligned} F_k = \sum _{t=1}^{T-k}\psi (H_k\{{\varvec{\eta }}(t)\}H_k\{{\varvec{\eta }}(t+1)\}), k = 1,2...10 \end{aligned}$$

(4)

where $\psi (x)$ is a section function which is defined below:

$$\begin{aligned} \psi (x)= {\left\{ \begin{array}{ll} 0 &{}\text {if x } \ge 0\\ 1 &{}\text {if x } < 0 \end{array}\right. } \end{aligned}$$

(5)

The fractal dimention (FD) as a feature measuring the complexity is widely used. There are many methods computing the FD feature. In this paper, we chose to use the Higuchi algorithm [11] to caculate the FD feature. To compute the FD feature, the EEG series is rewritten as:

$$\begin{aligned} \{{\varvec{\eta }}(p), {\varvec{\eta }}(p+q), {\varvec{\eta }}(p+2q), ... , {\varvec{\eta }}(q+[\frac{T-q}{q}]q)\},\ p = 1,2,...,q \end{aligned}$$

(6)

where [x] means the largest integer less than x. Then we could define the series $M_p(q)$ as below:

$$\begin{aligned} M_p(q) = \frac{T-1}{[\frac{T-p}{q}]q^2}\sum _{k=1}^{[\frac{T-p}{q}]}|{\varvec{\eta }}(p+kq) - {\varvec{\eta }}(p+(k-1)q)| \end{aligned}$$

(7)

Therefore we could further define the average:

$$\begin{aligned} F(q) = \frac{1}{N_p}\sum _{p=1}^{N_p}M_p(q) \end{aligned}$$

(8)

where $N_p$ means the maxium of p. According to paper [11], we knew that F(p) is proportional to $p^{-F_{FD}}$ (where $F_{FD}$ means the feature of FD). Therefore we could assumed:

$$\begin{aligned} F(q) = r\cdot {p^{-F_{FD}}} \end{aligned}$$

(9)

when we log the equation, we could obtain:

$$\begin{aligned} log(F(q)) = log(r) - {F_{FD}}log(p) \end{aligned}$$

(10)

Therefore we could obtain FD feature $F_{FD}$ by the slope of log(F(q)) to log(p).

Frequency Domain Features

For time series signal, spectrum analysis is an important method extracting features. Through short-time fourier transform (STFT) [12], which is a more robust method, we could get the power value as a function of time and frequecy: p(t, f). Further, the relation of the power and the frequecy could be written as below:

$$\begin{aligned} P(f) = \sqrt{\sum _{t=0}^{T}[p(t,f)]^2} \end{aligned}$$

(11)

As we know, effective EEG signals exists in low frequency band. Further, this valid low frequecy band could be divided to five smaller parts: $\delta$ (1–4 Hz) band, $\theta$ (4–8 Hz) band, $\alpha$ (8–12 Hz) band, $\beta$ (12–30 Hz) band, $\gamma$ (30–64 Hz) band. Therefore, for each band on [$f_a$,$f_b$], STFT band power could be caculated by the formula below. Additionally, the ratio of mean band powers $\frac{\beta }{\alpha }$ was computed.

• (a) Mean: $\mu _P = \frac{1}{f_b-f_a}\sum _{f=f_a}^{f_b}P(f)$

• (b) Minium: $P_{max} = max(P(f)), f\in [f_a,f_b]$

• (c) Maxium: $P_{min} = min(P(f)), f\in [f_a,f_b]$

• (d) Variance: $var(P) = \frac{1}{f_b-f_a}\sum _{f=f_a}^{f_b}|P(f)-\mu _p|^2$

Similarly, we could divide the frequency band in a higher resolution to obtain STFT bin power features. We divided the 1–64 Hz band into 32 subband, which means 32 ($\varDelta f = 2\,Hz$) bands are extracted for further processing. And then, for each $\varDelta f$ band, we could caculate the STFT bin power feature by the formula above.

As showed in Fig. 8, We concatenate the time domain features and the frequency domain features as the representation of original raw data. Therefore, the features number of one channel of one sample is 71.

Fig. 8.

Feature vector composition

4.2 Feature Selection

Principal component analysis (PCA), proposed by Pearson [13], is a effective and classic method to reduce dimensions of original matrix through keeping the lower order principal components. Therefore we use PCA to select the original features.

After the feature extraction steps, we could concatenate 32 channels feature to a long features vector and use all samples to constitute a feature matrix $\varvec{M} = (\varvec{\phi _1}, \varvec{\phi _2},..., \varvec{\phi _N})$, where $\phi _i$ means one sample features vector, N means the number of samples.

Then we use the samples to estimate the mean and the covariance matrix:

$$\begin{aligned} \bar{\varvec{\phi } }= \frac{1}{N}\sum _{i=1}^{N}\varvec{\phi _i} \end{aligned}$$

(12)

$$\begin{aligned} \varvec{\varSigma } = \frac{1}{N}\sum _{i=1}^{N}(\varvec{\phi _i}-\bar{\varvec{\phi }})(\varvec{\phi _i}-\bar{\varvec{\phi }})^T = \frac{1}{N}\varvec{X}{\varvec{X}}^T \end{aligned}$$

(13)

Then we could caculate the eigenvectors ($\varvec{\mu _i}$) and the eigenvalue ($\lambda _i$) of $\varSigma$. The value $\lambda _i$ means the importance of the corresponding eigenvector $\varvec{\mu _i}$. We sorted these eigenvalues in descending order to obtain the descending eigenvalue series: $\lambda _1^\prime$, $\lambda _2^\prime$,..., $\lambda _i^\prime$,..., $\lambda _L^\prime$, where L is the number of features. Correspondingly, the eigenvectors could be rearranged: $\varvec{\mu _1^\prime }$, $\varvec{\mu _2^\prime }$,..., $\varvec{\mu _i^\prime }$,..., $\varvec{\mu _L^\prime }$. Then we define the information integrity I by the formula below:

$$\begin{aligned} I = \frac{\sum _{i=1}^{k}\lambda _i^\prime }{\sum _{i=1}^{L}\lambda _i^\prime } \end{aligned}$$

(14)

In this paper, we chose

$$\begin{aligned} K = \mathop {\arg \max }_{k} \ I(k)>0.99 \end{aligned}$$

(15)

Then K is the feature dimension of feature matrix after dimensionality reduction. And the mapping matrix is $M_{PCA} = (\varvec{\mu _1^\prime }, \varvec{\mu _2^\prime },..., \varvec{\mu _K^\prime })$.

4.3 Classification

Support vector machine (SVM) is a effective classification proposed by Cortes and Vapnik [14]. Because the advantages of SVM in non-linear and high-dimensional pattern recognition, we applied it in our experiment as the classification.

When we classify data, each sample data is composed of a feature vector and a label value: $D_i = (\varvec{x_i}, y_i)$, where $\varvec{x_i}$ is the feature vector in high dimension, $y_i$ is the category the sample belongs. In particular, the classification problem of two category is taken as an example, and the multi-classification problem could be solved as the combination of multiple two classification problems. Specially, in two classification problem, we define the distance between one sample and a hyperplane is $\eta _i$, and $\eta _i = y_i(\varvec{\omega x_i + b})$. And to normalize the distance, $\varvec{\omega }$ and b are replaced by $\frac{\varvec{\omega }}{\Vert \varvec{\omega \Vert }}$ and $\frac{b}{\Vert \varvec{\omega \Vert }}$. The normalized distance could be rewritten as

$$\begin{aligned} \delta _i = \frac{|g(\varvec{x_i})|}{\Vert \varvec{\omega }\Vert } \end{aligned}$$

(16)

where $g(\varvec{x_i})=\varvec{\omega }\varvec{x_i}+b$. Because the relation between $\delta$ and misclassification number N is

$$\begin{aligned} N \le {(\frac{2R}{\delta })}^2 \end{aligned}$$

(17)

where $R=max\Vert \varvec{x_i}\Vert , i=1,2,...,n$, the larger the distance $\delta$ is, the smaller the N is. Therefore, the optimizing equation is

$$\begin{aligned} \begin{aligned}&\max \,\, \delta \\&s.t.\quad y_i(\varvec{\omega }\varvec{x_i}+b)-1 \ge 0,i=1,2,...,n \end{aligned} \end{aligned}$$

(18)

Further, according to Eq. 16, the optimizing target could be rewritten as below:

$$\begin{aligned} \begin{aligned}&\min \,\, \frac{1}{2}{\Vert \varvec{\omega }\Vert }^2\\&s.t.\quad y_i(\varvec{\omega }\varvec{x_i}+b)-1 \ge 0,i=1,2,...,n \end{aligned} \end{aligned}$$

(19)

Because $\varvec{\omega }$ determined by sample data, $\varvec{\omega }$ could be assumed as:

$$\begin{aligned} \varvec{\omega } = \sum _{i=1}^{n}a_iy_i\varvec{x_i}^T \end{aligned}$$

(20)

$a_i$ would be unequal to zero only when the sample is on the closest hyperplanes. In other word, hyperplanes are supportedfrac12 by the sample points which are close to these hyperplanes. Therefore, discriminant function could be written as:

$$\begin{aligned} g(\varvec{x}) = \varvec{\omega }\varvec{x}+b=\sum _{i=1}^{n}a_iy_i\varvec{x_i}^T\varvec{x}+b \end{aligned}$$

(21)

Since the samples on hyperplanes are known, b could be caculated by $y_i(\varvec{\omega }\varvec{x_i}+b)-1=0$ When test sample $\varvec{x}_{test}$ need to be classify, the value of $g(\varvec{x}_{test})$ decides the category $\varvec{x}_{test}$ belongs.

5 Results and Discussion

In our recognition experiment, we applied the ten-folder cross-validation method to test the accuracy robustness of the recognition algorithm. We randomly selected 10% original data as the test data, and chose the rest as the train data, and this process has been done 10 times for each specific parameter pair to eliminate the accidental errors.

5.1 Real Time Label Data Synthesis

As showed in Fig. 9, an example of the self-assessment score, the difficulty level score and the performance score was depicted. The trend of the difficulty level score and the self-assessment is basically the same.

Fig. 9.

Real time label data synthesis

After adding the performance score, as we assumed that the affective state is unstable even in the same task, there are some real time influction which meets the need for obtaining real time label data to study the influence for recognition accuracy when the multi-scale sliding window were used.

5.2 Multi-scale Sliding Window

In our recognition experiment, multi-scale sliding windows were applied into extracting features. Because the detecting window of the performance feature extraction in real time label data synthesis is 2 s long, the scale of the sliding window should be longer than 2 s. Therefore, we chose every 2 s time interval from 4 s to 30 s to explore the relation between the accuracy and the length of the sliding window.

Fig. 10.

Recognition accuracy about multi-scale sliding window and feature selection

As showed in Fig. 10, the recognition accuracy tends to increase slightly as a whole when the sliding window grows longer. Specifically, when the length of the sliding window is 4 s, the recognition accuracy using frequency domain features is about 65%, which is obiviously lower than the other length of sliding window. And when the length of selected window is longer than 20 s, the recognition accuracy using frequency domain features is almost stable at around 80%. This phenomena indicates that the affective state is a more stable state of mind that requires a longer period of time data to represent.

5.3 Feature Selection

As showed in Fig. 10, when we fix the parameter of the sliding window length, the accuracy is highest in using frequency domain feature. And the accuracy is about the same in using time domain feature and the combination of time and frequency feature. This phenomena indicates that PCA could not select the more important feature componets beacuse PCA reduces the dimension of feature map just by using the information of covariance. Therefore, the PCA which could barely use the label information is unsuitable for EEG signal process. And the higher recognition accuracy also indicates that frequency domain features are more important for affective recognition by using EEG signal.

6 Conclusion

In this paper, we combine the objective evaluation to the self-assessment to obtain real time label of the affective state. The result of combination reflects the influction of affective state. And the recognition experiment indicates that the affect is a state of mind that requires a longer period of time to be effectively characterized and recognized. In subsequent affective recognition experiments, the results shows that relatively longer EEG data is more appropriate for affective recognition. Meanwhile, the recognition experiment also illustrates that the frequency domain features are significantly more important than the time domain features. In future EEG analysis work, relatively long frequency domain features might be a more preferred option.