Wink based facial expression classification using machine learning approach
- 200 Downloads
Facial expression may establish communication between physically disabled people and assistive devices. Different types of facial expression including eye wink, smile, eye blink, looking up and looking down can be extracted from the brain signal. In this study, the possibility of controlling assistive devices using the individual’s wink has been investigated. Brain signals from the five subjects have been captured to recognize the left wink, right wink, and no wink. The brain signals have been captured using Emotiv Insight which consists of five channels. Fast Fourier transform and the sample range have been computed to extract the features. The extracted features have been classified with the help of different machine learning algorithms. Here, support vector machine (SVM), linear discriminant analysis (LDA) and K-nearest neighbor (K-NN) have been employed to classify the features sets. The performance of the classifier in terms of accuracy, confusion matrix, true positive and false positive rate and the area under curve (AUC)—receiver operating characteristics (ROC) have been evaluated. In the case of sample range, the highest training and testing accuracies are 98.9% and 96.7% respectively which have been achieved by two classifiers namely, SVM and K-NN. The achieved results indicate that the person’s wink can be utilized in controlling assistive devices.
KeywordsWink Facial expression Electroencephalography (EEG) Brain-computer interface (BCI) Machine learning
Plenty of neurological diseases including brainstem stroke, spinal cord injury, multiple sclerosis, cerebral palsy, muscular dystrophies and amyotrophic lateral sclerosis (ALS) may impair the person’s regular communication pathways . If individuals are drastically affected by these neural disorders, they may partially or completely lose their voluntary muscle control. In such situations, the subject is incapable of interacting with their surroundings in any other form of communication. To address this issue, researchers are trying to invent a variety of assistive technologies. Among these assistive technologies, the brain-computer interface (BCI) concept is widely investigating by the researchers.
In a BCI system, specific patterns of brain activity are translated into control commands in the purpose of particular devices operation . Mind-controlled wheelchair , home appliances , prosthetic arm controlling , spelling system , emotion detection system  and biometrics  are the popular BCI applications . Currently, BCI applications have been widened from medical to non-medical fields, for example, BCI based games and virtual reality . Both non-invasive and invasive brain activity recording modalities are contributing progressively in neuroscience research as well as in brain-computer interfacing . There are two invasive strategies are employed in BCI research including electrocorticography (ECoG) and intracortical neuron recording. The broadly used non-invasive modalities are Electroencephalography (EEG), Magnetoencephalography (MEG), Near-Infrared Spectroscopy (NIRS) and Functional Magnetic Resonance Imaging (fMRI). The majority of BCI studies have been utilized brain waves based on EEG recording. There are some positive aspects in preference of EEG including low-cost data capturing devices, ease of mobility and non-invasive manner of data acquisition. However, the Signal-to-Noise Ratio (SNR) of EEG does not always meet the satisfactory level. Moreover, the algorithm for EEG analysis in certain cases lessens the classification accuracy as well as the data transfer rate.
The brain activity due to the changes in facial expression can be used either separately or combined with EEG in the purpose of BCI applications. Raheel et al.  have been classified five facial expressions using brain activity and the classified facial expressions are smile, wink, looking up, looking down the eye. The brain activity has been captured through a 14-channel Emotiv EPOC EEG headset. Authors have extracted thirteen statistical features and these features have been classified using K-NN, Naive Bayes, SVM, and multi-layer perceptron. The highest classification accuracy of 81.60% has been achieved using K-NN which seems to poor. Ma et al.  proposed a hybrid approach consists of Electrooculogram (EOG) and event-related potential (ERP) for robot control. From EOG signals, Wink, eye blink, frown, and gaze have been detected whereas P300, N170 and VPP were evoked from ERP. Double and triple blink was used to stop and move (forward) the robot respectively, whereas the looking left and looking right were utilized to move the robot in the left and right direction respectively. Finally, the frown was used to stop and enter in ERP mode. Reyes et al.  investigated the possibility of using brain activity (generated by facial gestures and eye movements) in the purpose of controlling assistive devices.
In this study, the brain activity from different facial expressions including left wink, right wink, and no wink have been classified. The brain wave regarding the winks has been captured using the fives channel Emotiv Insight EEG headset. Emotiv Insight has also been employed in other studies [14, 15, 16, 17, 18, 19] to capture the brain wave. The remaining part of this paper has been organized in the following sections i.e. Sects. 2 and 3 discusses issues related to methodology, results and discussion respectively; finally, Sect. 4 deals with the conclusion.
The human brain is the control unit of the whole body. Hence, any change in facial expression can be recorded in the brain. The brain wave due to the left wink, right wink, and no wink has been analyzed in the current study. The brain waves have been captured from five different subjects using Emotiv Insight. Two different features have been extracted from the collected dataset and the selected features have been classified using machine learning techniques. The complete methodology of the proposed approach has been discussed in this section.
2.1 Experimental design for data acquisition
Experiment details of data acquisition
Number of trial (per class)
9.00 AM–9.30 AM
9.00 AM–9.30 AM
11.00 AM–11.30 AM
10.00 AM–10.30 AM
9.30 AM–10.00 AM
9.30 AM–0.00 AM
11.00 AM–11.30 AM
11.00 AM–11.30 AM
2.2 Data analysis framework
2.2.1 Fast fourier transform
2.2.2 Sample range
2.2.3 Linear discriminant analysis
LDA is employed to find the linear combinations of feature vectors that describe the characteristics of the corresponding signal. It utilizes hyperplanes to separate two or more classes. The isolating hyperplane is achieved by searching the projection which maximizes the distance among the classes’ means and minimizes the interclass variance [23, 24]. This technique has a very low computational requirement and it is simple to use. The LDA has been successfully applied in a variety of BCI systems.
2.2.4 K-nearest neighbor
The K-NN algorithm depends on the principle that the features corresponding to the several classes will form individual clusters in the feature space. The features that are closer to each other recognized as neighbors. This classifier takes k metric distances into account between the test sample features and those of the nearest classes, to classify a test feature vector. In K-NN architecture, the number of neighbor and the types of distance metrics are key factor . In this study, the Euclidean distance metrics has been utilized to build the K-NN model. In order to get optimum accuracy of K-NN model, we have picked different values of K and at K = 2, the K-NN model provides the best accuracy.
2.2.5 Support vector machine
The concept of SVM is to figure out the most suitable hyperplane within the feature space that classifies the data points distinctly. The hyperplane should be such that the gap between the hyperplane and each adjacent class is maximum . This hyperplane contributes to an increase in classification accuracy. Different types of kernel function and the regularization parameter C have a crucial role in the structure of SVM. The widely employed kernel functions of SVM are radial basis function (RBF), linear, polynomial and sigmoid . In this study, the Gaussian RBF kernel function has been employed. The C and gamma parameters are selected as default from the Classifier Learner App.
To validate the training model, k-fold cross-validation has been employed where the value of K is five. The complete data analysis section has been carried out in the Matlab environment. To classify the extracted features through the Machine Learning approach, we have used classifier learner apps which is a built-in toolbox of Matlab.
2.3 Performance evaluation
The performance of the proposed method has been analyzed in terms of confusion matrix, classification accuracy, true positive and false negative rate and AUC (Area Under the Curve) ROC (Receiver Operating Characteristics). The classification accuracy (CA) of the proposed method is calculated by Eq. (3) .
3 Results and discussion
In this section, the performance of the classifiers has been evaluated. Before training the classification model, the selected feature has been labeled. In this study, the left wink, right, wink and no wink have been labeled with 1, 2 and 3 respectively.
Feature extraction techniques
The Training accuracy of SVM and K-NN are 98.9% whereas LDA achieves 97.8% shown in Table 2. Once the models are trained, the test data have been utilized to test the models. The testing accuracy of LDA, SVM, and K-NN with respect to FFT and sample range shown in Table 2. The maximum testing accuracy with respect to FFT has been achieved by LDA whereas the SVM and K-NN have been achieved 96.7% with respect to sample range. The FFT and sample range have been employed separately to extract the feature. From Table 2, it is obvious that the training and testing accuracy with respect to the sample range is significantly higher than the FFT.
In this study, the facial expressions in terms of left wink, right wink, and no wink have been classified. The wink data has been recorded using a five-channel Emotiv Insight EEG headset in the form of brain activity. Two features namely, FFT and sample range have been extracted. The extracted features have been classified using SVM, K-NN, and LDA. The performance of the classifiers has been evaluated using classification accuracy, confusion metrics, AUC-ROC, TPR, and FNR. The feature, sample range has been achieved the better accuracy with all classifiers as compared to the FFT. The obtained classification accuracy signifies that the wink based facial expression can be utilized in the operation of assistive technology. However, there are some issues that need to be overcome. As the key objective of BCI technology is to assist the physically challenged people, the data should be captured from the targeted users. Moreover, the data should be collected from more subjects and the number of trials from each subject should also be increased. The classifier results should be transformed into device commands and in the meantime, the complete experiment should be developed in real-time.
This research was supported by the faculty of Electrical and Electronic Engineering, Universiti Malaysia Pahang, Malaysia through the Grant RDU180396.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- 3.Zhang R, Li Y, Yan Y, Zhang H, Wu S, Yu T, Gu Z (2016) Control of a wheelchair in an indoor environment based on a brain-computer interface and automated navigation. In: IEEE Transaction NEURAL systems rehabilitation engineering vol 24 https://doi.org/10.1109/TNSRE.2015.2439298 CrossRefGoogle Scholar
- 5.Bright D, Nair A, Salvekar D, Bhisikar S (2016) EEG-based brain controlled prosthetic arm.In: Conference on advances in signal processing CASP 2016 pp 479–483. https://doi.org/10.1109/CASP.2016.7746219
- 8.Pham T, Ma W, Tran D, Tran DS, Phung D (2015) A study on the stability of EEG signals for user authentication. In: 2015 7th International IEEE/EMBS conference on neural engineering (NER). pp 122–125. IEEE. https://doi.org/10.1109/NER.2015.7146575
- 9.Rashid M, Sulaiman N, Mustafa M, Khatun S, Bari BS (2019) The Classification of eeg signal using different machine Learning techniques for BCI application. In: Jong-Hwan, KimHyung Myung, SML (ed) Robot intelligence technology and applications. RiTA 2018. communications in computer and information science, vol 1015. pp 207–221. Springer, Singapore. https://doi.org/10.1007/978-981-13-7780-8_17 CrossRefGoogle Scholar
- 11.Raheel A, Majid M, Anwar SM (2019) Facial expression recognition based on electroencephalography. In: 2019 2nd International conference computer mathematics and engineering technologies iCoMET 2019 pp 1–5. https://doi.org/10.1109/ICOMET.2019.8673408
- 12.Ma J, Zhang Y, Nam Y, Cichocki A, Matsuno F (2013) EOG/ERP Hybrid human-machine interface for robot control. In: IEEE Interanational conference intelligent robots System pp 859–864. https://doi.org/10.1109/IROS.2013.6696451
- 13.Reyes CE, Rugayan JLC, Jason C, Rullan G, Oppus CM, Tangonan GL (2012) A study on ocular and facial muscle artifacts in EEG signals for BCI applications. In: IEEE Region 10 annual international conference proceedings/TENCON. pp 1–6 https://doi.org/10.1109/TENCON.2012.6412241
- 15.Pathirana S, Asirvatham D, Md Johar MG (2018) An agent based approach for electroencephalographic data classification. In: 13th International conference computer science education ICCSE 2018. pp 248–252. https://doi.org/10.1109/ICCSE.2018.8468709
- 16.Ahad R, Rahman KAA, Mustaffa MZ, Fuad N, Ahmad MKI (2019) Body motion control via brain signal response. In: 2018 IEEE EMBS Conference biomedical engineering science IECBES 2018-Proceedings pp 696–700 https://doi.org/10.1109/IECBES.2018.08626738
- 17.Queiroz RL, Bichara De Azeredo Coutinho I, Xexeo GB, Machado Vieira Lima P, Sampaio FF (2019) Playing with robots using your brain. In: Brazilian symposium games digital entertainment. SBGAMES. 2018–November pp 197–204. https://doi.org/10.1109/SBGAMES.2018.00031
- 18.Mamani MA, Yanyachi PR (2017) Design of computer brain interface for flight control of unmanned air vehicle using cerebral signals through headset electroencephalograph. In: Proceedings 2017 IEEE international conference aerospace signals, INCAS 2017. 2017–November, pp 1–4. https://doi.org/10.1109/INCAS.2017.8123499
- 19.Orenda MP, Garg L, Garg G (2017) Exploring the feasibility to authenticate users of web and cloud services using a brain-computer interface (BCI). In: Battiato S, Farinella G, Leo M, Gallo G (eds) New trends in image analysis and processing – ICIAP 2017. ICIAP 2017. Lecture Notes in Computer Science, vol 10590. Springer, Cham. https://doi.org/10.1007/978-3-319-70742-6_33 CrossRefGoogle Scholar
- 20.Insight Brainwear® 5 channel wireless EEG headset| EMOTIV, https://www.emotiv.com/insight/, last accessed 2019/10/03
- 21.EMOTIVPRO| The most advanced EEG software| EMOTIV, https://www.emotiv.com/emotivpro/, last accessed 2019/10/09
- 22.Biswas PC, Wahed S, Nath D, Rana MM, Ahmad M (2015) Determination of most effective rhythm, domain and stimulation technique for noninvasive brain computer interface through SVM algorithm. In: 2nd International conference on electrical engineering and information and communication technology, iCEEiCT 2015. https://doi.org/10.1109/ICEEICT.2015.7307368
- 26.Rakotomamonjy A, Guigue V, Mallet G, Alvarado V (2005) Ensemble of SVMs for improving brain computer interface P300 speller performances. In: Duch W, Kacprzyk J, Oja E, Zadrożny S (eds) Artificial neural networks: biological inspirations – ICANN 2005. ICANN 2005. Lecture Notes in Computer Science, vol 3696. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11550822_8 CrossRefGoogle Scholar