Implementation

Abstract

The proposed system comprises of three main parts, the first one being the wireless acquisition module, second is the signal processing module and the third, the last one is the output hardware module in which the user inputs get implemented as the final desired outputs. The following Fig. 4.1 shows the overall block diagram of the system implemented in our work. The workflow can be mainly classified into two parts. The first part is the electroencephalogram (EEG) acquisition and signal processing in real time. As shown in Fig. 4.1 next, the left side of the block diagram covers the first part of our work that is data acquisition, signal processing, and implementation of this processed data into real-time robot commands. The data that is referred to include only the raw EEG data provided by various electrodes in the EMOTIV EPOC neuroheadset. The second part of our work is acquisition, processing, and implementation of the gyroscope signals. Here, the gyroscope that is referred to is embedded in the EMOTIV EPOC neuroheadset. This has been mentioned on the right-hand side of the block diagram which can also be named as the mouse emulator part, that is, the data from the gyroscope is used to drive the mouse pointer on the computer to access the GUI in MATLAB. The third part is the design of GUI in MATLAB, shown in the right, bottom corner of the block diagram, which is to be concentrated upon by the user. When the abovementioned sections work at the same time simultaneously, the user can access his GUI in MATLAB to communicate with the Arduino-based autonomous navigation robot in real time.

4.1 Specifications Used

EMOTIV Research Edition Premium 1.0.0.5 application programming interface (API) has been used in the implementation as it allows the extraction of raw EEG data from the neuroheadset.

Fig. 4.1

(figure generated by the authors)

Flow diagram of the system designed as the per working process

The EMOTIV software development kit (SDK) is comprised of one or two SDK neuroheadsets, installation CD and one or two USB wireless receivers. The users’ brainwaves are captured by the neuroheadset and converted into digital form. Afterward, the brainwaves are sent to the USB receiver after some preprocessing. EMOTIVEmoEngine is a post-processing software which needs to be installed in the PC to apply the results obtained from the acquisition system to the EMOTIV application programming interface (API). The PC specifications are as follows:

• Intel Xeon processor (3.4 GHz, 16 GB RAM)

• 64-bit Windows 8.1 operating system

• MATLAB 2013b.

4.2 Acquisition

4.2.1 Signal Quality Check

The process of acquisition starts with the user putting on the neuroheadset correctly. The headset is switched on and then the Bluetooth dongle is connected to the PC for wireless reception of EEG information from the user’s scalp to the computer. When the Control Panel is viewed on the PC, the user first must select the corresponding user ID (needs to be created in case of new users). This displays the contact qualities of the electrodes, in the EMOTIV Control Panel, thereby providing guidance on how to get the best quality signal from the electrode contact. It is vital for the user to maintain quality contact for a better quality of classification of EEG. The following Fig. 4.2 depicts various contact qualities achieved in the neuroheadset during a session. Each circle represents a single sensor on the user’s scalp along with the approximate location of the sensor. The quality of each contact is denoted by the color of its representative circle. Various colors along with their significance are given below (Table 4.1):

Fig. 4.2

(figure generated by the authors during experimental sessions, Emotiv  2010)

Signal quality of EMOTIV EPOC

Table 4.1 Signal quality color representation (Emotiv 2010)

It must be tried that every electrode gives a status of green in the Control Panel. Only after the verification of signal quality, the user can proceed to the next step.

As shown in Fig. 4.3, the EMOTIV EPOC headset contains a single 16-bit ADC for conversion of the raw EEG and GYRO signals that are obtained in the analog domain into digital format. It uses a sampling frequency of 128 Hz. The headset also contains a 2-axis gyroscope embedded inside for detecting displacement of the headset in two axes, x and y. The built-in battery is of LiPoly type that can last for 12 h in working condition. Thus, a digitized form of EEG and GYRO data is acquired from the neuroheadset which is transmitted to the computer through a Bluetooth dongle. The wireless communication between the neuroheadset and the Bluetooth dongle takes place through some unknown proprietary method (in the name of EMOTIV). Next, the EPOC SIMULINK server helps to acquire all information into MATLAB SIMULINK. The EPOC server sends data to SIMULINK S-Function using a TCP/IP socket.

Fig. 4.3

(figure generated by the authors)

Process of acquisition in detail

4.2.2 SIMULINK EEG Importer Interfacing for Online Acquisition of Data

The EPOC SIMULINK EEG Importer has been used in order to acquire real-time EEG data from the EMOTIV EPOC headset to a MATLAB SIMULINK model. After successful installation of the EPOC SIMULINK Importer, the files, edk.dll, and edk_utils.dll are copied from C:\Program Files (x86)\EMOTIV Research Edition SDK v1.0.0.5-PREMIUM to C:\Program Files (x86)\EpocSIMULINK EEG Importer.

Fig. 4.4

(figure generated in SIMULINK by the authors)

Signal acquisition model through SIMULINK

Next, after achieving green signals in all electrodes as in the Control Panel, the EpocSignalServer (Fig. 4.5) from the desktop is started and the START button in the EpocSignalServer window is pressed. This shows green colored icons in the server status, user status, and signal status.

MATLAB is started next and the current working folder/directory is changed to the EPOC SIMULINK EEG Importer installation folder, that is, C:\Program Files (x86) \EpocSIMULINK EEG Importer/MATLAB. Next, the SIMULINK model, that is, compatible with the MATLAB version in use is started and the model is edited as per need. The model is named EMOTIVEpocEEG_testmodel_2012b.mdl. However, a few simulation settings are necessary before proceeding as listed down:

• Change the solver options type from variable step to fixed step.

• Set the solver at discrete (no continuous states) state.

• Set the fixed-step size (fundamental sample time) at 1/128 as the sampling frequency of EPOC headset is 128 Hz.

Figure 4.4 shows the complete SIMULINK model for extraction of EEG data from the Importer to MATLAB workspace. First of all, a SIMULINK block receives the EEG data in vector format, that is, the EPOC EEG block on the left-hand side. The incoming signals can be observed in the SCOPE1 which depicts all the available channels. Figure 4.5 shows the EPOC SIMULINK Server Control Panel which the enables the real-time acquisition and monitoring of the EEG and gyro signals. The de-mux (demultiplexer) extracts the components of the input signals and outputs these components as separate signals as shown in Fig. 4.6. On the other hand, the mux (multiplexer) combines its inputs into a single vector output, which is easier to work upon in MATLAB. Thus, the overall function of the de-mux and mux is that they extract the useful channels from the input vector and output these channels as a single vector.

Fig. 4.5

(figure captured by the authors during experimental sessions, Emotiv 2010)

EPOC SIMULINK Server Control Panel showing signal status

Fig. 4.6

(figure generated in SIMULINK by the authors)

Detailed SIMULINK model in operation

Fig. 4.7

figure captured during experimental sessions)

SCOPE 1 waveforms (all available channels along with gyro signals and packet data status signal;

The acquired data can be further processed in the SIMULINK environment. SIMULINK block, thus used is created out of a MATLAB MEX S-function and outputs a double data vector of size (McFarland and Wolpaw 2008). This vector is in the format of: COUNTER, AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4, GYRO-X, GYRO-Y, TIMESTAMP, FUNC ID, FUNC VALUE, MARKER, and SYNC SIGNAL. Figure 4.7 shows the output waveforms from SCOPE 1 before demultiplexing. Next, the acquired vector is partitioned as unique individual signals by the use of a demultiplexer as shown in Fig. 4.6. Afterward, the useful channels can be extracted from this vector. Using two multiplexers, the useful channels are separated from rest that is, using MUX 1 and MUX 3. MUX 2 is to extract the GYRO-X and GYRO-Y signals separately into a workspace variable. SCOPE 2 can be used to view all the useful channels in real time, which are: AF3, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4 (Fig. 4.8).

Fig. 4.8

(figure captured during experimental sessions)

SCOPE 2 waveforms showing 13 useful channels extracted by a demultiplexer

The channel F7 is ignored in our system as it got disrupted due to some mishandling by the users. Similarly, SCOPE 4 can be used to view the rest of the channels that include COUNTER, TIMESTAMP, FUNC ID, FUNC VALUE, MARKER, and SYNC SIGNAL.

SCOPE 3 can be used to view the gyroscope signals which show the movement of the user’s head in X-and Y-axis simultaneously (Fig. 4.9).

Fig. 4.9

(figure captured during experimental sessions)

SCOPE 3 showing the gyroscope movement sensing signals in X and Y axes

Signals that are to be used further for signal processing are stored in a Workspace variable in the MATLAB workspace as shown in Fig. 4.6. The SIMULINK block used for the same is the “To Workspace” block. For use of the model in real time, it is run for 0.5 s and then terminated. On termination, the model stops running and the workspace variables get updated. These data can be used for further steps of processing and then, the model is run for another 0.5 s for the next set of data, and so on.

For offline acquisition of data used for training purpose, the EMOTIV Testbench software can also be used. It allows the collection of data over a given period of time into.edf (European data format) form. This data can be converted into .csv (Comma separable value) form which can be easily read in MATLAB for further processing.

4.3 EEG Data Classification Module

The cycle shown in Fig. 4.10 is followed by the classification algorithm for classifying EEG data. Firstly, simulation in SIMULINK helps acquisition of data from the neuroheadset for 0.5 s. This data is processed by level 4 multi-resolution analysis resulting in DWT coefficients in the next stage. Out of these coefficients, the optimal one is chosen for principal component analysis (PCA), in order to remove redundancy from the signal. Then it is simulated by the neural network for the purpose of classification into the type of data it belongs to. Once the classification is done, the system comes to know about the intention of the user through his/her EEG and hence accordingly, drives the GUI in MATLAB for commanding the autonomous Arduino-based robot.

Fig. 4.10

(figure generated by the authors)

Flow diagram of EEG signal classification module in detail

All the processes mentioned in the block diagram above have been described below.

4.3.1 Signal Processing and Classification Algorithm

Preprocessing is done in order to lift up the quality of signals so that they can be made eligible to be processed in the subsequent steps. In our work, preprocessing has been done in the neuroheadset itself. No separate preprocessing has been done. The EMOTIV Testbench software displays the electroencephalographic data in real time along the fourteen monitored channels. Signals vary each time the user thinks something in a conscious or an unconscious manner. Variations may also be observed due to electromyographic data, such as facial expressions (smile, raise brow) and electrooculographic data, such as looking left, looking right, blinking, etc. The data obtained from Testbench software is used in order to form databases for four different classes namely, smile, raise a brow, neutral, and blink. This database is collected from three different subjects or users. Two of them are male and all of them are in the age range of 20–30 years (Table 4.2).

Table 4.2 List of Subjects

Ten trials for each class are made. The EMOTIV Testbench software gives the display of EEG data in real time, which can be stored by the user as per the required time period. Each trial is partitioned from the other by two minutes of rest. This is necessary as the user is to be brought to a stable condition before making him/her perform the next action.

The data obtained from the Testbench software is in .edf format and is converted into.csv format by the Testbench software itself. The .csv file thus obtained contains 36 columns of data in total. The first row contains a header notation. Columns 3–16 are the EEG readings of 14 channels. The other 22 columns of data are those gathered from preprocessed data. The rows denote the number of samples collected in the given time period. Since the sampling frequency used in EMOTIV EPOC is 128 Hz, 128 samples are obtained in 1 second. This means that a recording of 2 min gives us 128 * 2 * 60 = 15,360 samples.

In the next step, the unwanted data are removed from the .csv file and the 14-channel data are extracted into a MATLAB variable. Three databases are created for extracted data for the subjects separately. An important point to be noted is that, for creating databases, the samples from the duration 0.5–1.5 min are to be considered only. This is done because, whenever data is collected, the first and last 0.5 min has the possibility of containing the maximum amount of noise and randomness.

4.3.2 Analysis of Data

The behavior of the 14 channels can be observed by plotting them separately in MATLAB for each class. The following figures denote 0.25 min (1920 samples depicting 15 s) of data representing each of blink, raisebrow, smile, neutral data along with the channels where distinct variations can be found in each expression (Fig. 4.11 to Fig. 4.17).

Fig. 4.11

(figure captured during experimental sessions)

Blink waveform of all 14 channels shown in MATLAB

Blink can be seen in the P300 wave in the above waveforms. The clear variations are mostly observed in AF3, F7, O1, F4, and AF4 channels (Fig. 4.12). It can be seen in the form of a sudden peak and then, a fall in the wave.

Fig. 4.12

(figure captured during experimental sessions)

Channels where blink is distinctly visible as compared to others

Raise Brow is also a P300 wave seen in the Fig. 4.13. A sudden rise in the wave is followed by a slow falling with another probable peak in the mid. Maximum variations are observed in AF3, F3, T7, O1, FC6, and F8 channels (Fig. 4.14).

Fig. 4.13

(figure captured during experimental sessions)

Raise brow waveform of all 14 channels shown in MATLAB

Fig. 4.14

(figure captured during experimental sessions)

Channels where raise brow is distinctly visible as compared to others

Fig. 4.15

(figure captured during experimental sessions)

Smile waveform of all 14 channels shown in MATLAB

Smile signal is more of a random signal with several peaks in single smile signal intent of the user. Most variations are observed in F3, AF3, T7, T8, AF4, and FC6 channels (Figs. 4.15 and 4.16).

Fig. 4.16

(figure captured during experimental sessions)

Channels where smile is distinctly visible as compared to others

Fig. 4.17

(figure captured during experimental sessions)

Neutral waveform of all 14 channels shown in MATLAB

The neutral signal (Fig. 4.17) is observed in the system by making the subjects stay calm without any facial expressions or movements. The signal thus obtained shows no variable changes in all the channels.

4.3.3 Observations

It has been observed from the above studies that the distinct changes can be viewed only in a few channels that include all except O2 and P8. Training of neural networks is tried with and without the inclusion of the data obtained from these channels. It was observed that better classification accuracies could be obtained after the inclusion of all the channels without ignoring any of them in particular. So, the final training was done without neglecting the data from any of the channels. Otherwise, this step of reducing the number of channels could have been named as a feature extraction step.

4.3.4 DWT Analysis

The classification algorithm starts with a multi-resolution analysis of the samples to split the signal into different frequency ranges and examine their efficacy.

These signals can further be analyzed by applying Fourier transform and hence gathering the frequency components present in the signal. Short-time Fourier Transform gives us an idea of where in time the spectral components appear in the signal. The sampling frequency of our system is 128 Hz. Now, according to Nyquist criterion, the sampling rate, fs is twice the highest frequency component in the signal. This concept can be used to conclude that the highest frequency present in the signal essentially is 64 Hz (=128/2 Hz). This gives us an idea of applying multi-resolution analysis to split the signal in various frequency ranges and thus easy analysis of classification accuracy of various actions in those frequency bands. Figure  4.18 shows the whole process of multi-resolution analysis. It starts with our source signal x[n] that contains frequency components of (0–64) Hz. This signal is split into five frequency ranges:

Fig. 4.18

(figure generated by the authors)

Multi-resolution analysis in pictorial representation

• (0–4) Hz

• (4–8) Hz

• (8–16) Hz

• (16–32) Hz

• (32–64) Hz.

Four levels of decomposition give these five ranges of frequencies.

• In the first level, x[n], the source signal is passed through a low-pass filter (h[n]) and a high-pass filter (g[n]). This results in two frequency components ranging (32–64) Hz—high-frequency component and (0–64) Hz—low-frequency component. Then after, downsampling by a factor of two is done, which results into two DWT coefficients.

• The coefficient obtained by passing the signal through a high-pass filter (g[n]) and downsampled by two is called the “Detailed coefficient (D)”.

• The coefficient obtained by passing the signal through a low-pass filter (h[n]) and downsampled by two is called the “Approximation coefficient (A)”.

• In the next level, level 2, a similar process is repeated with source signal being the approximation coefficient from level 1 in place of x[n] as in level 1.

• Repetition of a similar process for four levels will provide four detailed coefficients, D1 (32–64) Hz, D2 (16–32) Hz, D3 (8–16) Hz, and D4 (4–8) Hz.

• An approximation coefficient is obtained from level 4, A4 (0–4) Hz.

The various ranges of frequencies in the EEG that can be linked to actions and different stages of consciousness are given here. An interesting point to be noted is that the DWT coefficients obtained by multi-resolution analysis share nearly the same ranges of frequencies as of the EEG frequency ranges, shown below:

• Gamma waves (31 Hz and up): This range reflects the mechanism of consciousness.

• Beta and Gamma waves show traits of attention, perception, and cognition.

• Beta waves (12–30 Hz): These are small and fast waves associated with focussed concentration and are dominant in the central and frontal areas of the brain.

• Alpha waves (7.5–12 Hz): Alpha waves are comparatively slower and are associated with relaxation and disengagement that are dominant in the frontal lobe and the back part of the brain.

• Theta waves (3.5–7.5 Hz): Theta waves are associated with inefficiency, daydreaming, frustration, emotional stress or disappointment, deep meditation, and creative inspiration. The lower levels of theta waves represent the line between awake and sleep state. However, higher levels of theta waves represent abnormalities in adults generally related to attention deficit/hyperactivity disorder (AD/HD).

• Delta waves (0.5–3.5 Hz): These are generally the slowest waves that occur when sleeping. If these waves occur when a person is awake, physical defects are assumed in the brain.

After level 4 DWT analysis in the algorithm, five frequency range signals are obtained as mentioned earlier, of which four are detailed coefficients and one approximation coefficient. The beta (D2) range of waves is chosen after multi-resolution. This is because beta waves are associated with focussed concentration related to the frontal and central areas of the brain. When a patient operates the GUI, these waves are witnessed to be the most dominant. The wavelet family used in our algorithm is “haar” which has been chosen after a prolonged study of obtained accuracies. Haar wavelet comprises of certain sequences of rescaled “square-shaped” functions forming a wavelet family altogether. It was proposed for the first time by Alfred Haar in 1909 (Haar 1911). The mother wavelet $\psi (t)$ of Haar wavelet transform is given as follows:

$$\psi \left( x \right) = \left\{ {\begin{array}{*{20}l} {1 } & {0 \le x \le \frac{1}{2}} \\ { -} 1& {\frac{1}{2} < x \le 1 } \\ {0}&{{\text{ otherwise}}} \\ \end{array} } \right.\quad \left( {{\text{Haar} \;1911}} \right)$$

(4.1)

$$\psi_{jk} (x) \equiv \psi \left( {2^{j} x - k} \right)\quad \left( {\text{Haar}\,\,{1911}} \right)$$

(4.2)

The obtained signal after applying DWT is then processed with principal component analysis (PCA) in order to remove correlations between the data.

4.3.5 PCA

The principal component analysis (PCA) is a method that can sometimes be used as a signal enhancer or can be applied for dimensionality reduction post feature extraction (Wold et al. 1987). PCA can also effectively find patterns in data which contain multiple variables for consideration, some of them probably being irrelevant. It is primarily a dimensionality reduction technique. It is one of the best methods for spotting the uniqueness or variability of the data. It converts the high dimensional data into lower dimensions by the method of projection of the least squares, capturing major inconsistencies, and ignoring minor inconsistencies. Before using PCA, it is important to standardize the input data by making the resultant mean, null.

PCA is a simple, nonparametric method that helps the extraction of germane datasets from raw information. The main intention of this procedure is to highlight the similarities and differences of the input data in order to spotlight the important patterns (Wold et al. 1987).

$${\text{Variation explained by each Principal Component }} = \frac{{{\text{Sum of Eigen Value}}}}{{{\text{Number of Variables}}}}\;({\text{Wold et al}}.\;1 9 8 7)$$

(4.3)

4.3.6 Neural Networks

Neural networks are based on self-learning algorithms which do not require programming skills of a programmer. The neural network models imitate the mechanism of biological processing of information in the human neural system. These models comprise of a set of inputs and outputs which are nonlinearly mapped with each other. The mapping is adaptive in nature.

Artificial neural networks (ANNs) are comprised of electronic processing elements (PEs) connected in a peculiar pattern. Its behavior is dependent upon the weights of the trained ANN. ANNs are quite advantageous over conventional electronic processing techniques which include flexible learning techniques, the ability of generalization, distributed memory, parallel processing, and redundancy.

In this work, the neural network has been used for pattern recognition. Pattern recognition, now, is one of the most successfully emerged applications of ANN. As shown in the following Fig. 4.19, the neural network under use has been developed with the help of MATLAB toolbox, comprising of 10 hidden neurons, classifying 2 types of output data. The input data contains vectors of size 13. In other words, each sample contains 13 sets of data in it.

Fig. 4.19

(figure captured during experimental sessions)

Neural network format designed in MATLAB

Scaled conjugate gradient has been used in this algorithm. Various ranges of frequencies have been tested upon for better classification accuracies as shown below. The figures shown later in this section depict confusion matrices of ANN training in alpha, beta, gamma, delta and theta ranges of frequencies in two subjects, 1 and 2. Training: Cross-Validation: Testing Ratio is set at 90:5:5.

4.3.6.1 Scaled Conjugate Gradient Algorithm

Scaled conjugate gradient algorithm is a subdivision of the conjugate gradient methods of feedforward neural network classification algorithms. The class of conjugate gradient methods (CGMs) comprises of second-order techniques which minimize multiple variable goal functions. The technical meaning of second order functions is that the second order derivatives of the goal function are used by these methods; while on the other hand, the first-order techniques make use of the first-order derivatives. Standard backpropagation techniques can be classified as first-order techniques. Out of the two variants, the performance of second-order techniques is better than first-order techniques when it comes to finding a way to the local minimum. Even after using second order information, SCG requires only $O(N)$ memory usage, where $N$ is the number of weights in the neural network. However, a great disadvantage is its higher computational cost. It needs to compute the response of the network to each set of training input, several times before confirming the network’s final makeover (Scaled Conjugate Gradient 2017). SCG algorithm is specifically designed for avoiding consumption of ample time for line searching. This algorithm is a combination of the model-trust region approach and the conjugate gradient approach together. There is no line search performed during each iteration. Speeding up of this algorithm depends upon the convergence criterion. The more accuracy needed, the higher speed is followed. SCG is a fully automated algorithm requiring no user-dependent parameters.

CGMs, just like other standard backpropagation methods, use an iterative process to reach the minimum value. The direction of iteration of conjugate gradient method is the conjugate of the directions of previous steps. This property makes it different from standard backpropagation methods. In the case of standard backpropagation methods, the direction is always along the error function gradient. In addition to it, each step of iteration is independent of the other in case of CGMs. For standard backpropagation methods, there are instances when the minimization achieved in one step of iteration is undone by the next step, although partially, and so on.

4.3.6.2 Main Features of SCG

The scaled conjugate gradient method of neural network classification differs from other conjugate gradient algorithms in two main aspects as given below.

In each iteration k, weight $w_{k}$ is computed such that the new conjugate direction is $R^{N}$ and the step size in this direction is $w_{k + 1} = w_{k} + \alpha_{k} \cdot p_{k}$. $p_{k}$ is eventually dependent on $\alpha_{k}$. Here, $\alpha_{k}$ is the Hessian matrix of the error function. In other conjugate gradient methods, the evaluation of $\alpha_{k}$ is simply done by a tedious procedure. SCG, on the other hand, computes the Hessian matrix by simple approximation of the error term $E^{\prod } (w_{k} )$.

SCG also uses a scalar $\alpha_{k}$ that can control the vagueness of the Hessian matrix so that there is no possibility of the Hessian not being a positive definite, preventing the algorithm from achieving good performance.

SCG has contributed to both the fields of optimization theory and neural learning and is also considered faster than standard backpropagation and other conjugate gradient methods.

4.3.6.3 Parameters of SCG

Computation of parameters $s_{k} = \frac{{E^{\prime}(w_{k} + \sigma_{k} \cdot p_{k} ) - E^{\prime}(w_{k} )}}{{\sigma_{k} }} + \lambda_{k}$. p_k and $\lambda_{k}$ are dependent on their respective values at step $k - 1$.There are two main initial parameters of SCG namely, the initial values $\lambda_{k}$ and $\sigma_{k}$. These values are dependent on the respective conditions $\lambda_{k}$ and $\sigma_{1}$. Based on observations, Moller has proved that bigger values of $0 < \sigma_{1} \le 10^{ - 4}$ can cause a time taking convergence. The third parameter $\lambda_{1}$ varies from $0 < \lambda_{1} \le 10^{ - 6}$ (can be compared to standard backpropagation).

Termination of the learning process of a neural network is usually determined by the user. Unfortunately, the $\sigma_{1}$ mechanism sometimes leads to assigning of too large values to $\Delta_{\hbox{max} }$ in cases when further progress is difficult. To overcome floating-point exceptions, a termination criterion is added to SCG based on a few CGMs. SCG has an important advantage of being a batch learning method which makes it immune to pattern shuffling.

The confusion matrix generated in MATLAB contains significant information on its own. Four matrices are generated at a time. They all are for different sets of data. The first matrix is for the training set, next is for the validation data set. Next, two are for test data set and an overall confusion matrix, respectively (Fig. 4.20).

Fig. 4.20

(figure captured during experimental sessions)

Confusion matrix example

The above diagram is an example of a generic confusion matrix. It can be explained in detail as follows. There are three classes of data in total. Hence, three rows and three columns can be observed for the given data. The first column depicts the class 1 targets. There were seven class 1 targets in total out of which, six were assigned correctly to the output class 1 depicted as green in color. The rest 1 data is assigned incorrectly to the output class 2 due to which it is shown in red color. 0 entries were classified to the output class 3, which can thus be ignored. Now, the percentage accuracy of classification for class 1 data can be calculated as 6/7 * 100 = 85.7% shown in GREEN in the first column bottom box. Incorrect assignment of data is similarly represented in RED that can be calculated as 1/7 * 100 = 14.3%. Similar readings can be taken from the other output classes for their corresponding columns. The overall accuracy is calculated as the percentage of input data that are correctly classified out of the total number of input data. In this case, 22 data are correctly classified out of 27 data which gives an overall accuracy of 22/27 * 100 = 81.5%. This value has been shown in the box of the confusion matrix. The diagonal elements shown in GREEN depict the correctly classified data in all the three classes.

Subject 1 training confusion matrices and performance graphs (Figs. 4.21, 4.22, 4.23, 4.24 and 4.25):

Fig. 4.21

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 1 alpha frequency range

Fig. 4.22

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 1 beta frequency range

Fig. 4.23

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 1 gamma frequency range

Fig. 4.24

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 1 delta frequency range

Fig. 4.25

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 1 theta frequency range

Subject 2 training confusion matrices and performance graphs (Figs. 4.26, 4.27, 4.28, 4.29 and 4.30):

Fig. 4.26

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 2 alpha frequency range

Fig. 4.27

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 2 beta frequency range

Fig. 4.28

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 2 gamma frequency range

Fig. 4.29

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 2 delta frequency range

Fig. 4.30

(figure captured during experimental sessions)

Confusion matrix and performance plot in Subject 2 theta frequency range

It has been clearly observed from the above confusion matrices that maximum classification accuracies are obtained in the beta and gamma ranges of frequencies. The theoretical explanation behind this is that beta waves are linked with focused concentration and the gamma waves are linked with traits of consciousness which are both dominant when the user is interacting with the GUI in MATLAB.

A higher accuracy has been obtained in the case of subject 1.

The database obtained during the acquisition phase of implementation is used in sections in order to train the neural network as shown till now. An important point to be noted is that the valuable samples from the database are first subdivided into sections of 64 samples before applying the necessary steps of signal processing. After the application of signal processing steps to those 64 sample sets of data, those applied to the training session of the neural network. We obtain the accuracies of classification as mentioned in this section.

Now, the best trained neural network is fed with the same size of 64 processed data samples in the real-time domain via SIMULINK. On doing the same, we receive a classified output at the same time and can process it further as a decision maker in the MATLAB GUIs designed to control robots as shown in the next section.

4.4 Gyroscope Signal Processing

It was observed that the gyroscope embedded in EMOTIV EPOC neuroheadset is designed to provide the angular velocity (rate of change of the angular position over time) measured in deg/s. The velocity data was then passed through a Kalman filter to remove unwanted noise and jitter. The Kalman filter operates as a state estimator for input data and helps taking care of jitter or high-frequency noise. Figure 4.31 shows the repetitive method followed in order to implement the mouse emulator.

Fig. 4.31

(figure generated by the authors)

Flowchart for the method of implementation of mouse emulator

Kalman is a recursive predictive filter that is based on the principle of covariance error reduction, thereby behaving as an ideal estimator. It is also termed as a Bayesian filter as it is based on recursive Bayesian estimation. The filter basically estimates the future states of an active system based on noisy incoming signals, considered as measurements, from the gyroscope. A continuous repetition of this process leads to a lessening of the estimated covariance error between the estimated state and the measurement. A new estimate of system’s present state is calculated using the previous time step’s state estimate and the present time step’s measurement. Consider an object moving in a straight path but affected by random uncontrollable forces. If its position is measured every $T_{\text{s}}$ second, but the measurements being affected by those uncontrollable forces, a model can be developed to depict the movement of that object.

$$\text{State transition matrix},\,A = \left[ {\begin{array}{*{20}c} 1 & 0 & {T_{\text{s}} } & 0 \\ 0 & 1 & 0 & {T_{\text{s}} } \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{array} } \right]$$

(4.4)

$$\text{Measurement matrix},\,C = \left[ {\begin{array}{*{20}c} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \end{array} } \right]$$

(4.5)

The two main processes followed in Kalman filtering are prediction and correction. The two-state equations (Anon 2017) governing the motion of the object are:

$$\bar{x}_{t} = Ax_{t - 1} + Bu_{t}$$

(4.6)

$$\bar{z}_{t} = C\bar{x}_{t} + E_{Z}$$

(4.7)

Figure  4.32 depicts the full recursive procedure followed by the Kalman filter. Prediction part of the filter solves the state Eqs. 4.6 and 4.8 describing the model at the time, $t + 1$. Here, $x$ is the state vector and $z$ is the measurement vector. After a step of prediction, the states are projected to the next step where correction of the error covariance is done. The following set of equations (Anon 2017) is used for minimization of error covariance.

$$\bar{E}_{t} = AE_{t - 1} A^{T} + E_{t}$$

(4.8)

Fig. 4.32

(figure generated by the authors; equations taken from Anon 2017)

Flowchart for implementation of Kalman filter

$\bar{E}_{t}$ denotes the estimated error covariance matrix generated in the step of prediction which follows the abovementioned equation. The next step, initializing the process of “Correction”, is to calculate the Kalman gain, $K_{t}$ using Eq. 4.9. $E_{Z}$, which denotes the covariance, when has a high value, leads to a less informative data and hence, a smaller $K_{t}$. In other words, the larger value of state covariance gives a more trustworthy value of measurement data and thus a larger value of $K_{t}$. Once the Kalman gain is calculated, the next state of the system can be calculated using Eq. 4.10, where the Kalman gain depicts the importance of the incoming measurement data.

$$K_{t} = \bar{E}_{t} C^{T} \left( {C\bar{E}_{t} C^{T} + E_{Z} } \right)^{ - 1} \quad \left( {\text{Anon 2017}} \right)$$

(4.9)

$$x_{t} = \bar{x}_{t} + K_{t} \left( {Z_{t} - C\bar{x}_{t} } \right)\quad \left( {\text{Anon 2017}} \right)$$

(4.10)

Next, the error covariance $E_{t}$ is evaluated using the previously estimated error covariance, $\bar{E}_{t}$ and the Kalman gain, $K_{t}$. So, $\bar{E}_{t}$ is the error covariance before an update and $E_{t}$ is the error covariance related to process noise vector whereas, $E_{z}$ is the error covariance related to the measurement noise vector.

$$E_{t} = (I - K_{t} C)\bar{E}_{t} \quad \left( {\text{Anon 2017}} \right)$$

(4.11)

Figure  4.33 shows how gyroscope data has been improvised using Kalman filtering. The process of filtering is followed by removal of offset value from both GYRO-x and GYRO-y signals, which are 1700 and 1600, respectively.

Fig. 4.33

(figure captured during experimental sessions)

Step-by-step plots of digital data

After this, integration is done in order to find out the relative displacement and hence obtain the final data that can be used for implementation. Angular velocity (deg/s) is given as

$$\dot{\theta } = \frac{{{\text{d}}\theta }}{{{\text{d}}t}}$$

(4.12)

In continuous and discrete form, respectively, angular displacement (Anon 2017) is given as

$$\theta (t) = \int\limits_{0}^{t} {\dot{\theta }(t){\text{d}}t \approx \sum\limits_{0}^{t} {\dot{\theta }(t)} T_{\text{s}} }$$

(4.13)

4.5 Implementation of BCI Controlling Robot  (Robot 1 and 2)

4.5.1 Robot 1 (Fire Bird V ATMEGA2560) Specifications and Implementation of “Control Using BCI”

A graphical user interface designed in MATLAB is used to drive the Fire Bird V-BOT (Fig. 4.34) in all its possible directions. Technical hardware and software specifications have been provided in Table 4.3.

Table 4.3 Robot 1 full specifications (Nex-robotics.com 2017)

Fig. 4.34

(figure captured during experimental sessions, Nex-robotics.com 2017)

FIRE BIRD V (robot 1) for BCI-controlled robot application 1

4.5.1.1 Interfacing with the PC Through MATLAB

Integrated development environments (IDEs) are a necessary part of programming for AVR microcontrollers. Many of them are available on the internet out of which, some work wholly only for a month or so. The free IDE, AVR Studio from ATMEL uses WIN AVR as an open-source C compiler at the back end. It also has built-in In-Circuit Emulator and an AVR instruction set simulator. A program can be written, compiled, and debugged in this software and then loaded onto the robot using an in-system programmer (ISP). It also supports included AVR Assembler and any external AVR GCC compiler in a complete IDE environment. The two main advantages of AVR Studio can be listed as follows:

The single application window can be sued to debug and edit also enabled with faster error tracking. Breakpoints are saved and restored between sessions, even if codes are edited.

After opening AVR Studio, a new project can be opened selecting “New Project”. The project type is to be chosen as “AVR GCC” and any name can be given. Thus, all the files can be created in this way inside the new folder. The location of the folder can be chosen as per wish in the location window. After this, the debugging platform and device are to be selected as “AVR Simulator” and “ATMEGA2560”, respectively. Another important aspect to be noted is the selection of Crystal frequency which is 11.0592 MHz in our case. Optimization level is set at “-O0”. “Optimization” option defines the optimization level for all files. Higher optimization levels will produce code that is harder to debug. Stack variables may change location, or be optimized away, and source level debugging may “skip” statements because they too have been optimized away. “-O0” stands for no optimization, that is, the same as not specifying any optimization.

Fig. 4.35

Interfacing between pins of ATMEGA 2560 and DIL-16 (Nex-robotics.com 2017)

Figure  4.36 shows the pin configuration of the motor driver L293D. There are four inputs here where input 1 is connected to pin 2, input 2 is connected to pin 7, input 3 is connected to pin 10, and input 4 is connected to pin 15. Port A of the microcontroller ATMEGA 2560 which has four pins (PA0, PA1, PA2, and PA3) connects to the input pins of the L293D. Figure  4.35 shows the interfacing between the microcontroller ATMEGA 2560 and DIL-16. Figure 4.37 shows the positioning of the DC geared motors and posiiton encoders in the robot. Now the pin configurations for different directions of motion are found out from the following truth Tables 4.4, 4.5 and 4.6.

Fig. 4.36

Pin configuration of L293D (Nex-robotics.com 2017)

Table 4.4 Truth table for motion direction control (Nex-robotics.com 2017)

Fig. 4.37

DC geared motors and position encoders in FIRE BIRD V (Nex-robotics.com 2017)

Table 4.5 Pin functions for motion control (Nex-robotics.com 2017)

Table 4.6 Pin functions of ATMEGA2560 for motion direction control and their hexadecimal equivalent (Nex-robotics.com 2017)

After compilation of the program in AVR Studio 8, it provides a “hex” file. This “hex” file needs to be loaded on the robot using in-system programmer (ISP) as mentioned earlier. The ISP programming is done by the Khazama AVR programmer. First, a chip signature is read from the command section. However, an important point to be noted here is, if the work is being done on Windows 8 platform, the digital signature verification is to be deactivated as Windows 8 does not allow software without digital signatures to run. The deactivation is done at start-up of the Windows. Afterward, the .hex file generated earlier is loaded to the buffer of Khazama programmer following which, the flash buffer is written to the chip, that is, ATMEGA2560.

This will burn the program into the chip. For testing of serial communication between PC and robot and getting the appropriate feedback, the terminal software can be used. Figure  4.38 shows how the commands are sent to FIRE BIRD V through HyperTerminal.

Fig. 4.38

(figure captured during experimental sessions)

Commands sent to FIRE BIRD V through HyperTerminal

The robot motions are controlled using BCIs. All possible motions that are included in the Graphical User Interface designed in MATLAB for controlling the Fire Bird V have been mentioned below:

Forward (both the motors move forward),

Backward (both the motors move backward),

Left (left motor moves backward and right motor moves forward),

Right (left motor moves backward and right motor moves forward),

Stop (both the motors stop),

Soft left 1 (left motor stops and right motor moves forward),

Soft right 1 (left motor moves forward and right motor stops),

Soft left 2 (left motor moves backward and right motor stops),

Soft right 2 (left motor stops and right motor moves backward),

The user communicates with the GUI (depicted in Fig. 4.39) with the help of clicks and cursor movements which have been made possible by acquiring data from the patient’s brain through the EPOC headset. This GUI has 9 control buttons (forward, backward, left, right, stop, soft left 1, soft right 1, soft left 2, and soft right 2). When the push button is pressed, the necessary command is passed on to the device FIRE BIRD V through serial communication.

Fig. 4.39

(figure generated by the authors)

GUI designed for robot 1 in MATLAB

The above events were generated as per the pin configurations and hex codes in hardware manual. The pins PA0, PA1, PA2, and PA3 of ATMEGA 2560 microcontroller chip were interfaced with pin 7, pin 2, pin 15, and pin 10 of L293D, respectively. Then the robot motions Forward, Backward, Left, Right, Stop, Soft Left1, Soft Right1, Soft Left2, and Soft Right2 were carried out when the device commands “8”, “2”, “4”, “6”, “5”, “7”, “1”, “3”, “9” were sent to the robot through serial communication. Figure  4.40 shows the flow diagram of an implementation of the robot motion control using BCI.

Fig. 4.40

(figure generated by the authors)

Implementation of robot 1 control through BCI—flow diagram

The patient or the user decides which direction he/she wants to go to through the MATLAB GUI and clicks the respective button in the GUI as per the desired direction.

4.5.2 Robot 2 (Arduino-Based Free-Wheeling Robot) Specifications and Implementation of “Robot Control Using BCI”

Graphical User Interface in MATLAB was designed for driving the robot in any desired direction and in different motions. The robot 2 is based on Freeduino MEGA 2560 board (Fig. 4.41) containing ATmega 2560 microcontroller. The overall hardware specifications have been given below in Table 4.7:

Table 4.7 Hardware specifications (Robot 2; Table generated by the authors)

The graphical user interface can be used to convert the facial expressions into real-time commands to be sent to the robot. The process starts with the BCI system converting signals from the patient into mouse click of the computer and then, the graphical user interface converts BCIs output signals into robotic commands in the Arduino board. The GUI also provides feedback to the user in the form of variable images whenever a command is sent to the robot through Arduino board. In this way, the user can teach himself/herself to use the GUI for the desired movements in the robot.

Fig. 4.41

(image taken from laboratory experiments)

Robot 2 Arduino-based autonomous robot with high powered speed controllable wheels

Arduino board has been interfaced with MATLAB software in order to move the robot in its possible directions. Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It is intended for artists, designers, hobbyists, and anyone interested in creating Interactive objects or environments. (Arduino.cc 2017) Arduino MEGA 2560 is a board with Freeduino MEGA 2560 microcontroller which directly receives serial data from the computer to get delivered to the robot.

The commands generated by MATLAB GUI are sent to the Arduino board which is connected through a USB cable. (Arduino.cc 2017) The Arduino board has 16 analog pins and 54 digital pins out of which 14 provide PWM output. Since the speed of the robot is needed to be controllable, the PWM pins are used.

The board uses Atmega8U2 as its USB to Serial converter. The pins 46, 47, 38, and 36 are, respectively, used to control the directions and braking of left and right motors, which come to the category of digital pins. Pins 3 and 4 are PWM pins used to control the speed of revolution of the robot wheels.

The Arduino MEGA 2560 board was first programmed with Arduino software (sketch) in order to make it capable of receiving and interpreting data successfully and then it is interfaced with the robot and computer to send motion signals and move the robot in any desired motion.

The Arduino digital pins are to be configured as either input or output pins. By default, they are programmed as input pins. The pin.Mode (pin, Mode) command can be used to configure the digital pins. Here, “pin” denotes the pin number to be configured and “Mode” is the desired mode, which can be either “input” or “output”.

4.5.2.1 How to Set up an Arduino

The required components are the board and a cable to link plug A and B. By this, the connection between the PC and the board is made. Next, the board is connected to the PC. COM port (Arduino, by default, gets access to one of the serial ports on the PC which are commonly known as the COM ports. Here COM4 is used). The integrated development environment (IDE) software called Arduino is downloaded. Compilation and verification of the sketch are done to initialize serial communication between MATLAB and Arduino board into this software and then this sketch is loaded into the board.

In serial port communication, information gets transferred at the rate of 1 bit at a time. Arduino IO Package allows communicating with an Arduino Uno or MEGA 2560 over a serial port through MATLAB software. It can be obtained as a support package from MATLAB. It comprises of a MATLAB API which is run on the host computer and a server program that runs on the Arduino. When operated altogether, the software allows accessing Arduino analog I/O, digital I/O, operate servo motors, read encoders, and even handling dc and stepper motors using the Adafruit Motor shield, all by using MATLAB command lines. Initialization of the Arduino board is done by a command as follows (Sets COM4 as the COM port for interfacing):

$${\text{a}} = {\text{arduino}}\left( {{\text{COM}}4} \right)$$

For the patient to gain proper control over the designed system, he/she needs to be trained with patience. After this process, the user is allowed to access the GUI. The elements included in the GUI have been explained in detail in the following section.

4.5.2.2 Possible robot motions

	Forward motion: Clicking this button will make the robot move in a forward direction. That is, both sides, the motors move in a forward direction.
	Moving left: Clicking this button will make the robot take a left turn. The right motors move forward and left motors move backward.

	Moving softleft_1: Clicking this button will make the robot take a very soft left turn. The left motors do not move and the right motors move in a forward direction.
	Moving softleft_2: Clicking this button makes  robot take another type of a very soft left turn. The right motors are braked (stopped) and left motors are moved backward.
	Stopping robot : Clicking this will be braking the robot to halt. It will stop all the motors.
	Moving softright_1: Clicking this pushbutton will make the robot take a very soft right turn marked by right motor braked (stopped) and left motor moving in the forward direction.
	Forward in High Speed: This attribute is to move the  robot at a comparatively higher speed as normal motion. Both side motors move in the same direction.
	Moving right: Clicking this button makes the robot take a sharp right turn marked by left motor moving in the forward direction and right motors in a backward direction.
	Moving softright_2: Clicking this button will make the robot take a soft right turn with the right motor moving in a backward direction and left motor stopped.
	Backward motion: Clicking this button will make the robot move backward, that is, both left and right motors will move in the backward direction.

The patient must decide in which direction he has to move and click the button accordingly with the help of his/her facial expression generated EEG signal. The full working diagram has been shown in the next flowchart (Fig. 4.42).

Fig. 4.42

(figure generated by the authors)

Implementation of robot 2 controlled by BCI