Performance evaluation of a motor-imagery-based EEG-Brain computer interface using a combined cue with heterogeneous training data in BCI-Naive subjects
© Choi et al; licensee BioMed Central Ltd. 2011
Received: 28 June 2011
Accepted: 12 October 2011
Published: 12 October 2011
The subjects in EEG-Brain computer interface (BCI) system experience difficulties when attempting to obtain the consistent performance of the actual movement by motor imagery alone. It is necessary to find the optimal conditions and stimuli combinations that affect the performance factors of the EEG-BCI system to guarantee equipment safety and trust through the performance evaluation of using motor imagery characteristics that can be utilized in the EEG-BCI testing environment.
The experiment was carried out with 10 experienced subjects and 32 naive subjects on an EEG-BCI system. There were 3 experiments: The experienced homogeneous experiment, the naive homogeneous experiment and the naive heterogeneous experiment. Each experiment was compared in terms of the six audio-visual cue combinations and consisted of 50 trials. The EEG data was classified using the least square linear classifier in case of the naive subjects through the common spatial pattern filter. The accuracy was calculated using the training and test data set. The p-value of the accuracy was obtained through the statistical significance test.
In the case in which a naive subject was trained by a heterogeneous combined cue and tested by a visual cue, the result was not only the highest accuracy (p < 0.05) but also stable performance in all experiments.
We propose the use of this measuring methodology of a heterogeneous combined cue for training data and a visual cue for test data by the typical EEG-BCI algorithm on the EEG-BCI system to achieve effectiveness in terms of consistence, stability, cost, time, and resources management without the need for a trial and error process.
Recently, BCI technology has progressed as state-of-the-art medical devices to control and communicate with applicable accessories such as artificial limbs, prosthetic and wheelchair using the signal of brain activity [1–3]. There have been numerous studies of brainwaves collected electrically from brain activity on the cortex related to the brain computer interface [4–6]. Other research has investigated the types of electrical brain activity that can be used to implement the EEG-BCI systems [7–9].
The electroencephalographic (EEG) mu rhythm is an 8-13 Hz rhythm generated by the sensorimotor cortex that is most prominent when subjects are resting and is attenuated or abolished when subjects move or observe biological movements [10, 11]. The μ-rhythm is capable of transforming by itself as opposed to the brain activity events . Motor imagery implies a thought activity of imagining of physical movement. Without such physical activity it can transform the μ-rhythm within the sensory motor cortex . In other words, when imagining a hand movement or actually moving the hand, ERD (Event-Related Desynchronization) occurs around the μ-rhythm area within the sensory motor cortex [14, 15].
However, the subjects in EEG-BCI system experience difficulties when attempting to obtain the consistent performance of the actual movement by motor imagery alone [7, 16]. Hence, it is necessary to find the condition that affect the performance factors of the EEG-BCI system to guarantee equipment safety and trust through an evaluation of the performance of the EEG-BCI system using motor imagery characteristics that can be utilized in the EEG-BCI testing environment.
In a naive homogeneous training data experiment, Kim found that the types of training data do not affect the level of accuracy . In an homogeneous training data experiment, Lee carried out a BCI experiment with experienced subjects and cross-compared three classifiers of combined cues (i.e., audio-visuals cues), analyzing the levels of statistical significance and cross-correlation . However, Lee's study lacked statistical confidence, as the subjects in the experiment built experience based only on ten trials. Thus, in addition to the sample size limitation, the experiment also lacked any consideration of naive subjects.
In an effort to mitigate the sample size issue and include additional variables designed to improve the concentration of the subjects. Furthermore, while the existing combined cue experiment is based on homogeneous training data, this experiment is based on the heterogeneous data, which provides a useful comparative analysis. It should be noted that for the heterogeneous case, the cue speed is a mix of four- and two-second durations. Hence, the key comparisons made here are the naive homogeneous training data experiment and heterogeneous training data experiment.
Subject and data acquisition
Characteristics of the experiments
Characteristics of experiment
Experienced homogeneous experiment
Naive homogeneous experiment
Naive heterogeneous experiment
Experience of subject
Naïve (i.e., no experience)
Number of subjects
Average age of subjects
23.9 ± 2.5
23.5 ± 1.8
Stimuli type of training data
Duration of a cue in one trial
6 or 8 seconds
Number of trials
The EEG data was classified using the least square (LS) linear classifier in case of the naive subjects through the common spatial pattern (CSP) filter. We calculated the accuracy using the training data set and test data set. The p-value of the accuracy was obtained through the statistical significance test (i.e., t-test). In case of the experienced subjects, we used three types of classifiers: the least square (LS) linear classifier using a linear matrix equation and a pseudo inverse matrix, the support vector machine (SVM) using a pattern classifier based on structural risk minimization, and linear discriminant analysis (LDA) using the method reducing the dimension of the data feature vector by maximizing the ratio between the intra-classes-distribution and the inter-classes-distribution. The accuracy as to how the EEG-BCI reading of the brainwave collected matches the actual instructed data is then recorded.
Cue-combinations in the experiment
Each combination consists of 50 trials. For the training data, the time for each cue is 6 or 8 seconds followed by 2 seconds of a blank screen, 2 seconds of a fixation cross and 2 or 4 seconds of cue intervals to train the motor imagery. For the test data, the time for each cue is 7 seconds, and this is followed by 2 seconds of a blank screen, 2 seconds of a fixation cross and 3 seconds of cue intervals to test the motor imagery. The subject imagines the motor imagery according to the audio-visual cue instructions presented on the computer monitor and speaker that are operated by the experiment manager.
One trial of cue presentation
Experienced homogeneous experiment
The experimental results of the experienced subjects using homogeneous training data derived from a homogeneous stimulus having the same cue time for motor imagery is presented for 3 seconds in case of the training data and 3 seconds in case of the test data, as shown in Table 3. For example, if we used the combined cue for the left motor imagery training tasks, a blank screen was presented to the experienced subject for 2 seconds and the fixation cross was continuously presented for 2 seconds. Finally, a red cylinder on the left side of the monitor with the beep sound and the "left" sound of the speaker were presented for 3 seconds simultaneously for motor imagery.
Naive homogeneous experiment
The experimental results of the naive subjects using the homogeneous training data derived from a homogeneous stimulus having the same cue time for motor imagery is presented for 3 seconds for training and 3 seconds for testing as shown in Table 3. For example, if we used the visual cue for the right motor imagery training tasks, a blank screen was presented to the experienced subject for 2 seconds and the fixation cross was continuously presented for 2 seconds. Finally, a red cylinder on the right side of the monitor with the beep sound was presented for 3 seconds for motor imagery.
Naive heterogeneous experiment
The experimental results using the naive heterogeneous training data derived from 2 types of heterogeneous stimuli having a different cue time was presented for 2 or 4 seconds in case of the training data as shown in Table 3. For example, if we used the combined cue for the right motor imagery training tasks, a blank screen was presented to the experienced subject for 2 seconds and the fixation cross was continuously presented for 2 seconds. Finally, a red cylinder on the right side of the monitor with the beep sound and the "right" sound of the speaker were presented for 2 or 4 seconds simultaneously for motor imagery. The duration of the last phase in this trial was randomized.
Experienced homogeneous experiment
The experiment with experienced subjects using homogeneous training data sought to investigate the responses to a combination of visual and auditory (i.e., audio-visual) cues. Furthermore, a comparative evaluation was done based on the three classifiers (LS, SVM, and LDA), followed by a statistical analysis to investigate the significance and correlation between the six combinations used in the audio-visual cue experiment.
Subject number 1 had an accuracy score of 0.8 for all three classifiers (LS, SVM, and LDA). The second highest performers were numbers 4 and 8, who scored above 0.6. Subject number 3 had the lowest accuracy mark for all three classifiers (LS, SVM, and LDA).
Naive homogeneous experiment
The analysis criteria are based on the six combinations described in Table 1. The results shown in Figure 3 are more general considering that they only show the general tendency in terms of the average. In comparison, Figure 4 is more indicative of the actual performance capacity, as the experiment was limited to subjects who scored above a certain level of accuracy.
Naive heterogeneous experiment
The analysis criteria are based on the six combinations described in Table 1. The results shown in Figure 5 are more general with respect to the fact that they only show the general tendency in terms of the average. In comparison, Figure 6 is more indicative of the actual performance capacity, as this experiment included only subjects who scored above a certain level of accuracy.
Experienced homogeneous data
The experienced homogeneous data is cross-compared against 3 classifiers, LS, SVM, and LDA. For the 10 subjects, the CV combination (training on a combined cue with testing was on a visual cue) showed the highest average accuracy. The same types of results were observed in all three of the classifiers (LS, SVM, and LDA). Furthermore, the combinations CA and CV out of the total of six showed the highest level of statistical significance (p < 0.05). On the basis of this result, an experiment was carried out on the 32 naive subjects. In this experiment, the comparison was made between experienced subjects using a combined cue as homogeneous training data and naive subjects using a combined cue as homogeneous training data.
Naive homogeneous data
Out of a total of six combinations in the naive homogeneous training data, the CV combination showed the highest average accuracy, at 0.562. The second highest was the VC combinations, at 0.56. However, when the comparison is confined to the subjects whose average accuracy is greater than 0.6, the percentage of the VC combination is higher than that of CV, at 37.5% and 34.4%, respectively. It is particularly of note that the VC combination included two subjects whose average accuracy score was greater than 0.8. This shows that the average accuracy alone does not sufficiently explain the individual characteristics of the subjects with all six combinations. The CV combination used a combined cue for the training data and a visual cue for the test data, while the VC used a visual cue for the training data and a combined cue for the test data. In other words, the combination of CV and VC is a cross-combination of combined and visual cues. The results suggest that a combination of these two types of cues results in higher average accuracy.
Naive heterogeneous data
The six cue-combinations used in the heterogeneous naive training data experiment showed that the CV combination led to the highest level of accuracy, at 0.564. The second highest level resulted from the VC cue, at 0.549. For the subjects with accuracy scores of 0.6 or above, the CV combination accounted for the highest percentage, at 34.4% and the second highest was VC at 28.1%. This shows a different trend from the homogeneous training data experiment.
Cross-comparison between the homogenous and heterogeneous experiment
On average, CV had the highest scores for both the homogeneous and heterogeneous criteria, and the heterogeneous data showed a higher average level of accuracy. In the case of VC, a contradictory result showed that the homogeneous data showed higher average accuracy levels than the heterogeneous data.
CV was trained on an audio-visual cue and tested on a visual cue. VC was trained on a visual cue and tested on an audio-visual cue. In both cases, there are common conditions that are designed to take advantage of an audio-visual cue and a visual cue as training or test data. In the end, the relationship between the two stimuli had a direct impact on the average performance. In the CV case, there was an improvement in the average performance in the experiment using the heterogeneous training data (p < 0.05), whereas in the VC case, there was decline in the average performance during the experiment using the heterogeneous training data (p < 0.05).
The VC case, whose training was on a visual cue and whose performance was evaluated on an audio-visual cue, showed the highest percentage of subjects whose average accuracy score was above 0.6 in the homogeneous training experiment. However, this case showed a lower percentage than the CV case in the heterogeneous training experiment. This suggests that the VC case, whose training was on visual cue and whose testing was on an audio-visual cue, is not affected by changes in the cueing conditions. This is indicative of more consistent performance.
When EEG-BCI-based motor imagery training tasks are carried out using naive subjects, the general applicability, stability and consistency of the accuracy levels are regarded the most essential. The set of experiments conducted here concluded that consistent accuracy can be achieved when the training data relies on a heterogeneous combined cue. Randomness of presenting time of the heterogeneous cue raises the power of concentration of the subjects, and this is thought to be the main cause of the consistency in performance.
An accuracy of the difference was 0.002 (p < 0.05) between 0.562 in the homogeneous one and 0.564 in heterogeneous one. A consistent result was obtained when the training data used a combined cue and the test data used a visual cue. Moreover, the combinations of a combined cue and a visual cue showed the highest at 0.562, 0.564 and the second highest accuracy at 0.56, 0.549.
We propose the use of this measuring methodology of a heterogeneous combined cue for training data and a visual cue as a testing cue by the typical EEG-BCI algorithm on the EEG-BCI system to achieve effectiveness in terms of consistency, stability, cost, time, and resources management without the need for a trial and error process.
- Dornhege G: Toward brain-computer interfacing. U.S.A: The MIT Press; 2007.Google Scholar
- McFarland DJ, Krusienski DJ, Sarnacki WA, Wolpaw JR: Emulation of computer mouse control with a noninvasive brain-computer interface. Journal of neural engineering 2008, 5: 101. 10.1088/1741-2560/5/2/001View ArticleGoogle Scholar
- Van Gerven M, Farquhar J, Schaefer R, Vlek R, Geuze J, Nijholt A, Ramsey N, Haselager P, Vuurpijl L, Gielen S: The brain-computer interface cycle. J Neural Eng 2009, 6: 041001–041001. 10.1088/1741-2560/6/4/041001View ArticleGoogle Scholar
- Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N: The hybrid BCI. Frontiers in neuroscience 2010., 4: Google Scholar
- Ang KK, Guan C, Chua KSG, Ang BT, Kuah C, Wang C, Phua KS, Chin ZY, Zhang H: Clinical study of neurorehabilitation in stroke using EEG-based motor imagery brain-computer interface with robotic feedback. 2010 Annual International Conference of the IEEE Buenos Aires, Argentina. IEEE 2010, 5549–5552.Google Scholar
- Blankertz B, Losch F, Krauledat M, Dornhege G, Curio G, Muller KR: The Berlin Brain--Computer Interface: accurate performance from first-session in BCI-naive subjects. Biomedical Engineering, IEEE Transactions on 2008, 55(10):2452–2462.View ArticleGoogle Scholar
- Wolpaw JR, McFarland DJ: Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. PNAS-Proceedings of the National Academy of Sciences of the United States of America 2004, 101(51):17849. 10.1073/pnas.0403504101View ArticleGoogle Scholar
- McFarland DJ, Anderson CW, Muller KR, Schlogl A, Krusienski DJ: BCI meeting 2005-workshop on BCI signal processing: feature extraction and translation. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 2006, 14(2):135–138. 10.1109/TNSRE.2006.875637View ArticleGoogle Scholar
- Pfurtscheller G, Neuper C: Motor imagery and direct brain-computer communication. Proceedings of the IEEE 2001, 89(7):1123–1134. 10.1109/5.939829View ArticleGoogle Scholar
- Arroyo S, Lesser RP, Gordon B, Uematsu S, Jackson D, Webber R: Functional significance of the mu rhythm of human cortex: an electrophysiologic study with subdural electrodes. Electroencephalography and clinical Neurophysiology 1993, 87(3):76–87. 10.1016/0013-4694(93)90114-BView ArticleGoogle Scholar
- Muthukumaraswamy SD, Johnson BW, McNair NA: Mu rhythm modulation during observation of an object-directed grasp. Cognitive Brain Research 2004, 19(2):195–201. 10.1016/j.cogbrainres.2003.12.001View ArticleGoogle Scholar
- Pfurtscheller G, Brunner C, Schlogl A, Lopes da Silva F: Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage 2006, 31(1):153–159. 10.1016/j.neuroimage.2005.12.003View ArticleGoogle Scholar
- Pfurtscheller G, Neuper C: Motor imagery activates primary sensorimotor area in humans. Neuroscience letters 1997, 239(2–3):65–68. 10.1016/S0304-3940(97)00889-6View ArticleGoogle Scholar
- Ince NF, Tewfik AH, Arica S: Extraction subject-specific motor imagery time-frequency patterns for single trial EEG classification. Computers in biology and medicine 2007, 37(4):499–508. 10.1016/j.compbiomed.2006.08.014View ArticleGoogle Scholar
- Blankertz B, Tomioka R, Lemm S, Kawanabe M, Muller KR: Optimizing spatial filters for robust EEG single-trial analysis. Signal Processing Magazine, IEEE 2008, 25(1):41–56.View ArticleGoogle Scholar
- Kim J, Kang D, Lee Y, Jung H, Lee I, Park H, Kim E, Lee M: Performance Evaluation of EEG-BCI Interface Algorithm in BCI(Brain Computer Interface)-Naive Subjects. Journal of biomedical engineering research: the official journal of the Korean Society of Medical & Biological Engineering 2009, 30(5):428–437.Google Scholar
- Lee Y: Performance Evaluation of Motor Imagery Characteristics based EEG-Brain Computer Interface for Ubiquitous Healthcare Service. Ph.D Yonsei, Biomedical Electronics & Information Lab 2010.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.