Skip to main content

Table 3 Comparison of common Level 2 BCI-Based AAC performance metrics

From: Performance assessment in brain-computer interface-based augmentative and alternative communication

Metric Description/Equation Evaluation criteria
Error correction Rate enhancement Control State detection Experimental protocol Practicality
Written symbol rate (WSR) WSR = 2 SR 1 c 0 , SR > 0.5 , else
where SR = ITR × c/log2(N)
    
Practical bit rate (PBR) PBR = 2 P 1 c 0 × log 2 N , P > 0.5 , else
The original reference did not report a formula, this was back-calculated from the results
   
Extended confusion matrix (ECM) Confusion matrix, as in Table 2, extended to allow for an extra detected class, “abstention.”    
EffSYS Ef f SYS = 1 L CW ¯ × ESC ¯
Where ESC ¯ = i = 1 N LA p ^ i 2 p i 1
Where LCW is the mean codeword length, NLA is the number of symbols in the logical alphabet, p ^ is the probability of a logical output appearing, and p(i) is the probability of identifying output i.
This equation is simplified from equation (20) of[17], assuming that all errors require two selections to correct. Interested readers are referred to[17] for alternative methods of error correction.
   
EffSYS Ef f SYS ' = 1 L CW ¯ × ESC ¯
Where ESC = i = 1 N LA p ^ × 1 p i + 1 p i 1 1 2 * p B 1
Where LCW is the mean codeword length, NLA is the number of symbols in the logical alphabet, and p ^ is the probability of a logical output appearing, as per[6]. The symbols p(i) and pb are the probabilities of correctly identifying output i and the backspace option, respectively.
  
Output characters per minute (OCM) OCM = Correct Characters Time Taken  
BCI-Utility metric U = E benefit / selection E time / selection
U must be derived for particular problems and can take different forms, several of which are presented in[16]. Several of these forms are redundant with other level 2 metrics, for example, in the experiment presented in[16], this metric becomes identical to practical bit rate.
*
  1. c: time per selection; N: number of choices; ITR: information transfer rate; P: probability of correct selection.
  2. *: Not currently practical, but can be with future work.
  3. Check marks indicate that the metric fulfills the evaluation criterion.