Skip to main content

# Table 3 Comparison of common Level 2 BCI-Based AAC performance metrics

Metric Description/Equation Evaluation criteria
Error correction Rate enhancement Control State detection Experimental protocol Practicality
Written symbol rate (WSR) $\mathit{WSR}=\left\{\begin{array}{c}\hfill \frac{\left(2\mathit{SR}-1\right)}{c}\hfill \\ \hfill 0\hfill \end{array}\right\\begin{array}{c}\hfill ,\mathit{SR}>0.5\hfill \\ \hfill ,\mathit{else}\hfill \end{array}$
where SR = ITR × c/log2(N)

Practical bit rate (PBR) $\mathit{PBR}=\left\{\begin{array}{c}\hfill \frac{\left(2P-1\right)}{c}\hfill \\ \hfill 0\hfill \end{array}\right\×{log}_{2}\left(N\right)\begin{array}{c}\hfill ,P>0.5\hfill \\ \hfill ,\mathit{else}\hfill \end{array}$
The original reference did not report a formula, this was back-calculated from the results

Extended confusion matrix (ECM) Confusion matrix, as in Table 2, extended to allow for an extra detected class, “abstention.”
EffSYS $\mathit{Ef}{f}_{\mathit{SYS}}=\frac{1}{\overline{{L}_{\mathit{CW}}}×\overline{\mathit{ESC}}}$
Where$\overline{\mathit{ESC}}=\sum _{i=1}^{{N}_{\mathit{LA}}}\frac{\stackrel{^}{p}\left(i\right)}{2p\left(i\right)-1}$
Where LCW is the mean codeword length, NLA is the number of symbols in the logical alphabet,$\stackrel{^}{p}$ is the probability of a logical output appearing, and p(i) is the probability of identifying output i.
This equation is simplified from equation (20) of[17], assuming that all errors require two selections to correct. Interested readers are referred to[17] for alternative methods of error correction.

EffSYS $\mathit{Ef}{f}_{\mathit{SYS}}\text{'}=\frac{1}{\overline{{L}_{\mathit{CW}}}×\overline{\mathit{ESC}}}$
Where$\mathit{ESC}=\sum _{i=1}^{{N}_{\mathit{LA}}}\stackrel{^}{p}×\left(\frac{1}{p\left(i\right)}+\left(\frac{1}{p\left(i\right)}-1\right)\left(\frac{1}{2*{p}_{B}-1}\right)\right)$
Where LCW is the mean codeword length, NLA is the number of symbols in the logical alphabet, and$\stackrel{^}{p}$ is the probability of a logical output appearing, as per[6]. The symbols p(i) and pb are the probabilities of correctly identifying output i and the backspace option, respectively.

Output characters per minute (OCM) $\mathit{OCM}=\frac{\mathit{Correct}\phantom{\rule{0.5em}{0ex}}\mathit{Characters}}{\mathit{Time}\phantom{\rule{0.5em}{0ex}}\mathit{Taken}}$
BCI-Utility metric $U=\frac{E\left(\mathit{benefit}/\mathit{selection}\right)}{E\left(\mathit{time}/\mathit{selection}\right)}$
U must be derived for particular problems and can take different forms, several of which are presented in[16]. Several of these forms are redundant with other level 2 metrics, for example, in the experiment presented in[16], this metric becomes identical to practical bit rate.
*
1. c: time per selection; N: number of choices; ITR: information transfer rate; P: probability of correct selection.
2. *: Not currently practical, but can be with future work.
3. Check marks indicate that the metric fulfills the evaluation criterion.