1. Introduction

Augmented reality (AR) or mixed reality (MR) approaches to augment the performance of device operators in several settings have attracted significant research interest and are seen as one of the main enablers of Industry 4.0 ^[1]. For industrial application scenarios in particular, the overlay of context-dependent information offers opportunities for new occupational configurations or training of the operators of AR/MR devices on-the-fly. This amalgamation of realities subsequently offers great potentials in various industry settings alike to positively modify human performance, indicated, e.g., by ^[2] for driving tasks, by ^[3] and ^[4] in the context of medical procedures, by ^[5] for museums and tourism, or by ^[6] for educational purposes. Psychological impacts, however, could yield a potential negative effect from the utilization of these devices types. The modification of the device operator performance could, for example, be negative due to too much information that requires processing. The result of this could be a negative performance impact due to increased cognitive strains for an operator.

We target this potentially negative impact scenario through an evaluation of a task that requires AR device operators to perform classification of media qualities. Specifically, operators were asked to rank the quality of differently impaired images presented in an AR setting, which requires different effort for discrimination of levels. One can consider the scenario in real life applications with the need to discern details in presented information, such as maintenance instruction or images detailing specific device features. A close approach is detailed in ^[22], where an AR content generating module creates virtual instructions from the assembly sequence of the specific product. In turn, untrained generalists become enabled to perform detailed activities otherwise requiring specialized training. However, this approach still requires detailed computer aided design files to derive the overall content. Our approach could be thought of as a more general version, where content is presented directly, e.g., in form of screen-shots or product manual images, or the likes. In turn, our contributions are $(i)$ an evaluation of the overall cognitive strain of AR device operators when performing tasks and $(ii)$ a classification into high and low cognitive load levels allowing for real-time adjustments of an operator's environment when performing tasks. While we focus on the content presentation components in this contribution, we note that in industrial settings, stressors can originate from a multitude of sources, such as limited time or noisy environments. Our study, thus, serves as an initial foray into the direct quantification of cognitive strain.

Several investigations into the capturing of cognitive strains were made in the past, with significant efforts put forth in educational contexts. For these scenarios, the skill acquisition of learners in any type of setting (including device operators in industrial environments acquiring new skills on-the-fly) represents a significant challenge. Learning theories, such as Cognitive Load Theory ^[7,8,9], provide a framework of memory and brain activities. The working memory and interdependent cognitive load has also attracted research efforts aiming at quantification of the underlying processes. Due to large inroads in human-computer interaction (HCI) that evolves around brain-computer interfaces (BCI) supported by electroencephalography (EEG), measurements of cognitive processes have become popular in research. For critical tasks, such as those where AR/MR might find explicit implementations in industrial processes, the impact that the HCI poses on a device operator through germane required cognitive processes, a deeper understanding of interplays is required ^[10]. While initial implementations of environmental augmentation under consideration of cognitive aspects have been proposed since quite some time, see, e.g., ^[17], these initial approaches required modification of the environment. Only in recent years have we witnessed an emergence of wearable augmented reality devices that allow for a blending of the physical world and information provided. In turn, considerations about the amount of information and its display within AR contexts has an interplay with the cognitive strain in the situation of utilization. Past research efforts were made to target this potential problem for industrial application scenarios. In ^[19], the authors evaluate task complexity for product development in a VR context with the help of graph theoretic approaches. Augmented reality considerations in design and manufacturing were showcased by ^[20] in an overview. More detailed, in ^[21], the authors introduce AR as a tool into the factory assembly workflow, employing markers and situational awareness to enable safe human-robot interaction during assembly processes.

Evaluations of computer utilization and cognitive or mental workloads has been the subject of research for quite some time. For example, frontal alpha and theta bands were identified as related to task difficulty, see, e.g., ^[11]. More recently, in ^[14] the authors analyzed the spectral analysis of EEG data with a focus on the alpha, beta, and theta bands. They find frontal cortex activity is high for cognitive work cycles and that the level of workload can be found reflected in the spectral density power levels. Similarly, a combination of theta and alpha bands was found to react to load levels in ^[13], where different load levels were successfully identified by a combination of frontal and parietal sensor locations. Specifically investigating the cognitive load in multimedia learning situations, it was found that different regions of the brain were activated for various tasks in different frequency bands in ^[12].

The recent advances in BCI allow for a reduced number of sensors to be utilized and similarly, dry electrodes enable non-intrusive designs to emerge in the near future. Specifically in this contribution, we consider a headband worn jointly with a head-worn AR device, which enables a combination of AR display and real-time sensing of EEG data when considering both current devices to be combined in future iterations. Such evaluations, though real-time, are actually nonintrusive in the utilization scenario with the benefit of real-time accessibility, whereas traditional experience sampling, such as with the NASA-TLX approach ^[18], are non-real-time and require an active participation of the human subjects.

The remainder of this article is structured as follows. In the following section, we describe our overall approach to the performance analysis. Our results are presented in Section 3 before a discussion and conclusion in Section 5.

---

2. Materials and Method

In this section, we describe our overall approach in an overview first before detailing our data analysis methodologies.

---

2.1. Experimental Data

The data we consider throughout this article is derived from human subject experiments and publicly available, see ^[15]. Specifically, participants were asked to rate multimedia quality levels of images (traditional and spherical) in a general meeting room setting with dimmed light. While the specific task given to the participants aims at determination of media quality, inherently the difficulty level for such rating can be extrapolated from the visual system based classification tasks required. In turn, we reason that the task difficulty level is especially complicated when considering unclear quality levels, derived from the visual fidelity of the media presented. In ^[15], the motivation was to determine the media quality from the captured EEG data, whereas in this paper, we evaluate the cognitive strain based on the complexity of the rating task.

The overall classification task asked participating human subjects to classify the quality of the images presented into 5 different quality levels on a Likert-type scale. Commonly, original and lowest impairment levels of media show no significant visual degradation and are easily classified. Similarly, the highest level of visual impairment also is relatively easy classified due to significant visual impairments. The harder to determine medium ranges of media fidelity, however, typically pose an increased cognitive load, as the classification in the medium ranges of visual impairments into three distinct categories can pose significant challenges. We subsequently map the different image levels to two cognitive load states, namely $(i)$ no impairment, lowest impairment, and highest impairment levels are mapped to a low mental workload, whereas $(ii)$ impairment levels 2–4 in the VIEW datasets ^[15] are mapped to a high mental workload.

We employ the available information contained in the dataset, which includes measurement data from four dry electrodes in the temporal-parietal and anterio-frontal positions (TP9, AF7, AF8, and TP10), illustrated in Figure 1.

We particularly focus on these electrode positions, as they could be readily integrated into future revisions of wearable AR device employing dry electrodes. Specifically, due to the sensor locations and wearable nature of AR devices, a merging of the currently employed headband and AR device would result in the ability of non-intrusive continuous monitoring in the future. The available data includes several different channel band measurements at the respective positions, namely:

• Low from 2.5–6.1 Hz,

• Delta from 1–4 Hz,

• Theta from 4–8 Hz,

• Alpha from 7.5–13 Hz,

• Beta from 13–30 Hz, and

• Gamma from 30–44 Hz.

As described in prior research efforts, especially the alpha, beta, and theta bands are of interest to identify different levels of cognitive load.

---

2.2. Performance and Data Analysis

Motivated by the prior advances in the determination of workload levels in the literature, e.g., in ^[13], we employ the results from the VIEW datasets by utilizing basic machine learning approaches. Let the image presentation for a particular image i at workload level l, $l \in \left\{ {0, 1} \right\}$ range from ${t_s}\left({i, l} \right)$ to ${t_e}\left({i, l} \right)$. We subsequently aggregate the measurement data points for alpha $\left(\alpha \right)$, beta $\left(\beta \right)$, and $\theta$ frequency bands at the different positions p as follows based on the coefficient of variation:

$\alpha _p^c = \frac{{1 + \sigma \left( {{\alpha _p}} \right)}}{{{{\bar \alpha }_p}}},$

(1)

where ${\sigma \left({{\alpha _p}} \right)}$ denotes the standard deviation of the alpha band power and ${\sigma \left({{\alpha _p}} \right)}$ denotes the corresponding average at a specific sensor position. Noting that the beta and theta band values $\beta _p^c$ and $\theta _p^c$ are determined similarly, we employ these to determine their respective ratios. The approach results in four different scenarios, namely:

1. ${}^{{{{\bar{\alpha }}}_{p}}}\!\!\diagup\!\!{}_{{{{\bar{\theta }}}_{p}}}\;$ employing the average alpha/theta bands ratio to lean on past research efforts,

2. ${}^{{{{\bar{\alpha }}}_{p}}}\!\!\diagup\!\!{}_{{{{\bar{\beta }}}_{p}}}\;$ employing the average ratio of alpha and beta,

3. ${}^{\alpha _{p}^{c}}\!\!\diagup\!\!{}_{\theta _{p}^{c}}\; =~\frac{{{{\bar{\theta }}}_{p}}\left(1+\sigma \left({{\alpha }_{p}} \right) \right)}{{{{\bar{\alpha }}}_{p}}\left(1+\sigma \left({{\theta }_{p}} \right) \right)}$ employing the variability of alpha and theta activity, and

4. ${}^{\alpha _{p}^{c}}\!\!\diagup\!\!{}_{\beta _{p}^{c}}\; =~\frac{{{{\bar{\beta }}}_{p}}\left(1+\sigma \left({{\alpha }_{p}} \right) \right)}{{{{\bar{\alpha }}}_{p}}\left(1+\sigma \left({{\beta }_{p}} \right) \right)}$ employing the variability of alpha and beta activity while under testing conditions.

The resulting four position-dependent ratios are utilized as our main evaluation criteria using the k-nearest-neighbor (KNN) classification and regression approach to evaluate whether the ratio levels can be correctly employed to identify the different cognitive load levels. We selected the KNN approach as for this particular contribution, we are evaluating a basic classification approach to determine the workload levels of the subjects under consideration based on directly gathered sensor information.

We configure the classifier for 3 neighbors utilizing the distance between neighbors as weights, whereby the linear combination of the four position ratios is used. We note that in our evaluations, we evaluated different numbers of neighbors, but found that three was a sensible trade-off. We calculate the performance of the classification into high and low cognitive load levels based on the number of true positive (TP), true negative (TN), false negative (FN), and false positive (FP) classification and/or prediction results. Similarly, we determine the suitability of the classifier on the entire dataset using the R² score, which in the case of this binary classification and prediction is the same as the precision attained by the approach.

---

3. Results

In this section, we describe our results for the regression and prediction of the QoE/cognitive load interplay in greater detail. We initially focus on the complete set of experiments and users before separating them into the traditional AR-VIEW and immersive SAR-VIEW scenarios. Furthermore, we consider the four outlined scenarios, taking the overall levels and/or variabilities in the different frequency bands into account.

---

3.1. Combined Users

The initial classification for all subjects' cognitive load performance indicator for 1169 cases results in a perfect outcome, i.e., all cases are successfully classified as TP (585) or TN (584). Next, we consider the prediction of an individual user based on the KNN training set of all other users. This mimics the approach of predicting a new subjects' cognitive state from already known ones. In this scenario, we have to again differentiate between the four outlined approaches under consideration. The overall results are provided in Table 1.

We observe that overall, the number of wrongly classified events is similar for all scenarios with only slight deviations. The inter-subject variability indicated by the standard deviation, however, suggests that these average results can vary significantly on a per subject basis. This is in line with the overall expected result for such a classification and prediction approach for highly subjective source data. Based on the R²score, our proposed ${{\bar \alpha }_p}/{{\bar \beta }_p}$ metric results in the highest average R²score overall.

We additionally consider the individual prediction of a subject's cognitive load based on the subject itself. Here, we perform a random 80/20 split of the available respective subject data into training and test data. Subsequently, we evaluate the performance of the KNN approach as in prior cases. We repeat the evaluation until the R²/accuracy score's 95% confidence interval width is within 5% of the mean. The results are presented in Table 2. We again observe a large spread of accuracies for the individual predictions across subjects. For example, the R² for ${{\bar \alpha }_p}/{{\bar \theta }_p}$ exhibits a spread from 0.343 for subject 0 to 0.671 for subject 19. Overarching, our proposed variability metric $\alpha _p^c/\beta _p^c$ approach yields the highest level of prediction accuracy based on individual subjects themselves (M = 0.473, SD = 0.08). One potential drawback here can be the number of available data points on a per-subject basis, limiting the overall attainable results.

---

3.2. Individual Image Types

While the overarching evaluation yields interesting results based on the brain activity of participating subjects, we additionally perform an evaluation based on the different image types. The motivation is to ensure that different media presentation mode effects could be captured separately, if required.

---

3.2.1. Traditional Images

For the 629 measurement points for traditional images (42 per subject with outliers), we again obtain perfect classifier performance. When shifting the prediction approach to those from others, we note an overall significant reduction in TP and TN, with the corresponding overview provided in Table 3.

We initially observe that the overall levels attained for the prediction are generally comparable to those stemming from the combined base data for the KNN algorithm. Closer inspection, however, reveals that our proposed variability level metric $\alpha _p^c/\beta _p^c$ results in the highest overall accuracy level on average (M = 0.53, SD = 0.09), while other metrics involving $\theta$ remain the same and the ${{\bar \alpha }_p}/{{\bar \beta }_p}$ even performs slightly worse.

---

3.2.2. Immersive Images

Lastly, for the 540 measurements from spherical images (36 per subject), classifiers result again in perfect performance. The subsequent prediction results exhibit similar trends to those prior observed for TP and TN, as presented in Table 4. Specifically, we note that while overall levels remain comparable, in this scenario our proposed ${{\bar \alpha }_p}/{{\bar \beta }_p}$ relationship yields the best prediction performance (M = 0.54, SD = 0.08).

---

3. Discussion

We now review our results, providing an additional view of the high-level R² outcomes for the evaluation in Figure 2. We note that our approach utilized a general view on three common bands of brain waves, evaluated at readily accessible positions for head-worn devices. Additionally, the KNN approach we employed throughout our contribution has the benefit of being employable in unsupervised learning scenarios, making it well suited for ad-hoc situations.

We note that on overall average, the ${{\bar \alpha }_p}/{{\bar \beta }_p}$ could be employed for an estimation of the cognitive high/low load levels and would yield better results than individual user approximations. (Noting the caveat of limited measurement data points for individual users in the underlying dataset.) Thus, an overall pool of measurements has the increased benefit of enabling future prediction scenarios. The prediction of high/low cognitive loads, in turn, enables an estimation of the performance of AR device users in several upcoming utilization scenarios. If, however, the type of media presented to users is known, for regular (non-immersive) media content, the ${{\bar \beta }_p}\left({1 + \sigma \left({{\alpha _p}} \right)} \right)/{{\bar \alpha }_p}\left({1 + \sigma \left({{\beta _p}} \right)} \right)$ relationship does provide an increase in the prediction performance over the alternative approaches we evaluated here.

Given the range of R² values we observed here, we note that this initial attempt of modeling the cognitive load in AR scenarios could be improved further. For example, additional user-specific knowledge could yield further increases in accuracy of predicted cognitive level states. Furthermore, other sensor positions that would still be able to be combined with head-worn devices for actual task usage could be considered as well.

The overall real-world impact for future industrial usage of media presented to AR device users is given as follows. Consider a scenario where users are to be augmented with respect to their capabilities, e.g., to perform tasks they did not have encountered before. Examples for such tasks include maintenance and operations, with instructions directly taken from manuals. As instructions are sent to the mobile user in form of multimedia (images, illustrations, etc.), in-network content adaptation is required to enable timely content delivery to the mobile device for timely display. As content is prone to further compression, additional cognitive strain is resultant for the device wearer leading to reduced task performance. We have successfully demonstrated a fist evaluation of how the impact could be quantified and, subsequently, be acted upon. Additional research is needed, however, to quantify the impacts in real-life industrial scenarios, such as ad-hoc maintenance and operations for machinery with the help of displayed AR content (e.g., maintenance manual pages or pictorial representations).

---

4. Conclusion

As the utilization of AR solutions in industrial settings increases, the content optimization for delivery and its display in interplay with the impact that these have on the operator performance will increase in importance. We provided a new evaluation of the cognitive load or strain for augmented reality device operators with regular and immersive image content, finding that a KNN approach yields R²scores around 0.5 and above in most cases, but individual prediction of the cognitive load for device operators is less accurate.

In ongoing research, we are evaluating virtual reality scenarios in addition to augmented reality scenarios. Furthermore, we consider different sensor placements and other forms of content suitable for future industrial scenarios of user capability augmentation. Another interesting avenue of follow-up research is the inclusion of different task scenarios and additional comparisons of experience sampling approaches and secondary indicators for cognitive load.

---

Acknowledgments

This material is based upon work supported by the Faculty Research and Creative Endeavors (FRCE) program at Central Michigan University under grant #48146.

---

Conflict of Interest

The authors declare that this is original work not under review anywhere else and that there is no conflict of interest.

---