Sinan Guo, Wanchen Jia, Lin Ding, Chongchong Miao (2024). Influences of Information Processing Modality on Mental Workload Recognition Performance Based on EEG Feature Extraction. In: Neuroergonomics and Cognitive Engineering. AHFE Open Access, vol. 126, pp. 171–179. https://doi.org/10.54941/ahfe1004750
Accurate recognition of mental workload is significant for optimizing the human-machine interaction and avoiding the regrade of task performance levels due to overloading or underloading of mental workload. In past studies, the use of Electroencephalogram (EEG) signals has shown high performance in the recognition of operators' mental workload levels, however, most of the studies were conducted using a single visual modality task or dual visual modality tasks. But in real-world operational tasks, auditory-visual modalities tasks are commonly involved, and there have been relatively few researches on the EEG recognition of mental workload levels in auditory-visual modalities tasks. Therefore, in this research, visual single modality task scenario and audio-visual dual-modality task scenario were set up based on simulated flight experiments. For each task scenario, two levels of mental workload were induced by the differences in task complexity. Twenty subjects were recruited and their NASA-TLX scales and EEG signals were collected during the experiment. Two types of feature extraction methods were used, including Power Spectral Density (PSD) and Common Spatial Pattern (CSP), to recognize the mental workload levels. The research results indicated that the information processing modality did not have a significant influence on the performance of recognition for mental workload based on EEG feature extraction.
Currently, with the continuous introduction of intelligent and informa- tization technologies into the aircraft cockpit human-machine interaction process, the pilot’s Information Processing load has risen dramatically, and the information processing modality has been enriched. It is likely to create problems such as high pilot mental workload, insufficient attention resources, and decreased situational awareness, and may eventually increase the poten- tial risk of human errors by pilots, resulting in operational effectiveness that does not satisfy the design requirements. Accurate recognition of men- tal workload is significant for optimizing the human-machine interaction
and avoiding the regrade of task performance levels due to overloading or underloading of mental workload.
Typical methods for measuring and evaluating mental workload include the subjective evaluation methods, the primary and secondary task per- formance evaluation methods, and the physiological evaluation methods (Wickens, 2008). Among the above methods, the subjective evaluation method and performance evaluation method are constrained by the fact that most of their evaluation mechanisms are post hoc measurements and cannot meet the requirements of real-time measurement and evaluation. Compared with the above two types of measurement methods, physiological evalua- tion methods are characterized by objectivity, real-time measurement, and less interference with the subjects, which has been applied more and more in the recent studies. Among the physiological signals, such as electroen- cephalogram (EEG), ocular electrodynamics, cardiac electrodynamics, and respiratory signals, EEG has been widely used in the study of mental work- load recognition because of its sensitivity to mental workload levels, high temporal resolution, and excellent applicability.
In past studies, the use of EEG signals has demonstrated good perfor- mance in recognizing the mental workload of operators, however, the tasks selected are generally single modality tasks (Dai, 2017), whereas pilots need to deal with the multi-modality information processing tasks, includ- ing the processing of visual and auditory information, in real operations. In this study, two types of experimental scenarios were set up to address the above problem, including visual single modality tasks and audio-visual dual-modality tasks, and high and low mental workload levels were induced for the two types of tasks, respectively. Based on the EEG signal data and NASA-TLX rating scale score, we conducted a research on the recognition of pilot’s mental workload, focusing on the performance of mental work- load recognition for multi-modality tasks with different feature extraction methods.
METHODS
Experiments In this study, 20 subjects completed two types of task scenarios which were set up based on simulated flight experiments (see Figure 1). In Task scenario 1, subjects only need to perform a simulated flight task based on single visual modality. In Task scenario 2, subjects need to process an alarm task based on auditory modality apart from the simulated flight task based on single visual modality. For each task scenario, two levels of mental workload were induced by the differences in task complexity. In low mental workload tasks, subjects were only required to complete a basic Airfield traffic pattern task, while in high mental workload tasks subjects were required to complete a complex ground-attack task.
Influences of Information Processing Modality 173
Figure 1: Task scenarios based on simulated flight experiments.
For the Airfield traffic pattern task, the subject only needs to depart from the initial position and return to the initial position after 4 turning points. For the ground-attack task, the subject needs to complete a series of complex operations such as holding the flight attitude, setting the avionic mode, setting the missile delivery mode, setting the radar mode, setting the bomb launching mode, searching for the target, and dropping the bomb. The auditory information processing task requires the subject to recognize and respond to audio alarms. The detailed experimental paradigm design is shown in Table 1. In order to eliminate practice and fatigue effects, the experimental sequence was based on a Latin-square design.
Data Acquisition The data collected in this experiment included EEG signals and NASA-TLX rating scale score. The 30-channel EEG signals were acquired using the Neu- roscanNeuamps EEG acquisition system (Compumedics Limited; Victoria, Australia), and all electrodes were made of Ag/Agcl, with 0–200 Hz record- ing bandwidth and 1000 Hz sampling rate. The left mastoid A1 was used as the online reference electrode. Vertical and horizontal eye electrooculograms were recorded simultaneously.
Data Preprocessing Band-pass filtering (1-30 Hz) was first performed on the raw data, and subsequently independent components analysis (ICA) was used to remove blink and eye movement artifacts during the experiment. After completing the independent components analysis, the correlation levels of the decom- posed components with the horizontal and vertical ocular electrodes were calculated, and the independent components that were highly correlated
with the ocular electrodes were identified and removed as ocular artifacts (see Figure 2), and then the remaining independent components were recon- structed. The EEGLab toolbox was invoked in the data preprocessing phase of this research.
Figure 2: Independent components highly correlated with ocular electrodes (red).
Feature Extraction Based on the EEG data, the frequency domain feature extraction method and the spatial filtering feature extraction method were selected. For the fre- quency domain feature extraction method, the Power Spectral Density (PSD) method is selected to construct the feature space; while for the spatial filter extraction method, the Common Spatial Pattern (CSP) method is selected to construct the feature space.
Power Spectral Density The EEG data was divided into segments containing 1024 sampling points and the neighbouring segment step length was 512 sampling points. The power spectral density of each segment was calculated using Equation (1).
× FFT∗
X(n)
X(n)
Pk = FFT
n × fs , k = 0, 1, . . . , N −1 (1) X(n) Where Pk is the power spectral density of each EEG data segment, FFT
is the Fast Fourier Transform process of a random signal sequence X(n) of length n, FFT∗
is the covariance expression of FFT
X(n)
X(n)
, n is the num- ber of sampling points in a single segment, which is 1024 here, and fs is the sampling rate, which is 1000 Hz here.
The energy features of EEG in delta (1-4Hz), theta (4-8Hz), alpha (8-12Hz) and beta (13-30Hz) rhythms were calculated on the basis of Equation (2) to (5).
Equation reproduced from the original PDF.
Influences of Information Processing Modality 175
Equation reproduced from the original PDF.
Where Pk is to represent the power spectral density in discrete frequency bands, and the energy features of the EEG signals in the four rhythms of Edelta, Etheta, Ealpha, Ebeta are acquired by summation. Since the channel dimension of the EEG signals acquired in this case was 30, and the energy features under 4 rhythmic bands were also extracted corresponding to each channel, the dimension of the feature vector obtained by the application of the power spectral density feature extraction method was 120.
Common Spatial Pattern The original common spatial pattern algorithm is a type of extraction method for the spatial filtering features in the binary classification condition, which can obtain a specific spatial distribution composition from the high- dimensional EEG signals acquired by multiple channels. The basic concept of the common spatial pattern algorithm is to obtain a set of optimal spatial filters for projection by using diagonalization of matrices, which could max- imize the difference between the two classes of signals after being processed by the optimal spatial filters, and then acquire the feature space with a large differentiation.
The input of the initial EEG signals used in this research consisted of data from the C3, Cz, C4, Fz, and Pz electrode points, for the reason that the data acquired from the above electrodes have demonstrated sensitivity to changes in mental workload levels in prior researches. In addition, 1024 sampling points were selected for single-segment, and the data step length of each seg- ment was 512 sampling points. The final the dimension of the feature vector extracted by the common spatial pattern algorithm was 4.
Mental Workload Recognition Support Vector Machine (SVM) was firstly proposed as a linear model for binary classification, subsequently, after a series of development, it is currently widely used in classification in complex cases of nonlinear classifi- cation problems. In practical classification cases, the data are often linear and indivisible, and the SVM model shows good performance for these cases. The model classification performance and specific generalization ability of SVM depend directly on the kernel function as well as the induction of slack vari- ables. The slack variable represents the distance at which the discrete points are separated, and in order to resolve the misclassification problem of such discrete points, a penalty factor is introduced to regulate it, and the larger the value of the penalty factor is, the higher the importance of the discrete points for the classification process of the model is. In addition, for nonlinear sample data, the support vector machine could introduce a kernel function to transform the input features into a higher dimensional feature space to realize the linear separability of the sample data. Among many kernel functions, the Radial Basis Function (RBF) has excellent classification performance, so the RBF method is selected to construct the model in this paper. The combination of grid optimization and k-fold validation is applied to optimize the penalty factor C and the RBF parameter gamma.
In this paper, the GridSearchCV Function of the scikit-learn toolbox was invoked to implement grid optimization and 4-fold cross-validation. Specifi- cally, the range of values of the penalty factor C is [0.001, 0.01, 0.1, 1, 100] and the range of values of the RBF parameter gamma is [0.1, 1, 10, 100, 1000]. Because of individual difference in EEG signals between subjects, this research was based on building a separate classifier model for each subject.
RESULTS
NASA-TLX Scale Score Results Descriptive statistics for the visual single modality task scenario under the low and high mental workload levels were 45.1±13.6 and 55.4±12.4, and for the audio-visual dual modality task scenario under the low and high mental workload levels were 49.1±13.0 and 61.2±14.2. The results of the T-test indicated that in the visual single modality task scenario, NASA-TLX scale scores of high mental workload tasks were significantly higher than scores of low mental workload tasks (t=-3.559, p=0.002), and similarly, the audio visual dual-modality task scenario showed this trend (t=-3.718, p=0.001). This result suggested that the experimental design successfully induced two levels of mental workload.
Mental Workload Recognition Results Based on the PSD features, the mental workload recognition accuracy results for the visual single modality task were 0.8914±0.0698; for the audio-visual dual-modality task, the mental workload recognition accuracy results were 0.8752±0.0646 (see Figure 3). T-test results showed that for the two types of task scenarios, there was no significant difference in the mental workload recognition accuracy results using the PSD features (p>0.05).
Figure 3: Mental workload recognition accuracy results based on the PSD features.
Based on the CSP features, the mental workload recognition accuracy results for the visual single modality task were 0.8630±0.0654, and for the audio-visual dual- modality task were 0.8330±0.0762 (see Figure 4). The T-test results showed that for the two types of task scenarios, there was no
Influences of Information Processing Modality 177
significant difference in the mental workload recognition accuracy results using the CSP features (p>0.05).
Figure 4: Mental workload recognition accuracy results based on the CSP features.
The mental workload recognition accuracy results using the PSD and CSP feature extraction methods in different task scenarios are shown in Table 2. Specifically, the T-test results showed that there was no significant difference (p>0.05) between the mental workload recognition performance results using both PSD and CSP features in the visual single modality task scenario, and the mental workload recognition performance results using both PSD and CSP features in the audio-visual dual-modality task scenario exhibited marginal significance difference (t=2.062, p=0.053).
Table 2. Recognition accuracy results using the PSD and CSP feature extraction methods.
In this study, two types of task scenarios, visual single modality and audio- visual dual-modality, were set up, and the results of the NASA-TLX subjective scale scores indicated that both types of task scenarios induced high and low levels of mental workload through Experimental design of task complexity.
On this basis, for both types of task scenarios, using different feature extraction methods of this research showed good mental workload recogni- tion performance in comparison with related studies. The comparison results of recognition performance were shown in Table 3.
Table 3. Comparison results of recognition performance with related studies.
Table available in the original PDF paper.
Ref. Task Scenario Feature Category Feature Accuracy PSD 0.89/0.88 This study Single modality/ Dual-modality
Frequency Domain Spatial Domain CSP 0.86/0.83 Single modality/ Dual-modality
0.5
(Dehais, 2019) Flying task Time Domain event-related potentials (ERP)
(Dimitrakopoulos, 2017) N-back Functional Connectivity Features
0.82
Phase-locking index
(Kakkos, 2019) Simulated flight Functional Connectivity Features
Meanwhile, there was no significant difference in the recognition perfor- mance of mental workload using two types of feature extraction methods in this research for both task scenarios. To analyze the possible reasons, the audio alarms in this experiment were brief and discrete which did not significantly affect the recognition performance.
The recognition accuracy using PSD as features showed a slightly greater trend than CSP, and this trend was more pronounced for the audio-visual dual-modality task scenario. However, the dimension of PSD features in this research was much larger than the dimension of CSP features, so the CSP fea- ture extraction method effectively reduces the complexity of mental workload recognition computation when the difference in recognition performance is not too significant.
For different subjects, the mental workload recognition performance also showed certain difference, to analyze the reason, the mental workload level itself is affected by the individual differences of the subjects obviously, the workload sensitivity of different subjects to the external task changes is not consistent, and the mental workload labels were set uniformly according to the experimental design, so it caused the individual difference problems in the recognition performance.
CONCLUSION
Based on PSD and CSP feature extraction methods, the recognition accuracy of mental workload for both visual single modality task and audio-visual dual- modality task did not demonstrate a statistically significant difference, indicating that the information processing modality does not have a sig- nificant influence on the performance of recognition for mental workload based on EEG feature extraction. Additionally, using the PSD feature extrac- tion method demonstrated a slightly better performance on mental workload recognition compared to CSP.
Influences of Information Processing Modality 179
ACKNOWLEDGMENT
The authors would like to acknowledge.
REFERENCES
Albuquerque, I., Tiwari, A., Gagnon, J. F., Lafond, D., Parent, M., Tremblay, S. and Falk, T. (2018), October. On the analysis of EEG features for mental workload assessment during physical activity. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 538–543). IEEE. Dai, Z., Bezerianos, A., Chen, A. S. H. and Sun, Y. (2017). Mental workload
classification in n-back tasks based on single trial EEG. Dasari, D., Shou, G. and Ding, L. (2017). ICA-derived EEG correlates to mental fatigue, effort, and workload in a realistically simulated air traffic control task. Frontiers in neuroscience, 11, p. 297. Dehais, F., Duprès, A., Blum, S., Drougard, N., Scannella, S., Roy, R. N. and Lotte, F.
(2019). Monitoring pilot’s mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors, 19(6), p. 1324. Dimitrakopoulos, G. N., Kakkos, I., Dai, Z., Lim, J., deSouza, J. J., Bezerianos, A. and Sun, Y. (2017). Task-independent mental workload classification based upon common multiband EEG cortical connectivity. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(11), pp. 1940–1949. Ding, Y., Cao, Y., Duffy, V. G., Wang, Y. and Zhang, X. (2020). Measurement
and identification of mental workload during simulated computer tasks with multimodal methods and machine learning. Ergonomics, 63(7), pp. 896–908. Dong, M., Feng, Y. and Feng, H. (2015). Psychophysiological measures based studies on mental workload assessment and adaptive automation: review of the last 40 years and the latest developments. J Electron Measur Instrum, 29(1), pp. 1–13. Fan, X., Zhao, C., Zhang, X., Luo, H. and Zhang, W. (2020). Assessment of men-
tal workload based on multi-physiological signals. Technology and Health Care, 28(S1), pp. 67–80. Kakkos, I., Dimitrakopoulos, G. N., Gao, L., Zhang, Y., Qi, P., Matsopoulos, G. K., Thakor, N., Bezerianos, A. and Sun, Y. (2019). Mental workload drives different reorganizations of functional cortical connectivity between 2D and 3D simu- lated flight experiments. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(9), pp. 1704–1713. Roy, R. N., Charbonnier, S., Campagne, A. and Bonnet, S. (2016). Efficient men-
tal workload estimation using task-independent EEG features. Journal of neural engineering, 13(2), p. 026019. Wickens, C. D. (2008). Multiple resources and mental workload. Human factors, 50(3), pp. 449–455. Yi-run, P. A. N. and Yu-fan, P. A. N. (2015). Recognition method of driving mental
workload based on EEG entropy. Journal of Southeast University (Science and Technology), 45(5), pp. 980–984.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Neuroergonomics and Cognitive Engineering
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-02-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional