N. Firat Özkan, Emin Kahya (2020). Measurement of Cognitive Workload by Use of Combined Methods Including Brain-Computer Interfaces. In: Advances in Social and Organizational Factors. AHFE Open Access, vol. 9. https://doi.org/10.54941/ahfe100358
Cognitive load measurement is a challenging research area that many methods have been provided to get more accurate results. Limitations which are pointed by Cognitive Load Theory (CLT) make this area both special and difficult. Subjective, behavioral, and physiological methods all have some advantages and disadvantages. That’s why the use of combined methods has been increasing to make more accurate and better cognitive workload predictions. Technologies such as eye tracking systems, physiological and physical parameters monitoring systems are able to support the relevant researches in this area. On the other hand, there have been some developments which aim different benefits. One of them is Brain-Computer Interface (BCI) technology that works based on Electroencephalography (EEG), detect P300 component of Event Related Potentials and Steady State Evoked Potentials (SSVEP) and offer motor imagery applications. Therefore, they have potential benefits to support cognitive workload measurement methods. In this study, relationships between brain potentials and cognitive workload are researched. Further, opportunities to create new combined cognitive workload measurement methods including BCI, eye-tracking, physiological, and physical parameters are discussed.
Reducing human errors is one of the most important areas relevant to human factor and ergonomics. Accidents which are caused by human errors may be fatal sometimes or they slow down the progression of systems. Since human beings have limited physical and mental capacity, work environments and human machine systems must be designed considering the human limits. Measuring physical capacity and fatigue is relatively easier than measuring cognitive components. That’s why a lot of studies about cognitive ergonomics have been done in recent years. Increasing technology helps researchers to use more various tools such as BCI and eye trackers in their studies. There are no any methods to measure cognitive load certainly. However it is possible to increase reliability of estimations by using hybrid methods.
Cognitive aspect of human beings’ performance is not only depends on their cognitive workload caused by the task. There are some other important cognitive factors such as demands and difficulties of the task, environmental and organizational stressors and personal mental strain of human beings. That’s why especially for highly critical tasks it may be better to monitor the cognitive state of the human beings before the task instead of measuring how much they are loaded mentally during the task.
COGNITIVE LOAD Cognitive load has a lot of definitions. However, it can be shortly defined as the relationship between requirements of the work and competences of the person. When cognitive load improves, risks of accidents decrease and satisfaction of the workers increases. Since there are lots of components of cognitive load it is extremely hard to measure it. However, there are three main factors that affect cognitive load presented below:
- Specifications of the work, - Skills and qualifications of the person, - Environmental factors.
Tracy and Alberts (2006) state that when the cognitive load increases, the ability to perform effectively slowly decreases until the person reaches a point of cognitive overload.
According to Hussain et al. (2011) monitoring cognitive load is crucial for developing adaptive systems aware of the user’s mental workload. Because such systems can reduce error related risks during task-critical operations.
Cain (2007) states that, a commonly accepted definition of workload does not exist. However, it is possible to characterize workload as a mental structure that reflects the mental strain caused by performing a task under specific environmental and operational conditions, besides the capability of the operator to respond to demands of the task.
Although there is no any widely common definitions of cognitive load most of the different approaches reach the same point indicating the relationship between cognitive load and work performance. That’s why measurement of cognitive load has an critical importance
COGNITIVE LOAD MEASUREMENT It is possible to summarize cognitive load measurement methods in three groups as:
Subjective methods reflect the person’s opinions and feelings directly. Participants answer some questions right after the completion of the task and a quantitative result is obtained based on their answers. NASA-TLX (Hart and Staveland (1988) is the most common subjective cognitive load measurement tool. Since these tools are easy to use and give a direct score lots of researches prefer to use them. Figure 1 illustrates an example of computer-based NASA-TLX application.
Figure 1. A Sample of Nasa-TLX Screens (www.nasatlx.com)
NASA-TLX has six sub-scales; mental demand, physical demand, temporal demand, performance, effort, and frustration. These subscales can be grouped according to the three factors assumed to produce workload (Embrey, 2006). The first one is task factor including mental, physical and temporal demand sub-scales. The second one is the factor of behavioral and skill and it is related to performance and effort subscales. The last factor is individual and it includes the frustration subscale.
Behavioral methods are based on tasks performances. It is possible to use only one task and estimating the cognitive load by using task completion time and performance. However, to increase sensitivity, participants usually work on two tasks that called primary task and secondary task. Primary tasks create the cognitive load and secondary tasks are used as metric of the performance. The both tasks are performed concurrently then participant’s task completion time and the level of the success are considered to estimate the cognitive load.
Physiological methods are relatively more objective methods, because physical symptoms that are used to estimate the cognitive load are not directly controlled by the participant. Most of the symptoms such as hearth rate variability, galvanic skin response, pupil dilation are controlled by autonomic nerve system. On the other use of EEG is one of the popular tools on this area. Investigating change of brain waves and event related potentials (ERP) gives some information about the cognitive situation of the participants. In fact, all of these symptoms are also relevant to the level of stress and anxiety of the participant. Since stress and anxiety have relationship with cognitive load level these facts can be used to estimate the level of cognitive load.
There are a lot of empirical and non-empirical studies that aim to measure cognitive load in the literature.
Antonenko et al. (2010) focused on comparing using NASA-TLX and EEG data and stated using EEG data is a more objective approach to measure cognitive load.
Haapalainen et al. (2010) stated using multiple physical symptoms makes the cognitive load estimation more realistic. The study is an experimental one and data were collected on 20 participants while they were completing the task given by the researchers. The physical symptoms are:
- Changes on pupil sizes, - Eye movements, - Number and duration of eye blinking, - Heart rate variability, - Respiration, and - Monitoring brainwaves.
Walcyzk et al. (2012) explain that there is a relationship between telling lies and pupil sizes. They conducted a scenario based experiment with 145 participants. Some of the participants watched a video record of real crimes and they were asked to tell lies about them. The rest of the participants were asked to tell truth. The results were compared using statistical analysis methods.
Based on the literature review, it can be stated that there is an increase in using remote eye trackers to investigate eye reactions as a metric of cognitive load. Palinko et al. (2010) conducted a study to estimate cognitive load of participants using driving simulator. Klingner et al. (2008) conducted three experiments about mental multiplication, short term memory and aural vigilation and they used task evoked pupillary response to measure participant’s cognitive load levels via remote eye tracker.
Eye trackers are technological systems using infrared cameras and following eye movements, fixations and pupil reactions. Figure 2 illustrates an example of screen type remote eye tracker.
Figure 2. Screen type eye tracker (www.tobii.com)
BRAIN COMPUTER INTERFACES (BCI)
BCI are technological systems including a computer, relevant software, EEG components (bonnet, electrodes etc.), and an amplifier. They work based on EEG and simply brain activities and convert them to comments for the computer using interfaces (See Figure 4). The goal of BCI research for the last thirty years has been to develope a brain-controlled communication device for paralyzed individuals who keep cognitive abilities (Mak et al., 2011). However it is also possible to use these interfaces for other cognitive researches related with ERP and other brain activities. Blankertz et al. (2010) published a review article to investigate non-medical use of BCI technology. In the study they mentioned about some possible areas to use BCI including mental state monitoring, cognitive load measurement and attention.
Figure 3. Operation Diagram of BCI (www.gtec.at)
Karagoz et al. (2005) explain that BCI’s basically use three main neurophysiological facts first of these facts is P300. This potential appears when a problem is solved, a decision is made or an uncertainty is gone in 300 ms. Alkac (2009) stated that P300 doesn’t appear with the standard stimulates but it appears with the target stimulates. That’s why this potential is used in the studies about gathering attention and focusing. The relevant interface of P300 is usually a letter matrix that columns and rows flash and the participant tries to focus on his/her target letter to type it on the screen (See Figure 5).
Since P300 is directly related with decision making, problem solving and detecting occasional target stimulus it is an attractive phenomenon also for cognitive load researchers. Schulteis and Jameson (2004) conducted an experimental study. The task of the experiment is reading written materials with various level of difficulty. They used an EEG, remote eye tracker and subjective scale to measure the cognitive load and according to the results more difficult texts led to lower reading speed, higher subjective load ratings, and a reduced P300 amplitude. Zhao et al. (2011) used P300 in their experimental study as an indicator of mental fatigue. In the study subjects completed a 90 min. Driving simulation task. In addition they were requested to perform an Oddball task at the beginning and at the end of the driving task.
Figure 4. Letter Matrix on BCI (www.gtec.at)
Another brain potential which is used by BCI’s is SSVEP. BCI’s use this potentials with flashing controls (See Figure 6). The controls flash with different frequencies and the participant tries to focus on her/his target control to give the comment. This application makes possible to control some electronic systems (Gencer et al., 2010). This application also requires a high-level of attention and concentration to be able to follow quick flickers. SSVEP has also become an attractive fact for cognitive load researchers. Pegelo et al. (2011) conducted an experimental study and they state that the test show SSVEP based BCI can be used for psychometric test.
Figure 6. A sample SSVEP Application (www.gtec.at)
The third neurophysiologic fact used by BCI’s is changes on motor sections of the brain when participant imagine moving to a specific direction (Gencer et al., 2010). When we think to move our right hand, an activity appears on the left side of our brain. BCI system catches this activity via electrodes and converts it to a comment for the computer. It makes possible to control cursor on the screen to respond visual stimulates. Creating the appropriate responses requires high level of attention and focusing.
Uses of some BCI applications are summarized at the Table 1.
Tablo 1. Summary of some BCI Applications
COMBINED METHOD Since the current cognitive load measuring methods are not enough to give an exact result, researchers tend to combine some of them in their studies to increase reliability. Some physical indicators of cognitive load, task performance results and subjective ratings are usually combined. Of course all methods have their own advantages and limitations (See Table 1) that’s why usually it is not possible to use more than a few methods concurrently.
Table 2. Summary of Some Cognitive Load Measurement Methods METHODS ADVANTAGES DISADVANTAGES
BCI’s are expected to bring new opportunities for combining methods to measure cognitive load. Their pre- developed interfaces and ability of tracing ERP make them potentially useful tool for cognitive researches. On the other hand applications of BCI that can be used as experimental tasks in the combined method are simple for everybody and they are not related to any personal qualifications or personal area of interest. That’s why task performances of this method are much more related to cognitive facts.
In this study, a BCI system developed by G.Tec is investigated. G.Tec’s BCI system works based on MATLAB and its amplifier allows using GSR, Respiration, and HR&HRV sensors as well. When a remote eye tracker and a subjective cognitive load method are added to the combination, a multi aspect process can be available.
BCI applications can be used as tasks and the tasks can be defined as follows:
- Letter Matrix (P300): Typing words, - SSVEP application: Making the target moves, - Motor Imagery: Giving the right directions to the cursor.
Possible steps of the method can be summarized as below:
- Completing BCI tasks respectively and recording physical data (HRV, GSR, Respiration, Pupil Size) concurrently, - Record the task performance (number of errors etc.), - Analysis of the data and comparing the results, - Conducting a subjective measurement method as basis.
There are some limitations of the methods:
- Not applicable in real work environment. Experiments should be conducted in a silent and properly arranged environment. - BCI tasks are not suitable to be used as a second task due to requirement of high level of attention and focusing. - Not very suitable to measure a specific real task’s cognitive load. It is better to estimate mental fatigue or monitor the person’s cognitive load just before or right after the real world task.
Strengths of the method can be summarized as follows:
- It is possible to trace most important physical cognitive load indicators concurrently,
- The task performance is almost completely related to attention and focusing level of the participant,
- It is possible to investigate and analyze EEG data as well,
- Task difficulties can be arranged by changing number and frequency of relevant stimuli,
- It makes possible to trace a person’s cognitive situation with periodical executions.
DISCUSSION AND CONCLUSION Relevant studies in recent years shows that cognitive load researchers tend to combine the existing methods. Developing technology brings some advantages to collect more sensitive data in shorter times.
Obtaining more realistic and objective results is required combining existing method and using the advanced technology. Nowadays, it is possible to give tasks to the participants with BCI. Tasks on the BCI’s eliminate the performance deflections caused by the qualifications and skills of participants. Because according to the Cognitive Load Theory, two persons may complete the same task at the same time with the same level of success. However, it does not mean that both of them have the same level of cognitive load. It is impossible to understand which of them is closer to make an error. When the personal differences and tendencies are eliminated by using such simple tasks as writing a word on the screen, moving a robot to somewhere and locating the cursor on the target point, it may be possible to see more clearly cognitive fatigue levels and capacities of the participants.
This study constitutes one of the initial steps in the exploration new combined cognitive load measurement methods including BCI’s. Future studies will include experimental data and statistical analyses to investigate the performance of proposed methods.
FUTURE WORK
The planned next steps of the study are listed as follows:
- Sectorial based data collection; experiments focus on the sectors that employees are under the high level of accident risk, - Age based data collection; since some cognitive skills may disappear in older age it is important to identify the age effect on the cognitive state, - Developing new risk analysis methods that including cognitive aspects.
REFERENCES
Alkaç U.,( 2009). “Beyin AraştırmalarıTarihinde Bir Gezinti:Elektronörofizyoloji”. Klinik Gelişim, 2009-3, 14-19. Blankertz, B., Tangermann, M., Vidaurre, C., Fazli, S., Sannelli, C., Haufe, S., Müller, K. R. (2010), “The Berlin brain–computer interface: non-medical uses of BCI technology” Frontiers in neuroscience, 4. Cain, B. (2007). “A review of the mental workload literature,” Defence Research And Development Toronto (Canada). Embrey, D., Blackett, C., Marsden, P., Peachey, M. (2006). “Development of a human cognitive workload assessment tool”.
Dalton : Human Reliability Associates. Gencer N., Usakli A.B., Erdogan H.B., Akinci B., Acar E., (2010), “ODTÜ Beyin Araştırmaları Laboratuvarı: Beyin-Bilgisayar Arayüzü çalışmaları” , ODTÜLÜ, 45, 20-21 Haapalainen, E., Kim, S., Forlizzi, J. F., Dey, A. K.., (2010), “Psycho-Physiological Measures for Assessing Cognitive Load”,
Ubicomp '10 Proceedings of the 12th ACM international conference on Ubiquitous computing, 301-310 Hussain, S., Chen, S., Calvo, R. A., & Chen, F. (2011), “Classification of Cognitive Load from Task Performance & Multichannel Physiology during Affective Changes”, In Paper Presented at the MMCogEmS: Inferring Cognitive and Emotional States from Multimodal Measures, ICMI 2011 Workshop. Karagöz M., Alkac U., Ergen N., Erdamlar N., Alpkan L., (2005), “Psikiyatrik Hastalıklarda Elektrofizyolojik (P300)
Yöntemler”, Düşünen Adam; 2005, 18(4),210-216 Klingner, J., Kumar, R., Hanrahan, P. (2008), “Measuring the task-evoked pupillary response with a remote eye tracker”, In Proceedings of the 2008 symposium on Eye tracking research & applications (pp. 69-72). Mak, J. N., Arbel, Y., Minett, J. W., McCane, L. M., Yuksel, B., Ryan, D., Erdogmus, D. (2011), “Optimizing the P300-based
brain–computer interface: current status, limitations and future directions”, Journal of neural engineering, 8(2). Palinko, O., Kun, A. L., Shyrokov, A., & Heeman, P. (2010), “Estimating cognitive load using remote eye tracking in a driving simulator”, In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications (pp. 141-144).
Pavlo A., Paas F., Grabner R., Tamara V.G., (2010), “Using Electroencephalography to Measure Cognitive Load”, Educational Psychology Review, 22 (4), 425-438. Perego, P., Turconi, A. C., Andreoni, G., Maggi, L., Beretta, E., Parini, S., Gagliardi, C. (2011), “Cognitive ability assessment by
Brain–Computer Interface: Validation of a new assessment method for cognitive abilities”, Journal of neuroscience methods, 201(1), 239-250. Schultheis, H., Jameson, A. (2004), “Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioral methods”, In Adaptive hypermedia and adaptive web-based systems (pp. 225-234). Springer Berlin Heidelberg. Tracy, J. P., Albers, M. J. (2006). “Measuring cognitive load to test the usability of web sites”. In Annual Conference-society for
technical communication (Vol. 53, p. 256). Walczyk, J. J., Griffith, D. A., Yates, R., Visconte, S. R., Simoneaux, B., Harris, L. L., (2012), “Lie Detection By Inducing Cognitive Load Eye Movements and Other Cues to the False Answers of “Witnesses” to Crimes”, Criminal Justice and Behavior, 39 (7), 887-909. Zhao, C., Zhao, M., Liu, J., Zheng, C. (2012), “Electroencephalogram and electrocardiograph assessment of mental fatigue in a
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2102-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional