Amir Dini, Michael Schneeberger, Martin Pszeida, Lisa Heiler, Lucas Paletta (2023). Towards Immersive Skill Training for First Responders with Biosensor-based Assessment of Situation Awareness. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 73, pp. 104–114. https://doi.org/10.54941/ahfe1003979
First responders engage highly stressful situations at the emergency site that may induce stress, fear, panic and a collapse of clear thinking (Putnam, 1995). However, their physiological and cognitive readiness is of highest importance to enable appropriate decision making (Frye & Wearing, 2014). A major aspect for keeping control of the situation is that first responders should always maintain situation awareness. Training of the first responders’ routines improves resilience towards stressors in severe hazard conditions. Advanced training of first responders, in particular, the use of VR environments for the training of situation awareness for stress-resilient decision-making behavior is one of the upcoming challenges of the near future.Methods A Virtual Reality environment was developed with a typical firefighter scenario to train and to evaluate the situation awareness (SA) of squad leaders. The operator has two major tasks, (i) searching for pocket of embers on the ground by using an infrared camera and whenever detected to call for a team member to work on it, (ii) to survey her team of 5 first responders for their visible presence as well as for their health condition and consequently to react in proper time instants whenever team members would be injured or disappear. The movement of the team members within a predefined skill-oriented territory is activated by an AI-driven method. A wearable multisensory-based non-invasive measurement suite is mounted on the operator including VR in-built eye-tracking, biosensors for cardiovascular, electrodermal and temperature data capture, to report by means of digital human factors analytics. From the sensor data stream we intend to deduce psychological constructs, such as, situation awareness, physiological strain, cognitive-emotional stress, and fatigue. A pilot study is planned with N=20 students, with pre- and post-study cognitive tests, such as, determination test (Schuhfried, 1987) and Psychomotor Vigilance Test (PVT; Dinges & Powell, 1985), as well as the Situation Awareness Rating Technique (SART; Taylor, 1990). From the SART we will research for to situation awareness-specific digital biomarkers by means of correlation analysis. This multi-tasking configuration will enable to measure (a) executive functions, such as, cognitive flexibility, and (b) level 1, 2 and 3 SA by gaze-driven virtual events, such as, viewing first responder avatars. Results: We will describe the system architecture as well as the features of the VR-based skill evaluation system (VR with eye-tracking, biosensors, treadmill, dashboard). Test users will move on a treadmill that triggers the VR-based experience accordingly, by means of a Cyberith Visualizer platform. Videos that were captured from the use of the platform and application of biosensors with eye tracking will demonstrate the usefulness for the evaluation of SA. We will report from a first usability study with feedback from friendly users. At the conference, we will be able to report on first results of the pilot study.Conclusion: Advanced training of first responders and emergency staff with typical operation scenarios is one of the upcoming challenges of the near future. First results motivate the use of VR environments for the training of situation aware and stress-resilient decision-making behavior of firefighters.
First responders engage highly stressful situations at the emergency site that may induce stress, fear, panic and a collapse of clear thinking (Putnam, 1995). However, their physiological and cognitive readiness is of highest importa- nce to enable appropriate decision-making (Frye & Wearing, 2014). A major aspect for keeping control of the situation is that first responders should always maintain SA. Training of the first responders’ routines improves resili- ence towards stressors in severe hazard conditions. Advanced training of first responders, in particular, the use of VR-based environments for the training of SA and stress-resilient decision-making behaviour is one of the upcoming challenges of the near future.
A VR-based environment was developed with a typical firefighter scenario to train and to evaluate the SA of squad leaders (Figure 1). The operator has two major tasks, (i) searching for pocket of embers on the ground by using an
Towards Immersive Skill Training for First Responders 105
Figure 1: First responder in the treadmill with biosensor evaluation kit for the VR-based study of SA in critical situations (credit: L. Paletta, JOANNEUM RESEARCH DIGITAL).
infrared camera and whenever detected to call for a team member to work on it, (ii) to survey the team of 5 first responders for their visible presence as well as for their health condition and consequently to react in proper time instants whenever team members would be injured or disappear. The movement of the team members within a predefined skill-oriented territory is activated by an AI-driven method.
A wearable multisensory non-invasive measurement suite is mounted on the operator including VR-in-built eye tracking, biosensors for cardiova- scular, electro-dermal and temperature data capture, to report by means of digital human factors analytics. From the sensor data-stream we intend to deduce psychological constructs, such as, SA (Paletta et al., 2022), physiolo- gical strain, cognitive-emotional stress, and fatigue. This paper describes the system architecture as well as the features of the VR-based skill evaluation system (VR with eye tracking, biosensors, treadmill, dashboard).
IMMERSIVE SYSTEMS FOR FIRST RESPONDER TRAINING VR-based training in disaster management has been recognized as an impor- tant additional modality to traditional real-life skill training (Freeman et al., 2001). The increased realism in the practice enables first responders to reinfo- rce their individual performance, in particular, to execute tasks appropriately under stress and apply decision making under conditions close to reality. Mills et al. (2019) estimated that a mass casualty triage training of para- medic students in a real-world simulation is about 13 times more expensive than in VR, while the simulation efficacy has been found nearly identical. Recent research (Koutitas et al., 2020) has even indicated superior perfor- mance in simple search tasks following VR- and AR-based training of first responders as opposed to traditional classroom and real world training. The
use of wearable sensor-based monitoring systems offers life-saving opportu- nities for advanced health care solutions for stress analysis. By means of a wearable biomonitoring platform, Rodrigues et al. (2018) demonstrated that an increase in the stress levels decreased the cognitive performance of firefi- ghters. Meina et al. (2020) showed that psychophysiological measurements are reliable indicators of stress even in ecological settings and appear pro- mising for chronic stress monitoring in high-risk jobs, such as firefighting. Wildland firefighters often have to make tactical decisions under stressful conditions in order to bring a fire under control. These decisions can be hin- dered by human factors such as insufficient knowledge of surroundings and conditions, lack of experience, overextension of resources or loss of situatio- nal awareness (Fryer et al., 2015). SA should be kept high even during simple exercises (Paletta et al., 2022).
ASSESSMENT OF SITUATION AWARENESS The concept of Situation Awareness (SA). SA has been suggested to directly affect human decision-making and performance in human–machine environ- ments. Building on her definition of situation awareness, Endsley (1995) designed a model to describe information processing errors in complex systems. The model allows identification of errors that can ‘lead to acci- dents in complex systems’, such as, a team-supported firefighting operation. Depending on the phase of information processing, errors are expected in (1) perception processes, (2) the understanding and interpretation of the situation, and (3) in the projection of the system state into the future/antici- pation. Without the perception of important information, the risk of creating a false picture of the situation increases (level 1 SA). However, in addition to perceiving stimuli, it is also necessary to combine, integrate, and store them. Information from several different sources must be integrated and interpreted in relation to personal or general goals (level 2 SA). The highest level of SA represents the ability to project current events onto anticipated future events (level 3 SA; Peissl, 2016).
Measures of Situation Awareness. To address the limitations of freeze- probe methods, alternative assessment formats have also been developed. In particular, self- and observer-rating techniques using questionnaires are also considered as direct SA measurements. The SA rating technique (SART; Tay- lor, 1990; Salas et al., 2018) and the SA behavioural rating scale (SABARS; Matthews & Beal, 2002) are two commonly used rating techniques. For example, after an experimental trial is completed, the SART assessment method asks participants to rate their awareness using a seven-point scale on the following dimensions: situation familiarity, attentional focus, infor- mation quantity and quality, situation instability, attention concentration, situation complexity, situation variability, arousal, and spare mental capacity (Salas et al., 2018). In contrast to the SART, another self-rating technique, the SABARS, is an observer-based rating method that relies on the observation of domain experts on participant behaviours. Specifically, it is a five-point rating scale on 28 observable SA behaviours of the participant (Matthews & Beal, 2002).
Towards Immersive Skill Training for First Responders 107
Measurement of Situation Awareness in VR. Ehrlich et al. (1997) repor- ted that Virtual Environment (VE) technology, which uses position tracking and real-time update of visual, auditory, and other displays (e.g., tactile) in response to the user’s motions to give the user the sense of being ‘in’ the environment, has the potential to improve simulation-based training for dismounted soldiers. One challenging area of research is matching U.S. Army training requirements with current and projected future VE capabilities. VE technologies were identified to be highly suitable to train particularly mem- bers of small dismounted units to acquire and maintain SA. Kaber et al. (2017) evaluated the efficacy of a computer-based SA measurement system for training dismounted infantry SA in an urban terrain VR-based simulation. The SA behaviour and communication ratings revealed differences among squads, which trended with experience.
Measures of SA were also consistent across the test scenario as a result of similar mission types and task difficulties.
Physiological Measures of SA. With respect to physiological measurement techniques, a recent systematic review by Zhang et al. (2020) reported that eye tracking was most commonly used for assessing SA.
Several studies associated SA with indirect cardiovascular activity measu- red by heart rate (HR) or heart rate variability (HRV). Three different studies used measurements related to EEG brain activity. Finally, one study used electro-dermal activity (EDA) and one used respiratory rate (RE). Among all articles, only one used multiple physiological measures.
Progress of presented work. The specific progress targeted in the presen- ted research work is achieved by (i) proposing a very specific skill training that is at the same time particularly suited for SA training, and (ii) appl- ying psycho-physiological measurements from wearable / portable sensors, in real-time. This work is based on previous results on biosensor-based esti- mation of situation awareness (Paletta et al., 2022) and will be performed applying pilot studies in the Human Factors Lab1 of Joanneum Research, Austria.
Human Factors Lab. The Human Factors Lab combines state-of-the-art human-centred digital measuring technologies with AI-enabled software for behaviour-based analytics and assessment of psychological constructs in digi- tal systems, such as, for digital cognitive biomarkers for cognitive screening in dementia, or, for estimating SA scores (Dini et al., 2017; Paletta et al., 2022). It analyses human stress as well as cognitive, affective, and motivational state by means of mobile and wearable technologies applied at work or at home. The infrastructure is equipped to investigate psychologically relevant para- meters in real-time including a wearable biosensor and Neuro-ergonomics (fNIRS, EEG) workplace as well as VR/AR-based technologies, eye tracking (stationary, mobile, wearable) and high precision motion capture devices, fur- thermore, a treadmill, social robots and software libraries for gaze based analytics. The Human Factors Lab has access to various IoT software libra- ries for permanent data acquisition for various sensors and gateways to set up wireless sensing network.
1http://www.joanneum.at/humanfactors
AI-BASED IMMERSIVE SKILL-TRAINING SYSTEM In this Section, we introduce the overall rationale for the implementation of the immersive training system. Then we present the system architecture that enables to perform a specific skill-oriented mission as well as to monitor specific aspects of the team behaviour and cognitive-emotional as well as SA-based parameters.
Overall Concept. A VR-based environment was developed with a typical firefighter scenario to train and to evaluate the SA of squad leaders. The ope- rator has two major tasks, (i) searching for pocket of embers on the ground by using an infrared camera and whenever detected to call for a team member to work on it, (ii) to survey her team of 5 first responders for their visible prese- nce as well as for their health condition and consequently to react in proper time instants whenever team members would be injured or disappear. The movement of the team members within a predefined skill-oriented territory is activated by an AI-driven method.
System Architecture. A schematic sketch of the overall hardware and sof- tware setup is depicted in Figure 2. From the input sensor devices, i.e., the eye tracking and VR-based event monitor, as well as the bio-signals from the BIOPAC system, data are received and analysed with respect to informa- tive features in the Feature Fusion component (centre). The results are then visualized in real-time in dashboards on nearby displays (top).
Figure 3 provides a sketch of the system architecture of the VR-based skill training system. From the system architecture in Figure 3 we can depict that there are basically two relevant configuration inputs, providing static and dynamic parameters into the system. The I/O handler from Unity is respon- sible to control the XR aspects in the system. The render pipeline provides control over the rendering of the application, including, in particular, the individual virtual cameras, such as the thermal imaging camera.
The Application Logic and Manager component implements the control loop and decision-making in the system. Control signals are then fed to the individual scoring, visualisation and user interaction modules (Cognitive Computing & Visualisation). Output data are fed in real-time to the logger as well as to the dashboard and the display in general (top).
Figure 2: A schematic sketch of the overall hardware and software setup of the human factors analysis system, primarily based on the ‘Virtualizer’ treadmill (bottom).
Towards Immersive Skill Training for First Responders 109
Figure 3: Schematic sketch of the system architecture of the VR-based system for SA assessment in the project SIXTHSENSE (Bijelic et al., 2022).
System Components. The Vive Pro Eye Headset is used for the study, which records the eye tracking data with 120 Hz. Cyberith Virtualizer 2 is used for locomotion, which transmits the walking speed and direction of the participant. The biometric data is transmitted bi-directionally over the network between AcqKnowledge BIOPAC and the VR application. The application (made with Unity) records the data of the participant, the instru- ctions of the participant and the current behaviour of the five AIs in each frame.
Virtual Test Environment. The environment spans an area of size 100 × 100 meters2 in which 64 pocket of embers are randomly distributed. The participant operates as a squad leader starting in the area centre. She has a certain amount of time available to find the pocket of embers and pro- vide orders to team members to extinguish them. There are five virtual team members designed as AI-based agents that are located in this area, progressing on predefined pathways within the area, and ready to serve to extinguish pocket embers that are located on their pathways. The objective of the mission is a dual-task optimisation problem within a predefined inte- rval of time, i.e., (1) to find and extinguish as many embers as possible and at the same time (2) to monitor the five team members and their state of health and presence as well as instruct them if an ember had to be extingui- shed. Figure 5 visualises the various tasks of the squad leader and the team members.
Behaviour Trees. A behaviour tree (Colledanchise et al., 2017) refers to a mathematical model of plan execution used in computer science, robotics, control systems and video games, and therefore is of high importance for VR-based environments as well. They describe switchings between a finite set of tasks in a modular fashion. Their strength comes from their ability to create very complex tasks composed of simple tasks, without worrying how
the simple tasks are implemented. Behaviour trees present some similarities to hierarchical state machines with the key difference that the main building block of a behaviour is a task rather than a state. Its ease of human under- standing make behaviour trees less error prone and very popular in the game developer community. Behaviour trees have been shown to generalize several other control architectures.
AI-based Agent Wayfinding and Team Behaviour. In the presented work, an A* search algorithm (Zeng and Church, 2009; Russell, 2018) was applied for wayfinding within the graph of potential way trajectories of the virtual team members. A* is a graph traversal and path search algo- rithm, which is used in many fields of computer science due to its com- pleteness, optimality, and optimal efficiency. Figure 4 depicts the beha- viour tree defining the behaviour of the virtual team members in the immersive training environment that cooperate with the user in a speci- fic way to serve by extinguishing the pocket of embers but also disap- pear or get injured a predefined number of time within a certain mission task.
Figure 4: The human factors analysis monitor for research and development applying real-time and posterior study insight into relevant features. Left: egocentric view on virtual environment, including the virtual IR camera to detect pocket of embers. Below: overview on status (present, injured, missing) of team members. Mid right: human factors measurements. Right: live view of VR user.
In Figure 6 the behaviours, such as, TaskPatrol, TaskExtinguish, and TasgGoEmber, are depicted as well as the decision points in the tree that discriminate about further onset of behaviours. Specific parameters are con- figured that define how often these five agents or team members are injured or disappeared which makes the complete mission controllable and compa- rable between different study participants. Finally, the AI was configured on the following four decision points: “patrol”, “extinguish founded embers on command”, and “get injured” (e.g., physiological breakdown), disappear (e.g., fallen down the ravine).
Towards Immersive Skill Training for First Responders 111
Figure 5: Typical situations in the VR-based skill training. (a) Detection of a pocket of embers (orange) in the display of the virtual, mobile IR camera. (b) Close-up of a female FR being active in the “patrol” mode. (c) Selection menu to activate an FR to extinguish a pocket of embers. (d) Surveillance of the health status and presence of remotely distributed FRs (marked by blue and yellow).Figure 6: The behaviour tree the AI-based agent and team behaviour is based upon represented as control node logic.
STUDY PLAN A pilot study is planned with N = 20 students, with pre-and post-study cognitive tests, such as, Determination Test (Schuhfried, 1987) and Psychomotor Vigilance Test (PVT; Dinges & Powell, 1985), furthermore the Situation Awareness Rating Technique (SART; Taylor, 1990). With respect to the SART we will research for situation awareness-specific digital cognitive biomarkers by means of correlation analysis. This multi-tasking configura- tion including the mission to remove pockets of embers as well as to monitor the presence and health status of a complete mission team will enable to mea- sure (a) executive functions, such as, cognitive flexibility, and (b) level 1, 2 and 3 situation awareness by gaze-driven virtual events, such as, viewing first responder avatars, as well as digital activities, the response time to virtual events, such as, team members getting injured or missing.
CONCLUSION
Conclusion: Advanced training of first responders and emergency staff with typical operation scenarios is one of the upcoming challenges of the near future. First results motivate the use of VR environments for the training of situation aware and stress-resilient decision-making behaviour of firefighters. A study with N = 20 participants is planned to take place in 2023 to generate the expected digital cognitive biomarkers, in particular, the estimation of situ- ation awareness scores from digital virtual events as well as digital biosensor measurements.
ACKNOWLEDGMENT
This work was partially funded from project SIXTHSENSE (grant No 883315, Horizon 2020 Research and Innovation Program of the European Commission).
REFERENCES
Bijelic, G., Stefanovi´c, C., Morschhauser, A., Carballo Leyenda, A. B., Paletta, L., Falk, A., Kosti´c, M., Strbac, M., Jorgovanovic, N., Jobst, G., Paradiso, R., Magenes, G., Fanjul Bolado, P., Eschenbacher, P., Vujic, A. (2022). ‘Sixth Sense: Smart Integrated Extreme Environment Health Monitor with Sensory Feedback for Enhanced Situation Awareness, 2022 IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks (BSN), Ioannina, Greece, 2022, pp. 1–4, doi: 10.1109/BSN56160.2022.9928493. Colledanchise, M., & Ögren, P. (2017). How Behaviour Trees Modularize Hybrid
Control Systems and Generalize Sequential Behaviour Compositions, the Subsum- ption Architecture, and Decision Trees. IEEE Transactions on Robotics. 33(2): 372–389. doi: 10.1109/TRO.2016.2633567 Dinges, D. F. and Powell, J. W. (1985). Microcomputer analyses of performance on a portable, simple visual RT task during sustained operations. Behaviour Research Methods, Instruments, & Computers, 17, pp. 652–655.
Towards Immersive Skill Training for First Responders 113
Dini, A., Murko, C., Paletta, L., Yahyanejad, S., Augsdörfer, U., and Hofbaur, M. (2017) Measurement and Prediction of Situation Awareness in Human- Robot Interaction based on a Framework of Probabilistic Attention, Proc. IEE- E/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, pp. 4354–4361, doi: 10.1109/IROS.2017.8206301. Ehrlich, J. A. et al. (1997) Team Situational Awareness Training in Virtual Environ-
ments: Potential Capabilities and Research Issues. Technical Report 1069, United States Army Research Institute for the Behavioural and Social Sciences. Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems, Human Factors, 37(1), pp. 32–64. Freeman, K. M., Thompson, S. F., Allely, E. B., Sobel, A. L., Stansfield, S. A., Pugh,
W. M. (2001). A virtual reality patient simulation system for teaching emergency response skills to US Navy medical providers, Pre hospital and Disaster Medicine, 16(1). Fryer, G., Dennison, P., Cova, T. (2013). Wildland firefighter entrapment avoidance: Modelling evacuation triggers. International Journal of Wildland Fire. https://dx .doi.org/10.1071/WF12160. Kaber, D., Riley, J., Endsley, M., Sheik-Nainar, M., Zhang, T., Lampton, D. (2013).
Measuring Situation Awareness in Virtual Environment-Based Training. Military Psychology. 25. 330. 10.1037/h0095998. Koutitas, G., Smith, S., & Lawrence, G. (2021). Performance evaluation of AR/VR training technologies for EMS first responders. Virtual Real. 25, 1(Mar 2021), 83–94. https://doi.org/10.1007/s10055-020-00436-8 Matthews M. D., Beal S. A. (2002). Assessing situation awareness infield training
exercises (Report No. ARI DTIC Report 1795): Army Research Institute for the Behavioural and Social Sciences. Meina, M., Ratajczak, E., Sadowska, M. et al. (2020). Heart Rate Variability and Accelerometry as Classification Tools for Monitoring Perceived Stress Levels—A Pilot Study on Firefighters, Sensors, 20. Mills, P., Dykstra, S., Hansen, A., Miles, T., Rankin, L., Hopper, L., Brook, Bar-
tlett, D. (2019). VR triage training can provide comparable simulation efficacy for paramedicine students compared to live simulation-based scenarios, Prehosp Emerg Care. Paletta, L., Pszeida, M., Schneeberger, M., Dini, A., Reim, L., Kallus, K. W. (2022).
Cognitive-emotional Stress and Risk Stratification of Situational Awareness in Immersive First Responder Training, 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), Ioannina, Greece, 2022, pp. 1–4, doi: 10.1109/BHI56158.2022.9926805. Peissl, S. (2013). Aufmerksamkeit, antizipative Verhaltenssteuerung und motorische Leistung. Dissertation, University of Graz, Institute of Psychology. Putman, T. (1995). The collapse of decision making and organizational structure on
Storm King Mountain, Wildfire, 4(2), pp. 40–45. Rodrigues, S., Paiva, J. S., Dias, D., Pimentel, G., Kaiseler, M., Cunha, M. (2018).
Wearable Bio monitoring Platform for the Assessment of Stress and its Impact on Cognitive Performance of Firefighters: An Experimental Study, Clinical practice and epidemiology in mental, pp. 250–262. Russell, Stuart J. (2018). Artificial intelligence a modern approach. Norvig, Peter (4th ed.). Boston: Pearson. ISBN 978–0134610993. Salas E., Dietz A. S., Taylor R. M. (2018). Situational awareness rating technique
(SART): The development of a tool for aircrew systems design. In Situational awareness, pp. 111–128, Routledge.
Taylor, R. M. (1990). Situational Awareness Rating Technique (SART): The development of a toll for aircrew systems design. In: AGARD Conference Proceedings No 478, Situational Awareness in Aerospace Operations. Aerospace Medical Panel Symposium. Zeng, W., Church, R. L. (2009). Finding shortest paths on real road networks: the
case for A*. International Journal of Geographical Information Science. 23(4), pp. 531–543. doi: 10.1080/13658810801949850 Zhang, T., Yang, J., Liang, N., Pitts, B. J., Prakah-Asante, K. O., Curry, R., Duerstock, B. S., Wachs, J. P., & Yu, D. (2020). Physiological Measurements of Situation Awa- reness: A Systematic Review. Human Factors, Doi: 10.1177/0018720820969071
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-49-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional