Nikol Figalová, Jürgen Pichen, Lewis L Chuang, Martin Baumann, Olga Pollatos (2022). System immersion of a driving simulator affects the oscillatory brain activity. In: Neuroergonomics and Cognitive Engineering. AHFE Open Access, vol. 42, pp. 113–121. https://doi.org/10.54941/ahfe1001825
The technological properties of a system delivering simulation experience are a cru- cial dimension of immersion. To create a sense of presence and reproduce drivers behaviour as realistically as possible, we need reliable driving simulators that allow drivers to become highly immersed. This study investigates the impact of a system immersion of a driving simulator on the drivers’ brain activity while operating a con- ditionally automated vehicle. Nineteen participants drove approximately 40 minutes while their brain activity was recorded using electroencephalography (EEG). We found a significant effect of the system immersion in the occipital and parietal areas, prima- rily in the high-Beta bandwidth. No effect was found in the Theta, Alpha, and low-Beta bandwidths. These findings suggest that the system immersion might influence the drivers’ physiological arousal, consequently influencing their cognitive and emotional processes.
Driving simulators allow researchers to study driving behaviour under con- trolled settings. The need to investigate driver behaviour in a simulated environment is becoming even more pressing with the increasing car automa- tion. The existing automated vehicles cannot be used for real-world studies due to restricted availability or indefensible dangers imposed on the driver or the other road users. However, the simulation does not entail all details of actual driving, and creating a realistic simulation experience proposes a recurring challenge (Hock et al., 2018). Fisher et al. (2011) suggest that par- ticipants of driving simulator experiments do not behave the same way they would in their vehicle. In order to reproduce the driver’s behaviour as realisti- cally as possible, we need reliable driving simulators which allow participants to get highly immersed in the situation (Michael et al., 2014).
Immersion is a phenomenon experienced by an individual when they are in a state of deep mental involvement. Their cognitive processes cause a shift in their attentional state such that one may experience disassociation from the awareness of the physical world (Agrawal et al., 2020). Nilsson et al. (2016) identified three dimensions of immersion: (a) system immersion, i.e. immer- sion as a property of the technological system delivering the experience and the perceptual response of the user; (b) challenge immersion, i.e. immersion as a mental response to challenges; and (c) narrative immersion, i.e. immersion as a cognitive response to the narrative characteristics of the experience. The quality of the generated simulation directly moderates the system immersion. A high-fidelity simulator provides higher system immersion, and the overall user’s experience feels more realistic, while low-fidelity simulators may evoke unrealistic driving behaviour and, therefore, could produce invalid research outcomes (de Winter et al., 2012).
Measuring immersion using self-report questionnaires can be problematic as the questionnaires rely on participants subjective opinions and requires them to have a fair understanding of what it means to be “immersed”(Jennett et al., 2008). A more objective and reliable way to measure immersion would be beneficial in the research context. With the boom of low-cost and reliable electroencephalography (EEG) headsets, studying brain activity as a proxy of immersion seems viable.
Neural Correlates of Immersion The effect of immersion on brain activity has been previously studied in non- driving contexts. Nacke et al. (2011) reported significant increases in both theta (4-8 Hz) and beta (10-30 Hz) bands with higher immersion during video gameplay. Kruger et al. (2016) investigated the effect of subtitles on an immersion while watching a movie and found that average beta cohere- nce was reduced between the prefrontal and the parietal cortex. This finding suggests greater immersion for the subtitled film viewing experience. Lim et al. (2019) compared concentration and immersion. They suggest that the occipital lobe channel is highly active in the immersion state and report an increase in beta power during an immersion state compared to the rest state and concentration state.
Furthermore, we detected several studies focused on phenomena rele- vant to immersion. Derbali and Frasson (2010) studied players’ motivation in a serious game environment. They observed increased high-Beta power with higher motivation in the area of the left motor cortex (C3). Stenberg (1992) reported an increase in parietal high-Beta bandwidth in conditions with emotional valence. Moreover, he suggests that parietal high-Beta could be interpreted as a measure of alertness or sustained attention. Choi et al. (2015) assessed EEG relative to environmental stress. They found an incre- ase in Alpha and Theta power in non-stressful conditions in parietal (P3) and temporal (T4) regions, respectively, and an increase in high-beta power in the temporal and parietal (P4) region in stressful conditions. Increased motiva- tion, emotional involvement, and stress could indicate higher challenge and narrative immersion in the game, according to the theory of Nilsson et al. (2016).
System Immersion of a Driving Simulator Affects the Oscillatory Brain Activity 115
No studies on the neural correlates of immersion in driving simulator stu- dies were found. Michael et al. (2014) reported higher immersion when using a head-mounted virtual reality device compared to a standard screen; how- ever, only self-report data were collected. Nevertheless, understanding the effect of different simulator settings on brain activity is crucial to ensure that future research is reproducible and consistent. This study aims to explore the effect of system immersion on the drivers’ oscillatory brain activity when operating a conditionally automated vehicle (SAE level 3; SAE, 2021) in a driving simulator.
METHODS
Experimental Design and Apparatus We implemented a between-group design. The high-immersion group (HI) operated a fixed-base driving simulator with a 190 ° field of view, equipped with a realistic car mock-up. The low-immersion group (LI) operated the same simulator, but the side screens providing peripheral visual cues were turned off, and only the front screen was used.
Participants operated a conditionally automated vehicle (level 3; SAE, 2014) through four driving scenarios (approx. 10 minutes each). The order of the scenarios was counterbalanced using the balanced Latin square. The ego vehicle travelled at a set speed of 100 km/h on a rural road with mild oncoming traffic. There was light rain and fog consistently spread through- out the scenarios. Participants simultaneously performed an adapted Visual Search Task (Treisman and Gelade, 1980) on a dashboard screen while the vehicle was operating in an autonomous mode. To minimize the movement artefacts in EEG signal, participants conducted the secondary task using a keyboard placed comfortably in the reach of their hand. Ten auditory take- over requests were issued. These prompted participants to disengage from the secondary task, deactivate the automation, manually overtake a slow vehicle, activate the automation, and return to the secondary task. All participants went through a training session (approx. 15 minutes) before the experiment started.
Participants We recruited 24 participants; three were excluded from the final analysis due to technical problems, two participants (both from the HI group) were exclu- ded due to simulator sickness. The HI group consisted of nine participants (seven female) with an average age of 22.6 years (SD = 3.8). The LI group consisted of ten participants (eight female) with an average age of 23.0 years (SD = 4.2). All participants had normal or corrected to normal vision, no neurological or psychiatric disease, and a valid drivers license. Participants were primarily recruited from the Ulm University students and a database of contacts maintained by the dept. Human Factors, Ulm University. All partici- pants were offered either financial compensation (16 to 20 Eur) or study credits for their participation. All participants provided informed written consent before participating in the experiment. The experimental procedure
Figure 1: The HI setting (3 screens) and the LI setting (front screen only).
was performed in accordance with Helsinki’s Declaration and approved by the local Ethics Committee of Ulm University.
EEG data The EEG was recorded using 32 Ag/AgCl active shielded electrodes placed according to the International 10–20 System with a reference at the FCz posi- tion. We focused on the frontal (F3, Fz, F4); parietal (P3, Pz, P4); occipital (O1, Oz, O2); and temporal (T7, T8) regions (see Figure 2). The impeda- nces were kept below 25 kOhm. Horizontal and vertical eye movements were recorded using four electrodes. Data were recorded with a sampling rate of 500 Hz using a LiveAmp amplifier (BrainProducts, Munich, Ger- many). Noisy electrodes were topographically interpolated, and the signal was re-referenced to a common average reference. A bandpass filter betw- een 0.5 and 40 Hz was applied, as well as a 50 Hz notch filter. The data from the four driving scenarios were segmented into one second long inte- rvals, and independent component analysis (ICA) was used to remove the eye movement artefacts. We used the fast Fourier transformation with Han- ning window to calculate the relative spectral power density in the Theta (4–8 Hz), Alpha (8–12 Hz), Beta (12–30 Hz), low-Beta (12.5–16 Hz), and high-Beta (22–30 Hz) range. The power has been normalized as a percen- tage of the absolute power of the signal in the range from 0.5 to 40 Hz. The pre-processing steps were done using the BrainVision Analyzer 2.1. The Mann-Whitney U test and mixed ANOVAs were conducted using the SPSS 26. The Bayesian statistics were calculated in JASP 0.16.
System Immersion of a Driving Simulator Affects the Oscillatory Brain Activity 117
Figure 2: Positions of the electrodes which we used for the analysis.
RESULTS
Firstly, we compared the total relative mean power in the Theta, Alpha, Beta, low-Beta, and high-Beta bandwidths between the LI and the HI group across the four rides. Given the sample size and the characteristics of the data, we used the Mann-Whitney U test. We found no difference between the groups in the Theta, Alpha, and low-Beta bandwidths. There was a significant dif- ference between the groups in the Beta bandwidth at the Cz electrode (LI Mdn = 1.05, HI Mdn = 1.24, U = 18.00, p = .03, η2 = .27). Moreover, we found a significant difference in high-Beta bandwidth at the Oz electrode (LI Mdn = 0.70, HI Mdn = 1.20, U = 19.00, p = .04, η2 = .25), the O2 electrode (LI Mdn = 0.58, HI Mdn = 1.14, U = 17.00, p = .02, η2 = .29), the P3 ele- ctrode (LI Mdn = 0.65, HI Mdn = 0.94, U = 18.00, p = .03, η2 = .27), the P8 electrode (LI Mdn = 0.63, HI Mdn = 1.06, U = 20.00, p = .04, η2 = .23), and the Cz electrode (LI Mdn = 0.56, HI Mdn = 0.86, U = 13.00, p = .01, η2 = .38).
To assess the changes over time, we analyzed the differences between the four sequential rides using both frequentist and Bayesian mixed-model ANO- VAs. The normality assumption was checked using the Shapiro-Wilk test of normality. As the mixed ANOVA is relatively robust to violations of norma- lity, we proceeded with the analysis when 85% or more of the dependent variables were normally distributed. Due to this requirement, the low-Beta bandwidth was excluded from the analysis as almost half of the dependent variables did not meet the criteria. For the Alpha bandwidth, the two-way mixed ANOVA revealed a significant effect of time in the parietal region (F(1, 15) = 15.74, p < .001), temporal region (F(1,15) = 12.56, p = 0.003), and frontal region (F(1,15) = 5.69, p = 0.031). No significant effect was found for group or interaction between time and group. The two-way mixed ANOVA revealed no significant effects in any of the observed regions for the Theta, Beta, and high-Beta bandwidths.
We calculated the Bayes factor (BF10, Table 1) to understand the results further. The Bayesian approach suggests moderate evidence (0.33 > BF10 > 0.10) against the effect of time on frontal and parietal Theta; on occipi- tal and parietal Beta; and occipital high-Beta. Furthermore, we found strong evidence (BF10 < 0.10) against the effect of the group on frontal, tempo- ral, and parietal Alpha. Moreover, moderate to strong evidence against the
Figure 3: The mean Alpha power over the four consecutive driving scenarios.Table 1. Results of the Bayesian mixed ANOVA (moderate and strong effects are in bold).
interaction of the two main effects was found on frontal and parietal Theta; occipital, temporal, and parietal Beta; and occipital and parietal high-Beta. Additionally, we found equal support (BF10 = 1) for H0 and HA for the effect of time on temporal Theta, frontal, temporal, and parietal Alpha; and equal support for H0 and HA for the effect of Group on parietal Beta and high-Beta.
DISCUSSION
In the present experiment, we explored the oscillatory brain activity of a dri- ver while operating a conditionally automated vehicle in a driving simulator. We compared the EEG of drivers operating a highly immersive simulator and drivers operating a low immersive simulator.
We found no significant effect of the system immersion of the simulator on the Theta, Alpha, and low-Beta bandwidths. This finding was further supported by the Bayesian approach, which revealed moderate to strong
System Immersion of a Driving Simulator Affects the Oscillatory Brain Activity 119
evidence against the effect of group in the above mentioned bandwidths. This observation does not seem to align with the findings of Nacke et al. (2011), who compared boring and immersive video game experiences. They observed higher activation in the Theta bandwidth when participants expe- rienced the immersive trial compared to the boring trial. However, Nacke et al. did not focus on the system immersion but immersion’s challenge and narrative components. They manipulated the architectural complexity of the scenarios. They concluded that the changes in Theta activity originate in the need for episodic and semantic memory activation, path integration, and lan- dmark navigation. None of these was necessary in the present experiment, as we manipulated only the system immersion. Therefore, we believe that our results are not conflicting but rather complementary.
A significant effect of the system immersion of the simulator was found predominantly in the high-Beta bandwidth in the occipital and parietal areas. This result is in line with the findings of Lim et al. (2019), who descri- bed an increase in beta power and high activation of the occipital lobe in the immersion state. According to Abhang et al. (2016), high-Beta waves can be associated with stress and high arousal. In the context of the pre- sent study, it could suggest that the highly immersive environment leads to higher physiological arousal. As cognition is connected to arousal (Choi et al., 2015; Stenberg, 1992), it would be desirable to study further how the system immersion affects the driver’s cognitive process and psychological presence. Moreover, high-Beta waves were associated with emotion (Stenberg, 1992). This could play a significant role in driving simulator experiments, e.g., for user experience assessment or risk perception in simulated environments.
When comparing the four sequential rides, we found an increase in alpha power as the effect of time in both groups. This can be attributed to the time- on-task effect (Figalová et al., 2021; Zhao et al., 2012). However, no effect of group or interaction was found. Based on this observation, we assume that the different setting of the simulator does not induce progressive changes in brain activity over time.
Several limitations can be found in this study. First, the sample size is rela- tively small, which was also suggested by the results of the Bayesian tests. Collecting a larger sample size would benefit the robustness of the results. Moreover, it would be interesting to compare the self-reported immersion using a scale; however, this was not possible in the present study. In future studies, we also recommend studying the effect of system immersion on simulator sickness, given that only participants from the HI group showed symptoms of simulator sickness. We also recommend combining both the frequentist and Bayesian approach, which seems to be a good way to under- stand the results comprehensively. Finally, it would be interesting to assess the effect of immersion on the cognitive and emotional processes of the driver.
CONCLUSION
The system immersion of a driving simulator affects the oscillatory brain activity of the driver. We found empirical evidence for changes in the high- Beta bandwidth in the parietal and occipital areas and Beta bandwidth at the
Cz electrode. No effect of the system immersion was observed in the Theta, Alpha, and low-Beta bandwidths. The change in the psychophysiological acti- vation suggests that the system immersion modulates the arousal of drivers and could therefore affect their cognitive and emotional processes, perceived stress, and presence.
ACKNOWLEDGMENT
This project was funded by the European Union’s Horizon 2020 rese- arch and innovation program under the Marie Skłodowska-Curie grant agreement 860410. LLC acknowledges funding support by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project-ID 416228727-SFB 1410.
REFERENCES
Abhang, P. A., Gawali, B., and Mehrotra, S. C. (2016), “Introduction to EEG-and speech-based emotion recognition.” London, Academic Press. Agrawal, S., Simon, A., Bech, S., Bærentsen, K. and Forchhammer, S. (2020), “Defi-
ning Immersion: Literature Review and Implications for Research on Audiovisual Experiences.” Journal of the Audio Engineering Society, 68(6), 404–417. Choi, Y., Kim, M. and Chun, C. (2015), “Measurement of occupants’ stress based on electroencephalograms (EEG) in twelve combined environments.”Building and Environment, 88, 65–72. Derbali, L. and Frasson, C. (2010), “Players’ motivation and EEG waves patterns in
a serious game environment.” Intelligent Tutoring Systems, 297–299. De Winter, J., van Leeuwen, P.M. and Happee, R. (2012), “Advantages and disadvantages of driving simulators: A discussion.” Proceedings of Measuring Behavior 2012, 47–50. Figalová, N., Delp, A.L., Nick, A., Chuang, L.L., Baumann, M. and Pollatos, O.
(2021), “Alpha and Theta Power in Repeated Take-over Requests.” Neuroergo- nomics Conference 2021. Retrieved from https://www.shorturl.at/guL25. Fisher, D., Rizzo, M., Caird, J. and Lee, J. D. (2011), “Handbook of Driving Simulation for Engineering, Medicine, and Psychology.” Boca Raton, CRC Press. Hock, P., Kraus, J., Babel, F., Walch, M., Rukzio, E., and Baumann, M. (2018), “How
to design valid simulator studies for investigating user experience in automated driving.” Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., and Walton, A., (2008), “Measuring and defining the experience of immersion in games.” International Journal of Human-Computer Studies, 66 (9), 641–661. Kruger, J.L., Doherty, S. and Ibrahim, R. (2017), “Electroencephalographic beta
coherence as an objective measure of psychological immersion in film.” Inter- national Journal of Translation (RITT Rivista internazionale di tecnica della traduzione), 19, 99–112. Lim, S., Yeo, M., and Yoon, G. (2019), “Comparison between concentration and immersion based on EEG analysis.” Sensors, 19(7), 1669. Michael, D., Kleanthous, M., Savva, M., Christodoulou, S., Pampaka, M., and
Gregoriades, A. (2014), “Impact of immersion and realism in Driving Simula- tor Studies.” International Journal of Interdisciplinary Telecommunications and Networking, 6(1), 10–25.
System Immersion of a Driving Simulator Affects the Oscillatory Brain Activity 121
Nacke, L.E., Stellmach, S., and Lindley, C.A., (2010), “Electroencephalographic assessment of player experience.” Simulation & Gaming, 42(5), 632–655. Nilsson, N.C., Nordahl, R., and Serafin, S. (2016), “Immersion revisited: A review
of existing definitions of immersion and their relation to different theories of presence.” Human Technology, 12(2), 108–134. SAE International (2021), “SAE J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles.” Retrieved from https://www.sae.org/standards/content/j3016_202104/ Stenberg, G. (1992), “Personality and the EEG: Arousal and emotional arousability.”
Personality and Individual Differences, 13(10), 1097–1113. Treisman, A.M. and Gelade, G. (1980), “A feature-integration theory of attention.”
Cognitive Psychology, 12(1), 97–136. Zhao, C., Zhao, M., Liu, J., and Zheng, C. (2012), “Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator.” Accident Analysis & Prevention, 45, 83–90.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Neuroergonomics and Cognitive Engineering
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-18-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional