Elisabeth Shi, Klaus Bengler (2022). Overall effects of non-driving related activities’ characteristics on takeover performance in the context of SAE Level 3: A meta-analysis. In: Human Factors in Transportation. AHFE Open Access, vol. 60, pp. 69–77. https://doi.org/10.54941/ahfe1002435
Since users of Level 3 driving automation systems may engage in non-driving related activities, it is of interest to investigate effects of non-driving related activities on traffic safety. The current meta-analysis aims at estimating non-driving related activities’ effects on takeover behavior based on theory and empirical findings on task switching and modality shifting. Results indicate that takeover time increases after non-driving related activities that (1) physically require the user to free his/her hands from the respective activity before takeover, (2) obstruct concurrent visual perception of the driving environment, (3) are dissimilar to the driving task in terms of required cognitive processing modules.
Effects of non-driving related tasks (NDRTs) on takeover performance in the context of SAE Level 3 have been investigated in multiple studies. These have been summarized by – to our knowledge – three meta-analyses: First, Zhang et al. (2019) focused on takeover time at SAE Levels 2 and 3, and looked for factors influencing takeover time after either a request to inte- rvene or a critical event. They find many situational variables that impact the duration of takeover (e.g. situation’s urgency, modalities of the request to intervene), but also some factors that can be influenced by the user (e.g. expe- rience with takeover situations, specifics of the non-driving related activity) (Zhang et al. 2019). Second, Soares et al. (2021) report a meta-analysis on simulator studies and used k-medoids clustering to answer the question how primary studies’ experimental conditions may have influenced the measured outcomes (e.g. takeover time). Based on the described search procedures and study coding, it can be assumed that Soares et al. looked at driving automa- tion systems of Level 2 and higher levels, similar to Zhang et al. (2019). Third, Weaver and DeLucia (2020) conducted a meta-analysis to examine whether (1) time budget, (2) engagement in NDRTs and (3) information provided to
Figure 1: Takeover situation in Level 3 from the perspective of Rubinstein et al.’s task switching theory (2001). Speaker symbol indicates system’s request to intervene.
the driver in the takeover situation influences takeover performance. In con- trast to the former two meta-analyses, Weaver and DeLucia focused on SAE Level 3 exclusively because of the distinctive role of the fallback-ready user. The meta-analysis at hand adds to the former by focusing on existing empiri- cal research on effects of NDRTs in SAE Level 3 from the perspective of task switching and modality shifting known from psychological basic research (Rubinstein et al. 2001, Spence et al. 2001). Thereby, we investigate a the- oretical basis to differentiate non-driving related activities’ effects on traffic safety based on their respective characteristics.
SAE Level 3 Automation from a Switching and Shifting Perspective Sustained driving automation systems (Shi et al. 2020) of SAE Level 3 (SAE International/ISO 2021) perform the entire dynamic driving task within its system limits, thereby releasing and relieving in-vehicle human user from the driver role. S/He may engage in other non-driving related activities. Beyond its limits, the Level 3 system cannot perform the dynamic driving task on a sustained basis. When a system limit is approached, the system issues a request to intervene expecting the in-vehicle human user to respond by taking over the driving task.
Upon onset of the request to intervene, the user needs to physically and mentally switch from a potential non-driving related activity to the driving task. Regarding physical switching, the user might need to free her/his hands from the respective non-driving related activity (e.g. smartphone, laptop). If physical adjustments are necessary, they will by nature prolong takeover time compared to when physical adjustments are not required. The mental swi- tching to the driving task will be described in the following by first applying Rubinstein et al.’s stage model of executive control for task switching (2001) and second, by applying the modality shifting effect (Spence et al. 2001).
Takeover Situation at SAE Level 3 in Terms of Task Switching Considering the Level 3 takeover situation as an example for the task swi- tching paradigm is not new (e.g. Zeeb et al. 2017). We apply Rubinstein et al.’s task switching theory (2001) to differentiate NDRTs’ effects on take- over behavior. In their stage model of executive control for task switching, Rubinstein et al. (2001) assume task processes and executive control proces- ses, with executive control processes being specific to task switching, and task
Overall Effects of Non-Driving Related Activities’ Characteristics 71
processes taking place whenever working on a task. Task processes include stimulus identification, response selection and movement production related to the respective task at hand. Executive control processes include goal shif- ting and rule activation required when switching from one task to another. Rubinstein et al. explicitly assume that goal shifting may take place before stimulus identification of the following task started (Rubinstein et al. 2001, p. 771). Transferred to the takeover situation at SAE Level 3, we estimate effect sizes comparing NDRTs that allow for executive control processes and task processes of the subsequent driving task to take place earlier compared to tasks that do not allow so. For instance, if the NDRT does not involve the visual modality (e.g. listening to music), the visual modality can be used to observe the traffic situation which is part of the subsequent driving task. In terms of the described task switching theory, the NDRT “listening to music” allows for task processes related to the subsequent task to take place earlier compared to another NDRT “playing Tetris”.
Modality Shifting Effect in the Takeover Situation at SAE Level 3 Task processes such as stimulus identification might be influenced by the so-called modality shifting effect which refers to the relative costs in rea- ction time and error rates when the modality in which a target is presented, differs from the modality of a previous stimulus compared to when both modalities match. Since NDRTs and takeover situations occur in real traffic, differentiation of NDRTs by sensory input modality is less practical than in the laboratory experiments (e.g. as long as the user does not close his/her eyes, there will always be visual input. Since sleeping violates the fallback- ready user’s responsibilities at Level 3, it is unlikely that the user would close his/her eyes for a longer period of time). We therefore make use of the moda- lity specific cognitive processing modules to describe the similarity between NDRTs and the driving task. Based on Baddeley’s working memory model (Repovs and Baddeley 2006), the processing modules required for each task (NDRTs, driving task) were evaluated first, and then compared. The simi- larity of the tasks’ demand profiles serve as an indicator for the two tasks’ similarity.
We considered empirical evidence suggesting the visuo-spatial sketchpad to consist of two separate subsystems instead of one combining visual and spa- tial features (e.g. Klauer and Zhao 2004). Moreover, most everyday tasks, including driving and NDRTs, need to be deemed “episodic”. To allow for differentiation, we deviated from the episodic buffer’s original focus, and chose deliberate long-term memory retrieval as the criterion, e.g. in case of quiz tasks when knowledge needs to be actively retrieved. Consequen- tly, the following cognitive processing modules were coded: phonological loop, visual system, spatial system, central executive, deliberate long-term memory retrieval. For example, the driving task requires the visual, spatial system and the central executive, but does not require deliberate long-term memory retrieval or the phonological loop. The similarity between driving task and playing Tetris (requires the visual, spatial system, central executive to align action to input, but no deliberate long-term memory retrieval) is thus
72 Shi and Bengler
higher than between driving task and listening to music (requires the phono- logical loop, but not the visual, spatial system, central executive or deliberate long-term memory retrieval). According to Rubinstein et al. (2001), however, listening to music allows task and executive control processes related to both switching and the subsequent driving task to take place earlier compared to playing Tetris.
Aim and Hypotheses Starting from the outlined theoretical basis, we investigate the following ove- rall effect sizes on takeover times: (1) overall effect size of monitoring task vs. active NDRT, (2) overall effect size of required vs. not required visual sensory system for active NDRTs, (3) overall effect of physical need vs. no need to free hands before takeover, (4) overall effect size of similarity between NDRTs.
METHODS
Search Procedure Articles were searched for in following databases (for the last time in January 2020): IEEE Xplore, Web of Science and APA PsycArticles. Search terms were “non-driving related task AND automated driving” and “non-driving related activities AND automated driving”. Additionally, the reference list of a catalogue on non-driving related activities developed in the German Ko- HAF project (Hohm et al. 2018) was screened.
Inclusion Criteria The following criteria were applied to select studies for this meta-analysis: (1) articles had to be full-text reports of a quantitative study written in English or German language, (2) the implemented SAE Level 3 driving automation function had to fulfill all criteria as defined by SAE Standard J3016, (3) the Level 3 function was active until takeover by the in-vehicle human driver, (4) a manual driving phase of SAE Level 0 had to follow the Level 3 driving automation phase, (5) the ride had to take place in a passenger car, (6) articles had to include either a control group or another NDRT condition to allow for calculation of an effect size, and (7) articles had to include a takeover time measure.
Coding of Studies Effect Size Calculation Cohen’s d of the difference in takeover time between two NDRTs was prefe- rably directly derived from the primary study, or calculated based on reported means and standard deviations using formulas by Borenstein et al. (2009). If these data were not reported, effect sizes were either calculated using reported test statistics (t- or F-statistic) provided the numerator’s degrees of freedom was 1, or converted from other reported effect size measures. For effect sizes based on within-subject designs the correlation r is needed. If it was neither reported nor could be provided, r = .50 was assumed.
Overall Effects of Non-Driving Related Activities’ Characteristics 73
Similarity of Tasks Similarity of tasks was coded based on the cognitive processes they require (as outlined in the introductory section on the Modality shifting effect in the takeover situation at SAE Level 3). Following criteria were coded for each NDRT: phonological loop, visual system, spatial system, central executive, deliberate long-term memory retrieval. For each pair of compared NDRTs, the similarity of their resulting demand profiles was calculated.
Meta-Analytic Procedure and Analyses The meta-analytic procedure followed the guidelines proposed by Field and Gillett (2010) and was conducted using R version 4.1.0 and the robumeta package (Fisher and Tipton 2015). Meta-regression models were estimated using robust variance estimation methods and a correlated effects model with small sample correction. Robust variance estimation methods were used because of the complex data structure that includes multiple effect size estimates drawn from the same or overlapping samples, which violates the traditional meta-analysis’ assumption of independent effect sizes. We deli- berately did not reduce data to one effect size estimate per study (by either averaging effect size estimates or selecting one effect size per primary study) since this would imply loss of information. Due to unknown covariance stru- cture underlying the effect sizes, we decided against conducting a multivariate meta-analysis. A multilevel meta-analysis was not chosen because the hiera- rchical effects model assumes independent sampling errors within clusters, which can be assumed to be not given in our effect size sample where part of the effect size estimates are based on overlapping participant samples. Robust variance estimation methods are not intended to be used for calcula- ting variance parameter estimates or testing null hypotheses on heterogeneity parameters, but more suitable for calculating meta-regression coefficients and mean effect sizes (Tanner-Smith et al. 2016) which is suitable for our aims and hypothesis. Average effect sizes were calculated by meta-regression using an intercept-only model. For the within-study effect size correlation, rho, the default value of .80 was used.
RESULTS
Table 1 shows a summary of average effect size estimates, reported separately in the following.
DISCUSSION
Our meta-analysis indicates that engagement in a NDRT increases takeover time. Activities that occupy users’ hands prolong takeover times compared to activities that do not occupy their hands. This attribute has the strongest effect on takeover times. Next, the visual distraction accompanying non- driving related activities prolongs takeover time. And lastly, dissimilarity to the driving task in terms of involved cognitive processes prolong takeo- ver time. Practical relevance can be deducted from the absolute size of the estimated effect sizes. Regarding theoretical implications, the meta-analysis
74 Shi and Bengler
Table 1. Summary of average effect sizes.
Table available in the original PDF paper.
t-statistics
Comparison No. of studies / outcomes
Equation reproduced from the original PDF.
Engagement in an active task compared to monitoring
Equation reproduced from the original PDF.
Need to put away task 11 / 47 d = .625, t(9.89) = 4.34, p < .01 Visual task compared to non-visual task 22 / 129 d = .096, t(10.4) = 2.31, p < .05.
Similarity of cognitive processing modules
provides first indication that task switching and modality shifting might be suitable to differentiate between NDRTs’ effects on takeover behavior.
Implications for Theory Acknowledging that the Level 3 driving automation is accompanied by a paradigm shift away from dual-task (from Level 0 to Level 2) towards task switching (e.g. Zeeb et al. 2017, Weaver and DeLucia 2020) is not new. By merging task switching and modality shifting, we contribute to founding a theoretical basis for differentiating NDRTs regarding their effects on take- over. Differentiation thereby considers both the Level 3 automated driving context and the diverse physical and cognitive demands an activity puts on the human user. The resulting overall effect size estimates are in accordance to our prior theoretical assumptions, indicating that the hypothesized pro- cesses of task switching and modality shifting seem to contribute to variance in takeover time.
Implications for Practice The absolute size of the overall effect size estimates indicates the practical relevance of the respective factors. The need to put away a task and the enga- gement in an active task compared to monitoring influence takeover time strongest. In practice, for instance the effect of the necessity to put away a task can be used in multiple regards. First, the probably most obvious infe- rence is to either render taking up remote devices unnecessary (suitable for activities involving electronic devices, such as smartphones and tablets) or create possibilities to quickly store or place items upon request to intervene (suitable for activities such as reading books or eating and drinking). Second, it is possible to constrain the range of possible non-driving related activities for a given system.
It should be noted that our meta-analysis refers to Level 3 driving auto- mation only. Inferences on other driving automation levels cannot be drawn from our analysis. Next, generally, conclusions about takeover quality must not be drawn since we focused on takeover time only. Unfortunately, nume- rous studies report only reaction times but not takeover performance in a systematic way. This means our results cannot suggest any better or worse
Overall Effects of Non-Driving Related Activities’ Characteristics 75
takeover depending on NDRTs’ characteristics. Our results rather indicate that absolute takeover times will be systematically underestimated when only considering situations in which the user has previously engaged in a passive monitoring task. This situation can be also assumed to be unrealistic as most users name non-driving related activities as one of the major advantages of automated driving.
REFERENCES
Borenstein, M., et al., 2009. Introduction to Meta-Analysis. Chichester, UK: John Wiley & Sons, Ltd. Bueno, M., et al., 2016 - 2016. How different mental workload levels affect the
take-over control after automated driving. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC): IEEE, pp. 2040–2045. Diederichs, F., Bischoff, S., and Reilhac, P., 2015. Welchen Einfluss hat das HMI?: Vergleich von Übergabezeiten in Notfallsituationen bei SAE Level 2 und 3 Auto- matisierung mit neuartiger Smartphone-Integration im direkten Fahrersichtfeld und mit Lenkradbedienung. In: C. Wienrich, T. Zander, and K. Gramann, eds. Trends in Neuroergonomics: 11. Berliner Werkstatt Mensch-Maschine-Systeme. Berlin: Universitätsverlag der TU Berlin, pp. 74–81. Dogan, E., et al., 2019. Effects of non-driving-related tasks on takeover performance
in different takeover situations in conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 62, pp. 494–504. Feldhütter, A., et al., 2018. Effect of prolonged periods of conditionally automated driving on the development of fatigue: With and without non-driving-related activities. Cognition, Technology & Work, 12 (1), pp. 309–318. Field, A.P. and Gillett, R., 2010. How to do a meta-analysis. The British journal of
mathematical and statistical psychology, 63 (Pt 3), pp. 665–694. Fisher, Z. and Tipton, E., 2015. robumeta: An R-package for robust variance estimation in meta-analysis. Gold, C., Berisha, I., and Bengler, K., 2016. Utilization of Drivetime – Performing
Non-Driving Related Tasks While Driving Highly Automated. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59 (1), pp. 1666–1670. Hohm, A., et al., 2018. KO-HAF - Kooperatives Hochautomatisiertes Fahren: (Projektübergreifender Schlussbericht). Jarosch, O., Bellem, H., and Bengler, K., 2019. Effects of Task-Induced Fatigue in
Prolonged Conditional Automated Driving. Human factors. Jarosch, O., et al., 2017. It’s Out of Our Hands Now! Effects of Non-Driving Related Tasks During Highly Automated Driving on Drivers’ Fatigue. In: Driving Assessment Conference, pp. 319–325. Klauer, K.C. and Zhao, Z., 2004. Double dissociations in visual and spatial short-
term memory. Journal of experimental psychology. General, 133 (3), pp. 355–381. Köhn, T., et al., 2019. Improving take-over quality in automated driving by interrupting non-driving tasks. In: W.-T. Fu, et al., eds. Proceedings of the 24th International Conference on Intelligent User Interfaces. New York, NY, USA: ACM, pp. 510–517. Körber, M., et al., 2016. The influence of age on the take-over of vehicle control in
highly automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 39, pp. 19–32.
76 Shi and Bengler
Louw, T., Merat, N., and Jamson, H., 2015. Engaging with Highly Automated Driving: To be or Not to be in the Loop? In: Driving Assessment Conference, pp. 190–196. Naujoks, F., et al., 2019. Noncritical State Transitions During Conditionally Auto-
mated Driving on German Freeways: Effects of Non-Driving Related Tasks on Takeover Time and Takeover Quality. Human factors, 1–18. Petermann-Stock, I., et al., 2013. Wie lange braucht der Fahrer? Eine Analyse zu Übernahmezeiten aus verschiedenen Nebentätigkeiten während einer hoch- automatisierten Staufahrt. 6. Tagung Fahrerassistenzsysteme. Der Weg zum automatischen Fahren. Radlmayr, J., et al., 2014. How Traffic Situations and Non-Driving Related Tasks
Affect the Take-Over Quality in Highly Automated Driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58 (1), pp. 2063–2067. Radlmayr, J., et al., 2018a. Peripheral Monitoring of Traffic in Conditionally Automated Driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62 (1), pp. 1828–1832. Radlmayr, J., Fischer, F.M., and Bengler, K., 2018b. The Influence of Non-driving
Related Tasks on Driver Availability in the Context of Conditionally Automa- ted The Influence of Non-Driving Related Tasks on Driver Availability in the Context of Conditionally Automated Driving. Volume VI: Transport Ergono- mics and Human Factors (TEHF), Aerospace Human Factors and Ergonomics, pp. 295–304. Repovs, G. and Baddeley, A., 2006. The multi-component model of working memory: explorations in experimental cognitive psychology. Neuroscience, 139 (1), pp. 5–21. Rubinstein, J.S., Meyer, D.E., and Evans, J.E., 2001. Executive control of cogni-
tive processes in task switching. Journal of experimental psychology. Human perception and performance, 27 (4), pp. 763–797. SAE International/ISO, 2021. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. 400 Commonwealth Drive, Warrendale, PA, United States: SAE International (J3016). Available from: https://www.sae.org/standards/content/j3016_201806/ [Accessed 17 Mar 2021]. Schartmuller, C., et al., 2019. Type-o-Steer: Reimagining the Steering Wheel for
Productive Non-Driving Related Tasks in Conditionally Automated Vehicles. In: 2019 IEEE Intelligent Vehicles Symposium (IV): IEEE, pp. 1699–1706. Shi, E., et al., 2020. The Principles of Operation Framework: A Comprehensive Classification Concept for Automated Driving Functions. SAE International Journal of Connected and Automated Vehicles, 3 (1). Soares, S., et al., 2021. Takeover performance evaluation using driving simulation:
a systematic review and meta-analysis. European Transport Research Review, 13 (1). Spence, C., Nicholls, M.E., and Driver, J., 2001. The cost of expecting events in the wrong sensory modality. Perception & psychophysics, 63 (2), pp. 330–336. Tanner-Smith, E.E., Tipton, E., and Polanin, J.R., 2016. Handling Complex Meta-
analytic Data Structures Using Robust Variance Estimates: a Tutorial in R. Journal of Developmental and Life-Course Criminology, 2 (1), pp. 85–112. Vogelpohl, T., Gehlmann, F., and Vollrath, M., 2020. Task Interruption and Control Recovery Strategies After Take-Over Requests Emphasize Need for Measures of Situation Awareness. Human factors, 62 (7), pp. 1190–1211.
Overall Effects of Non-Driving Related Activities’ Characteristics 77
Vogelpohl, T., et al., 2018. Transitioning to manual driving requires additional time after automation deactivation. Transportation Research Part F: Traffic Psychology and Behaviour, 55, pp. 464–482. Wan, J. and Wu, C., 2018. The Effects of Vibration Patterns of Take-Over Requ-
est and Non-Driving Tasks on Taking-Over Control of Automated Vehicles. International Journal of Human–Computer Interaction, 34 (11), pp. 987–998. Wandtner, B., 2018. Non-driving related tasks in highly automated driving: Effects of task characteristics and drivers’ self-regulation on take-over performance. Inaugural-Dissertation. Julius-Maximilians-Universität Würzburg. Wandtner, B., Schömig, N., and Schmidt, G., 2018a. Effects of Non-Driving Related
Task Modalities on Takeover Performance in Highly Automated Driving. Human factors, 60 (6), pp. 870–881. Wandtner, B., Schömig, N., and Schmidt, G., 2018b. Secondary task engagement and disengagement in the context of highly automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 58, pp. 253–263. Wandtner, B., et al., 2018. Non-driving related tasks in highly automated driving –
Effects of task modalities and cognitive workload on take-over performance. In: AmE 2018 - Automotive meets Electronics; 9th GMM-Symposium, pp. 8–13. Weaver, B.W. and DeLucia, P.R., 2020. A Systematic Review and Meta-Analysis of Takeover Performance During Conditionally Automated Driving. Human factors, 18720820976476. Wintersberger, P., et al., 2018. Let Me Finish before I Take Over. In: AutomotiveUI
‘18, ed. Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI ‘18. New York, New York, USA: ACM Press, pp. 53–65. Wu, Y., et al., 2019. Assessing the Mental States of Fallback-Ready Drivers in Automated Driving by Electrooculography. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC): IEEE, pp. 4018–4023. Yoon, S.H. and Ji, Y.G., 2019. Non-driving-related tasks, workload, and takeover
performance in highly automated driving contexts. Transportation Research Part F: Traffic Psychology and Behaviour, 60, pp. 620–631. Zeeb, K., et al., 2017. Why is steering not the same as braking?: The impact of non-driving related tasks on lateral and longitudinal driver interventions during conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 50, pp. 65–79. Zhang, B., et al., 2019. Determinants of take-over time from automated driving: A
meta-analysis of 129 studies. Transportation Research Part F: Traffic Psychology and Behaviour, 64, pp. 285–307.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-36-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional