Sergey Drobinsky, Patrick Korte, Rastislav Pjontek, Armin Janß, Verena Nitsch, Klaus Radermacher (2023). Development of a Prospective Method for Rating Surgical Task Workloads. In: Emerging Technologies in Healthcare and Medicine. AHFE Open Access, vol. 116, pp. 284–293. https://doi.org/10.54941/ahfe1004382
Surgical adverse events can have serious consequences for patients ranging from temporary injuries to death. Thereby, up to 40% of surgical adverse events are preventable and over 60% of causal factors were found to be linked to human factors. To improve surgical performance and safety, computer-assisted surgical (CAS) systems can be used to reduce excessive workloads. This paper presents a method for prospective assessment of surgical task workloads. S-TAWL, developed with the support of a senior neurosurgeon and a usability engineer, consists of three parts: surgical task decomposition, workload rating scale application, and performance shaping factors characterization. For the proposed rating scales, composed of reference operators, relative workloads were determined by 11 neurosurgeons through pairwise comparison. Afterwards, one senior neurosurgeon, not involved in method development, analysed workloads of four common surgical tasks with the proposed method S-TAWL and a reference workload rating method Surg-TLX. Qualitatively, S-TAWL provides more detailed information about workloads with respect to human resources compared to the reference method. Quantitatively, however, the reliability of the results is still limited, as indicated by high standard deviations. Further research is needed to develop reliable and valid rating scales, compute compound workloads and identify overloads. Incorporating quantitative workload assessment in prospective human performance analysis will provide valuable information for targeted model-based design of assistance systems, supporting safe and successful surgery in the future.
Surgical adverse events can have serious consequences for patients ranging from temporary injuries to death [Schwendimann et al., 2018]. Up to 40% of surgical adverse events are preventable and over 60% of causal factors were found to be linked to human factors [Zegers et al., 2011]. Workloads, i.e., all external influences acting on a human, have an impact on human performance [VDI 4006 Part 1 2015]. Inadequate workloads can lead to human performance deficiencies and therefore must be avoided for safe and successful surgery [Suliburk et al., 2019].
Development of a Prospective Method for Rating Surgical Task Workloads 285
To reduce overloads computer-assisted surgical (CAS) systems, such as cooperative robotic assistants, can be used [Schleer et al., 2021]. How- ever, introducing technical support systems into the operating room can also increase the overall complexity of the socio-technical system, therefore increasing risks of human errors [Liu et al., 2021]. Interaction with complex systems, forces surgeons to allocate more attention on monitoring and con- trolling the support systems and less on the patient. Out-of-loop unfamiliarity and associated risks, like overreliance, skill deterioration, mode confu- sion, and reduced situational awareness can cause human errors, limiting the safety and usability of technical support systems [Endsley and Kiris 1995].
Model-based usability and human-centered risk analysis can support evidence-based design and integration of surgical assistance systems [Janß 2015] [Catchpole et al., 2022]. Various methods and models like CPM-GOMS [John and Gray 1995], Multiple Resource Theory [Wickens 2008], and mAIXuse [Janß 2015] have been proposed in literature. In mil- itary contexts, prospective workload assessment methods such as LHX or TAWL have been developed to predict task loads [Hamilton and Bierbaum 1990]. These methods involve domain experts assessing the workloads of tasks by assigning reference tasks with known workloads [Mitchell 2009]. This paper presents a method for surgical task and workload assessment (S-TAWL).
MATERIAL AND METHODS S-TAWL is developed iteratively in accordance with the problem-solving cycle [Haberfellner et al., 2019] supported by a senior neurosurgeon and a senior usability engineer. Therefore, existing process and workload models like GOMS [Card et al., 2008], mAIXuse [Janß 2015], MRT [Wickens 2008], SPM [Neumuth 2015], TAWL [Bierbaum et al., 1989], Imprint [Mitchell 2009], and CTAWC [Knisely et al., 2021] were reviewed, adapted, and extended where necessary. The development and evaluation of S-TAWL were conducted in German and later translated into English for this publication, with the assistance of a native speaker.
S-TAWL consists of three parts: task decomposition, workload assess- ment using rating scales, and performance shaping factor characterization (see Figure 1). The application scope of the proposed workload assessment method is constrained to neurosurgery. Task interdependencies and resource conflicts of parallel activities are not considered and must be evaluated separately.
In a first step, the surgical process is decomposed into elementary com- ponents by a domain expert and a modelling expert. For the proposed task decomposition, terms and concepts are defined in accordance with Lalys and Janin [2014]. A surgical process consists of surgical phases that consist of a set of surgical tasks performed to achieve a surgical objective. A surgical task forms the smallest logical unit from a medical domain perspective. From a human factors domain perspective, surgical tasks are further decomposed into surgical activities that form a perception-cognition-action cycle [IEC 62366–1 2015]. Surgical action-type activities are also called surgical gestures or surgemes in the literature [Lin et al., 2006].
Figure 1: Structure of the proposed prospective workload assessment method S-TAWL.
In a second step, relative workload magnitudes of surgical activities are assessed by choosing suitable operators, i.e., standardized generic activities [John and Gray 1995]. Then, corresponding rating scales composed of refer- ence operators with known relative workloads are applied. Thereby, various sets of operators were proposed for surgical and non-surgical process mod- elling in the literature [Card et al., 1980; Card 1981; Rasmussen 1983; Hamilton and Bierbaum 1990; John and Gray 1995; Hoc and Lemoine 1998; Troccaz et al., 1998; Anderson et al., 2001; Holleis et al., 2007; Estes 2021]. For the presented study a relevant subset of operators was identified and clinically meaningful reference operators were defined in workshops with clinical experts (Table 1). Further components and sources of workload are analyzed by identifying and characterizing relevant PSFs [VDI 4006 Part 1 2015]. Within the scope of this paper, a formative evaluation of the proposed rating scales is presented.
For perception and action operators, relative workloads on a 7-point scale (1- lowest workload) were determined through pairwise comparison by 11 neurosurgeons from the department of neurosurgery of the university hospital Aachen. For cognition operators, reference operators and rela- tive workload magnitudes were defined based on an adaptation of Bloom’s taxonomy [Anderson et al., 2001].
S-TAWL was then evaluated by analyzing the workloads of four common surgical tasks in posterior cervical spine fixation. This surgical procedure is an example of a very complex surgical procedure associated with high risks. The examined tasks were: (1) navigated drill guide alignment, (2) non-navigated drill guide alignment, (3) canulated pedicle screw placement, and (4) non- canulated pedicle screw placement. First, these tasks were decomposed into surgical activities with the support of a senior neurosurgeon. Subsequently, a different senior neurosurgeon from the university hospital Aachen, who had conducted approximately 50 cervical spine fixations, assessed the work- loads. For each activity, the surgeon was asked to identify suitable reference operators that they believed to be most similar in terms of workload. The reference operators were presented without numeric workload values and were not ordered on a rating scale. This approach ensured that the surgeon assigned reference operators based solely on relative comparisons, rather than absolute workload assessments. Cognitive activities were not predetermined;
Development of a Prospective Method for Rating Surgical Task Workloads 287
instead, the surgeon was asked to select all cognitive operators they deemed necessary to complete each surgical task under analysis. Following the work- load assessment, the surgeon was asked to indicate which surgical activity presented the greatest challenge in assigning a reference operator. Surg-TLX [Wilson et al., 2011], an adaptation of NASA-TLX, was used as a reference method to assess workloads. The participating surgeon was not involved in the development of S-TAWL.
RESULTS
For perception and action operators, the identified mean relative workloads and standard deviations are shown in Table 1. Reference operator workload estimates show that the proposed rating scales are not equidistant. Due to high standard deviations (up to ± 2 points) some rating scales are ambiguous, such as for the operator primarily coordinative actions. For the two operators visual search and haptic interpretation, two reference operators within one rating scale were determined to have the same workload. Furthermore, no rating scale covers the entire 7-point range. The smallest rating scale span is 2.9 points (visual search, primary physical actions) and largest rating scale span is 4.5 points (visual identification).
Workloads assessed with the proposed method S-TAWL and the refer- ence method Surg-TLX are shown in Table 2. For drill guide alignment, reported total workload with Surg-TLX (0-lowest, 100-highest) was sim- ilar with (32) and without (33) navigation. Raw ratings were highest for mental demands and task complexity and lowest for physical demands. For pedicle screw placement, total weighted workloads are lower for placing can- ulated screws (22) than for placing non-canulated screws (35). Raw ratings indicated that for canulated screw placement physical demands were high- est, while for non-canulated screw placement mental demands were highest. Weights determined through pairwise comparison show that mental demands (w = 5) are more important than any other factor, while physical demands are not relevant at all (w = 0). Therefore, despite being rated higher for pedi- cle screw placement compared to drill guide alignment, physical demands do not contribute at all to the total weighted workload.
With S-TAWL main differences for assessed workloads were related to perception type activities. S-TAWL task decomposition suggests that drill guide alignment only includes visual perception, while pedicle screw place- ment includes visual and haptic perception. For navigated and non-navigated drill guide alignment, workloads for monitoring and verifying the alignment using anatomical landmarks were rated highest. Alignment verification in X-ray imaging was rated as a high-workload visual identification task, while monitoring alignment on a navigation display was rated as a low-workload visual search task. For pedicle screw placement, workloads for visual checks of the screw alignment are rated lower for canulated screws (2.1) than for non-canulated screws (4.4). Visual screw depth monitoring is rated as the lowest visual search task (2.1) for non-canulated screws and as the lowest visual identification task (2.5) for canulated screws. For haptic alignment control, different reference operators were assigned, however both having the
Table 1. Rating scales and relative workloads for cognition-type surgical operators adapted from [Anderson et al., 2001]. Rating scales for perception-
Table available in the original PDF paper.
and action-type surgical operators and relative workloads identified in expert workshops and surveys (1-lowest workload, 7-highest workload).
Development of a Prospective Method for Rating Surgical Task Workloads 289
Table 2. Task decomposition and workload assessment of four common tasks in posterior cervical spine fixation using S-TAWL. Reference workloads
Table available in the original PDF paper.
determined using Surg-TLX method (bar height corresponds to raw TLX values, bar width to weighting factors).
same identified workload (5.1). Regarding cognition type activities, all four tasks require recalling (1), inferring (2.5) and planning (6.5). For navigated drill guide alignment and canulated pedicle screw insertion decision-making (simple) (4.5) is chosen, while for non-navigated drill guide alignment and non-canulated pedicle screw placement decision-making (difficult) (5.5) is chosen. For non-canulated screw placement 8 reference operators are chosen, while for canulated screw placement only 5 are chosen. Regarding action type activities, primarily physical actions were rated high (5.7), while workloads for primarily coordinative actions were rated medium (4.4-4.7) for all four tasks. In applying the method, the surgeon reported that they struggled most with assigning cognitive reference operators.
DISCUSSION
As human factors are a significant contributor to preventable adverse events in surgery, accurate and reliable workload assessment is key for safe and suc- cessful surgery. However, surgical workload assessment is challenging due to task variability and a large number of interdependent influencing factors. While performance shaping factors indicate sources of workload, assessment of compound workload magnitudes of individual activities is challenging [Heard et al., 2018]. In this paper a prospective workload assessment method based on rating scales composed of reference tasks was proposed. The limited scope of the formative evaluation study with only one participating surgeon does not allow for final conclusions about reliability or validity of the pro- posed method. However, some general conclusions about the design of the method can be drawn for future considerations.
Comparing the assessed workloads using Surg-TLX and S-TAWL is chal- lenging due to differences in workload components and the current lack of a method to compute compound workloads with S-TAWL. Nonetheless, some results show similar tendencies: Both methods rate non-canulated pedi- cle screw placement to be more demanding than canulated pedicle screw placement. With Surg-TLX ratings are higher for complexity and mental demands, while S-TAWL shows increased workloads for visual perception activities. However, there are also some disparities: Surg-TLX reports higher raw ratings for physical demands in pedicle screw placement compared to drill guide alignment, while S-TAWL rates physical demands similarly high for both tasks. However, for Surg-TLX, differences in raw ratings do not affect compound workloads because the determined weighting factor is 0.
By enabling a more detailed assessment of workloads related to percep- tual, cognitive, and motor demands S-TAWL facilitates the identification of resource conflicts and excessive workloads when combined with existing model-based risk analysis methods [Dell’Anna-Pudlik 2022]. However, com- putation of compound workloads with S-TAWL remains an open research question. To compare workloads across multiple operators weighting fac- tors may be added [Virtanen et al., 2022]. Furthermore, operator interfer- ences could be considered using a conflict-matrix as proposed by Wickens [2002]. Finally, researching limits for acceptable compound workloads with
Development of a Prospective Method for Rating Surgical Task Workloads 291
respect to human performance capacity is crucial to determining excessive workloads.
When using S-TAWL’s rating scales, which are composed of reference operators, workloads are assessed through relative comparison. Studies indi- cate that relative rating methods, such as paired comparisons, appear to have greater validity and reliability than absolute rating scales, such as the NASA TLX [Vidulich and Tsang 1987]. However, the presented evaluation study showed that adaptation of the proposed rating scales is necessary. The proposed rating scales for perception and action operators exhibit limited reliability as indicated by high standard deviations in the determined rela- tive workloads. Measurement sensitivity is affected by scale non-linearity, a limited scale span, and varying numbers of scale steps. Additionally, there are potential limitations in the measurement range due to scale attenuation effects. Regarding cognition operators, further research is necessary to vali- date the proposed linear rating scale, which is based on Bloom’s taxonomy, originally developed to describe student learning.
For the assignment of reference operators to surgical activities, the optimal choice of reference operators is an open research question. Highly spe- cific reference operators from the user’s field of expertise may improve the assignability, while more general operators, such as those form a surgical training curriculum [Acosta et al., 2018], may increase the method’s applica- tion scope. During the study, the participating neurosurgeon reported having the most difficulty with the assigning cognitive operators. This may be either because cognitive activities were not further specified or because cognitive reference operators were not selected from clinical examples. In addition, when multiple operators were selected it was unclear whether the assigned reference operators referred to separate cognitive activities. Due to the hierar- chical structure of Bloom’s taxonomy higher-level reference operators include lower-level operators.
CONCLUSION
Human factors play a significant role in preventable adverse events, high- lighting the importance of accurate and reliable workload assessment to proactively identify risks arising from excessive workloads. The proposed task decomposition and operator definition in S-TAWL offer valuable insights into the workloads associated with perceptual, cognitive, and motor demands of surgical tasks. However, determining compound workloads still presents a challenge. Future research is essential to incorporate task interrelation- ships, resource utilization, and all relevant performance shaping factors in the evaluation of surgical task workloads. Extending existing model-based risk analysis methods with an accurate and comprehensive workload analy- sis can support evidence-based design of assistance systems and promote safe and successful surgery in the future.
ACKNOWLEDGMENT
This research work has been funded within the project “5G FORUM – Flexi- ble operating room use and monitoring” by the German Federal Ministry for Economic Affairs and Climate Action, grant no. 01MJ22009C.
REFERENCES
Acosta, D., Castillo-Angeles, M., Garces-Descovich, A., Watkins, A. A., Gupta, A., Critchlow, J. F., and Kent, T. S. 2018. Surgical Practical Skills Learning Curricu- lum: Implementation and Interns’ Confidence Perceptions. Journal of surgical education 75, 2, 263–270. Anderson, L. W., Bloom, B. S., and Krathwohl, D. R. 2001. A taxonomy for learning,
teaching, and assessing. Longman. Bierbaum, C. R., Szabo, S. M., and Aldrich, T. B. 1989. Task Analysis of the UH-60 Mission and Decision Rules for Developing a UH-60 Workload Prediction Model. Volume 1. Summary Report. Anacapa Sciences Inc, Ft Rucker, Al. Card, S. K. 1981. The Model Human Processor: A Model for Making Engineering
Calculations of Human Performance. Proceedings of the Human Factors Society Annual Meeting 25, 1, 301–305. Card, S. K., Moran, T. P., and Newell, A. 1980. The keystroke-level model for user performance time with interactive systems. Commun. ACM 23, 7, 396–410. Card, S. K., Moran, T. P., and Newell, A. 2008. The psychology of human-computer
interaction. CRC Press Taylor and Francis Group, London. Catchpole, K., Cohen, T., Alfred, M., Lawton, S., Kanji, F., Shouhed, D., Nemeth, L., and Anger, J. 2022. Human Factors Integration in Robotic Surgery. Human factors, 187208211068946. Dell’Anna-Pudlik, J. 2022. Modellbasiertes Risikomanagement offen vernet- zter Chirurgie-Systeme für eine zentrale konfigurierbare Fußbedieneinheit. Dissertation. Endsley, M. R., and Kiris, E. O. 1995. The Out-of-the-Loop Performance Problem
and Level of Control in Automation. Human factors 37, 2, 381–394. Estes, S. 2021. Cogulator: A Primer. [White Paper]. The MITRE Corporation. Haberfellner, R., Weck, O. L. de, Fricke, E., and Vössner, S. 2019. Systems Engineering. Fundamentals and Applications. Springer eBook Collection. Birkhäuser, Cham. Hamilton, D. B., and Bierbaum, C. R. 1990. Task Analysis/Workload (TAWL):
A Methodology for Predicting Operator Workload. Proceedings of the Human Factors Society Annual Meeting 34, 16, 1117–1121. Heard, J., Harriott, C. E., and Adams, J. A. 2018. A survey of workload assessment algorithms. IEEE Trans. Human-Mach. Syst. 48, 5, 434–451. Hoc, J.-M., and Lemoine, M.-P. 1998. Cognitive Evaluation of Human-Human and
Human-Machine Cooperation Modes in Air Traffic Control. The International Journal of Aviation Psychology 8, 1, 1–32. Holleis, P., Otto, F., Hussmann, H., and Schmidt, A. 2007. Keystroke-level model for advanced mobile phone interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, M. B. Rosson and D. Gilmore, Eds. ACM, New York, NY, USA, 1505–1514. IEC 62366–1. 2015. Medical devices - Part 1: Application of usability engineering
to medical devices, 62366–1. Janß, A. 2015. Modellbasierte Risikoanalyse und -behandlung sicherheitskritischer Mensch-Maschine-Schnittstellen in der Medizintechnik. Dissertation, RWTH Aachen; Shaker Verlag GmbH. John, B. E., and Gray, W. D. 1995. CPM-GOMS: an analysis method for tasks
with parallel activities. In Conference companion on Human factors in computing systems, 393–394. Knisely, B. M., Joyner, J. S., and Vaughn-Cooke, M. 2021. Cognitive task analysis and workload classification. MethodsX 8, 101235.
Development of a Prospective Method for Rating Surgical Task Workloads 293
Lalys, F., and Jannin, P. 2014. Surgical process modelling: a review. International Journal of Computer Assisted Radiology and Surgery 9, 3, 495–511. Lin, H. C., Shafran, I., Yuh, D., and Hager, G. D. 2006. Towards automatic skill eval-
uation: detection and segmentation of robot-assisted surgical motions. Computer aided surgery : official journal of the International Society for Computer Aided Surgery 11, 5, 220–230. Liu, P., Zhang, R., Yin, Z., and Li, Z. 2021. Human errors and human reliability.
In Handbook of human factors and ergonomics, G. Salvendy and W. Karwowski, Eds. Wiley, 514–572. Mitchell, D. K. 2009. Workload analysis of the crew of the Abrams V2 Sep. Phase I baseline IMPRINT model. Army Research Lab. Neumuth, T. 2015. Surgical Process Modeling - Theory, Methods, and Applications Rasmussen, J. 1983. Skills, rules, and knowledge; signals, signs, and symbols, and
other distinctions in human performance models. IEEE Trans. Syst., Man, Cybern. SMC-13, 3, 257–266. Schleer, P., Vossel, M., Heckmann, L., Drobinsky, S., Theisgen, L., La Fuente, M.
de, and Radermacher, K. 2021. Usability of cooperative surgical telemanipulation for bone milling tasks. International Journal of Computer Assisted Radiology and Surgery 16, 2, 311–322. Schwendimann, R., Blatter, C., Dhaini, S., Simon, M., and Ausserhofer, D. 2018. The occurrence, types, consequences and preventability of in-hospital adverse events - a scoping review. BMC health services research 18, 1, 521. Suliburk, J. W., Buck, Q. M., Pirko, C. J., Massarweh, N. N., Barshes, N. R., Singh,
H., and Rosengart, T. K. 2019. Analysis of Human Performance Deficiencies Associated With Surgical Adverse Events. JAMA network open 2, 7, e198067. Troccaz, J., Peshkin, M., and Davies, B. 1998. Guiding systems for computer-assisted surgery: introducing synergistic devices and discussing the different approaches. Medical image analysis 2, 2, 101–119. VDI 4006 Part 1. 2015. Human reliability - Ergonomic requirements and methods
of assessment, 4006 Part 1. Vidulich, M. A., and Tsang, P. S. 1987. Absolute Magnitude Estimation and Relative Judgement Approaches to Subjective Workload Assessment. Proceedings of the Human Factors Society Annual Meeting 31, 9, 1057–1061. Virtanen, K., Mansikka, H., Kontio, H., and Harris, D. 2022. Weight watchers:
NASA-TLX weights revisited. Theoretical Issues in Ergonomics Science 23, 6, 725–748. Wickens, C. 2002. Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science 3, 2, 159–177. Wickens, C. D. 2008. Multiple resources and mental workload. Human factors 50,
3, 449–455. Wilson, M. R., Poolton, J. M., Malhotra, N., Ngo, K., Bright, E., and Masters, R. S. W. 2011. Development and validation of a surgical workload measure: the surgery task load index (SURG-TLX). World journal of surgery 35, 9, 1961–1969. Zegers, M., Bruijne, M. C. de, Keizer, B. de, Merten, H., Groenewegen, P. P., van
der Wal, G., and Wagner, C. 2011. The incidence, root-causes, and outcomes of adverse events in surgical units: implication for potential prevention strategies. Patient safety in surgery 5, 13.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Emerging Technologies in Healthcare and Medicine
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-92-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional