Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Multidimensional, Intuitive and Augmented Interaction Models for Robotic Surgery

Department of Engineering, University of Campania “Luigi Vanvitelli” Aversa (CE), 81031, Italy ↗
Publication year
Digital Object Identifier10.54941/ahfe1002320

Abstract

Surgical robotics in operating rooms is an innovative and rapidly evolving field, and performance levels need to be improved. Despite technological advances, there are still many limitations in surgeon-robot interaction, such as the lack of tactile feedback from the surgeon and visualization issues arising from the surgeon's position relative to the operating table. Therefore, among the challenges of robotic surgery is the design of efficient and ergonomic human-machine interaction systems that can improve and enhance the capabilities of the surgeon and the robot (Boyraz et al., 2019) while ensuring risk reduction and high levels of ergonomics and safety. This will improve the surgeon's perception and eliminate possible accidental contact with tissues and injuries. In this scenario, the contribution illustrates the reconnaissance and analysis activities carried out to identify the limitations and advantages of the current interfaces and visualization technologies applied to robotic surgery to verify their usability and the ways of surgeon-robot interaction.

INTRODUCTION

Safety in the working environment is a very important topic of study that requires continuous updates to improve performance levels through pre- vention planning and the influence of environmental factors and work organization by respecting ergonomic principles. The paper documents the survey and analysis activities carried out on current interfaces and visualiza- tion technologies applied to robotic surgery to highlight their limitations and opportunities.

By analyzing the behavioural and decision-making processes involved in human-robot collaboration and interaction, it has been possible to identify interaction factors for the control of surgical robotics and for the configura- tion of adaptive, intuitive and natural multimodal interfaces that reduce the probability of error and improve human-machine performance.

SAFETY IN THE OPERATING ROOM Currently, there are no universal, specific and detailed standards to certify the safety of a robotic surgery system, and industrial robot safety procedures

© 2022. Published by AHFE Open Access. All rights reserved. 130

Multidimensional, Intuitive and Augmented Interaction Models for Robotic Surgery 131

are not always transferable to the operating room (OR) (Dibekci and Bebek, 2018). The standards that currently govern human-robot collaboration are ISO10218-1:2011, ISO 10218-2:2011 “Robots and robotic devices” and ISO/TS15066:2016 “Robots and robotic devices - Collaborative robots”. These standards, which refer only to the industrial sector, include risk analysis techniques as a fundamental requirement for collaborative robot applications (Inam et al., 2018).

ISO/TS 15066 states that risk reduction is achieved by ensuring the separa- tion of the operator from the robotic system. For this reason, it is necessary to identify the user tasks associated with those of the robotic system, identifying all combinations of risks and tasks: the transition from a non-collaborative to a collaborative operation and the movement of the robotic system until the task is completed. Risk reduction measures are based on safe design resulting from substitution and the application of standards that safeguard personnel, ensuring a safe state.

For risk identification, ISO/TS 15066 according to ISO 101218 divides the analysis into risks related to the collaboration with the robot, to the robotic systems and related to the application.

Among the characteristics identified in this risk identification process, the characteristics of the robot (such as weight, speed, force, etc.), the position and movement of the operator concerning the parts of the robotic system, the design and positioning of the robot control device and ergonomic design requirements including accessibility, use and control modes are considered. However, to date, there are insufficient standards to ensure the safety of col- laborative systems (Inam et al., 2018) and there are no specific measures that can regulate robots placed in surgical environments.

To determine the level of safety, ISO 13849-1:2015 “Safety of machinery - Safety-related parts of control systems - Part 1: General principles for design” and IEC 62061:2021 “Safety of machinery - Functional safety of safety- related control systems”, standards have been introduced, which are based on the severity of the hazard, the frequency of exposure and the possibility of reducing this risk.

The industrial robot is defined in ISO 8373:2012 “Robots and robotic devices -Vocabulary” as: ≪a programmed actuated mechanism with a cer- tain degree of autonomy, which moves within its environment, to perform expected tasks≫, while the service robot is defined as: ≪a robot that per- forms useful tasks for people or equipment, excluding industrial automation applications≫.

These definitions do not consider movement mechanisms as a robotic system, therefore, surgical robots such as the da Vinci Surgical System are excluded, as they do not have decision-making capabilities but only tele-operated functions.

Only in 2021, this standard was updated, including among the definitions the concept of Medical Robot, as: ≪robot intended for use as medical ele- ctrical equipment or medical electrical systems≫(ISO 8373:2021 “Robotics – Vocabulary”), in addition, it is specified that this type of robot cannot be considered as an industrial robot or service robot.

The diversity of functions and configurations makes it extremely difficult to regulate and standardize the field of surgical robotics, which in many countries do not require approval like medical devices. The European Union requires compliance with the Medical Devices Regulation and the Machi- nery Directive. The safety requirements and standards for medical devices are derived from the applications in which they are used. To determine which standards are applicable, it is necessary to identify the end-use, clinical functions and intended users.

THE CHALLENGES OF HUMAN-ROBOT INTERACTION Robots as active systems have physical, social and emotional connotations that distinguish them from computers and therefore interact in physical space with users (Young et al., 2011). This implies changes in the interaction pro- cesses between users and new technologies and the definition of new ways of evaluating human-robot interaction (Cunningham et al., 2013).

For the robotic surgery Aaltonen and Wahlström (2018a) identify five areas of user-centred Human-Robot Interaction (HRI) research: 1 – ergonomics and workload: it is shown that in robotic surgery impro- vements have been made to the working environment, with a reduction in physical and cognitive workload, due to ergonomic postures and a redu- ction in mental stress, which varies according to the surgeon’s experience and training (Hubert et al., 2013) (Lee et al., 2014). 2 – visual and haptic feedback: the importance of using 3D displays to obtain visual signals is highlighted, as surgeons have found a significant improvement in the time taken to perform procedures. According to Van Der Meijden et al. (2009), the introduction of haptic feedback can reduce surgical errors and improve patient safety. 3 – human factors in image-guided navigation: Current image-guided navigation systems are found to support the surgeon’s spatial orientation during navigation phases, leading to improvements in surgeon performa- nce and patient safety. However, studies show that such systems are not yet efficient as they cause increased mental workload. 4 – learning and training issues: it is emphasized that learning in robotic surgery varies according to the complexity of the procedures, the sur- geon’s experience in using robotic technology and the knowledge of the procedure to be performed. To support the training and learning of surge- ons and OR operators, tools such as virtual reality simulators have been introduced, which when applied to complex procedures have brought substantial improvements (Albani and Lee, 2007). According to Schreu- der et al. (2012), this learning modality will play an important role in the training of future robotic surgeons. 5 – team-level interaction: there was variation in communication models (verbal and non-verbal), timing and workflows depending on user experience (Cunningham et al. (2013) and the type of robot to be used.

Due to the widespread lack of surgeon confidence in robotic systems, the perceptual and user acceptance components inherent in the human-robot

Multidimensional, Intuitive and Augmented Interaction Models for Robotic Surgery 133

relationship and collaboration are critical factors for HRI in the operating room (Smids et al. 2019). According to Gillan (2020), the fundamental cha- racteristics that robotic systems should have related to performance such as reliability, predictability, fault tolerance and usability, to provide greater user confidence

MODES OF INTERACTION AND COLLABORATION The users-operators of collaborative surgical robotic systems believe that such equipment increases precision, flexibility and control during surgical phases compared to traditional procedures and enables the performance of complex tasks. Many of the collaborative robotic systems developed for sur- gery include devices placed close to the operating field as an interface between the surgeon and the robot for visualization and control.

There are different types of robots that collaborate with users to perform different surgical procedures. In the literature review of the industrial field, several classifications of human-robot collaboration related to the ways of interaction and coexistence can be found (Formati et al., 2021). Among these, the proposal of Aaltonen et al. (2018b) is the most satisfactory for the application to the surgical field. This classification is based on four levels:

- no coexistence, where there is a physical separation between man and robot. Autonomous robots, such as the STAR suture system, can fit into this category. Such a surgical anastomosis robot “Smart Tissue Autonomous Robot”(Ficuciello et al., 2019) integrates several advanced technologies. The vision system is based on NIRF, i.e., optical tags placed in the intestinal tissue that when delivered to an optical contrast agent or fluorescent probe, emit light in the near-infrared range. The NIRF camera follows the markers while the 3D camera records images of the surgical field and the combination of these data allows the “STAR” robot to remain stationary and focused on the target tissue. The robot processes the surgical plan using this data and modifies it during the operation when the tissue moves. The robot’s visual system has been described as “suprahuman”because it can recognize the type of tissue it is looking at, identify temperature and check the natural rhythm of organs, such as the heartbeat.

- coexistence, where users and robots work towards different goals in a shared space. One example is the ArtisZeego system, from Siemens Health- care, which is robotic instrumentation designed to enhance vascular, cardiac, cardiovascular, neurosurgery, trauma and orthopaedic surgery. The flexibi- lity and adaptability of the instrumentation are provided by the multi-axis structure that allows positioning according to the surgeon’s needs.

This helps to reduce fatigue during long procedures and increase com- fort and precision. The robotic system also features an imaging apparatus using x-ray technology and contrast media where blood vessels are made visi- ble during angiography. This creates a hybrid OR that increases workflow efficiency and patient care.

- cooperation, is when users and robots work towards a common goal and perform different tasks in a shared space.

Among these, we can identify the Robot Mazor Renaissance, which specia- lized in spinal surgery. From CT images, a 3D reconstruction of the vertebrae is made. This allows the software to virtually simulate the insertion of the screws, thus analyzing positions and trajectories and allowing the surgeon to plan the different stages of the operation that will be carried out by the robot with precision, using surgical instruments and fixation means.

- collaboration, in which users and robots work simultaneously towards the same goal in a shared space.

The MAKO haptic system is a collaborative surgical device configured to be manipulated by the users-operators while performing procedures such as bone implantation. During the procedure, the device provides tactile guidance to the surgeon by delimiting a virtual boundary that can be defined by a virtual tactile object generated by the computer system and associated with the patient’s anatomy. Thus, a relationship is established between the patient’s body and the instrument, defining the desired position, orientation, speed and/or acceleration.

When the instrument comes into contact with the virtual boundary, the haptic device guides the form of haptic feedback such as vibration or force- resistance feedback to the surgeon’s movement; thus, the virtual boundary guides the surgeon in cutting. In orthopaedic applications, the haptic device supports the criticality of unpredictable and non-repeatable bone prepara- tion, guiding the surgeon in the correct “sculpting” of the bone and allowing more precise and repeatable bone resections.

ADVANCED VISUALIZATION TO DESIGN SURGERY 5.0 Minimally invasive procedures require small incisions and the use of small instruments such as catheters. During such procedures, the surgeon cannot see and touch the patient directly, which is why medical imaging technologies such as X-ray and ultrasound imaging or navigation technologies are used to get images of the patient and guide their actions.

Despite the development of technologies in medical imaging for image guidance and advanced robot control of the human-machine interface, it is still not possible to fully replace many of the procedures that are performed through manual surgery (Haidegger, 2019).

The Vostars (Video-Optical See-Through Augmented Reality System) vie- wer developed by the University of Pisa, is a surgical navigator able to provide the surgeon with a view of the operating theatre, information about the pati- ent and information about the organs involved in the operation. This allows the surgeon to have all the necessary information in the action view (see Figure 1). Focusing on virtual objects implies that real objects are out of focus because the eye perceives them at two different distances, which is why it has not been possible so far to use virtual information to guide surgical ope- rations. Additional information about the patient and the operation has to be displayed on external monitors, forcing the surgeon to shift his gaze and concentration from the patient to the monitor, which is tiring and sometimes ineffective.

Multidimensional, Intuitive and Augmented Interaction Models for Robotic Surgery 135

Figure 1: Augmented Reality during surgery performed with Vostars (Biomed CuE, Vostars: la realtà aumentata entra in sala operatoria a Bologna, per la prima volta. 19 February 2020).
Original PDF Figure 1:

The Vostars viewer, complemented by a video camera, combines the ima- ges in front of the surgeon with the patient’s X-ray images, ensuring that the two remain perfectly coherent and in focus. In addition, during phases of sur- gery where accurate virtual guidance is not required, the viewer can become transparent, allowing the surgeon to have a natural view of the operating field.

Another case is ApoQlar, a three-dimensional visualization assistant that superimposes the virtual three-dimensional image on the patient’s body. The Virtual Surgery Intelligence (VSI) software uses Microsoft HoloLens mixed reality glasses supplied by Bechtle. It uses artificial intelligence to project data, such as three-dimensional magnetic resonance imaging or CT scans, onto the glasses, which are superimposed onto the patient’s body, rendering a full anatomical image of physical structures that can be moved freely around the room. Surgeons can see and interact with their real environment and have their hands free to operate.

The 3D glasses, integrated with Wi-Fi, allow the user to move freely around the room, recognize the patient’s position and place 3D objects in the most comfortable and useful for them to perform the necessary movements and interactions, even when the head is moving. Surgeons can adjust and remove layers and structures in the air using gestures and voice commands. The new technology makes it easier to perform operations, improves training for doctors and can also help inform patients before an operation.

CONCLUSION

A survey of the transformations and evolutions in the field of surgical robotics in operating rooms defines new multidimensional, intuitive and augmented

modes of interaction capable of increasing the ergonomics and usability of the surgeon, reducing stress and fatigue factors, ensuring safety and error tolera- nce. Interfaces for surgical robots that meet the needs for intuitive and natural interaction modes require an artificial intelligence system that can emulate human communication skills. However, although the precision of robotic systems has improved, the use of robots adds new tasks that are not natural to the surgeon, and for this reason, a human-machine interface capable of effi- cient and comfortable communication is required. The integration between the real and the virtual will consist of breaking down barriers with physical reality, to the benefit of the sensory and bodily dimensions. It is a question of designing more interaction with the environment, not only through the sense of sight but also through touch and hearing, to ensure the adaptability of machines to environmental variations. To achieve the surgery 5.0 paradigm, it will be necessary to configure flexible environments that with the help of intelligent devices can adapt to the needs and characteristics of the user to manage and analyze information, through a constant input-output exchange with the surrounding systems, improving the interaction and visualization of information.

REFERENCES

Aaltonen, I. E., Wahlström, M. (2018a). Envisioning robotic surgery: Surgeons’ needs and views on interacting with future technologies and interfaces. The International Journal of Medical Robotics and Computer Assisted Surgery, 14(6), e1941. Aaltonen, I., Salmi, T. and Marstio, I. (2018b). Refining levels of collaboration to sup-

port the design and evaluation of human-robot interaction in the manufacturing industry. Procedia CIRP 72, 93–98. Albani, JM., Lee, DI. (2007, March). Virtual reality-assisted robotic surgery simulation. J Endourol, 21(3):285-7. doi: 10.1089/end.2007.9978. PMID: 17444773. Boyraz, P., Dobrev, I., Fischer, G., and Popovic, M. B. (2019). Robotic Surgery.

Biomechatronics, 431–450. Cunningham, S., Chellali, A., Jaffre, I., Classe, J. and Cao, C. G. (2013). Effects of experience and workplace culture in human-robot team interaction in robotic surgery: a case study. International Journal of Social Robotics, 5(1), 75–88. Dibekci, A., Bebek, O. (2018, August). Improving the safety of medical robotic

systems. In 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob) (pp. 73–78). IEEE. Ficuciello, F., Tamburrini, G., Arezzo, A., Villani, L. and Siciliano, B. (2019). Autonomy in surgical robots and its meaningful human control. Paladyn, Journal of Behavioral Robotics, 10(1), 30–43. Formati F., Laudante E. and Buono M. (2021). Human-Centered-Design for Defi-

nition of New Collaborative Scenarios. In: Raposo D., Martins N., Brandão D. (eds) Advances in Human Dynamics for the Development of Contemporary Soci- eties. AHFE 2021. Lecture Notes in Networks and Systems, vol 277, pp. 78–85. Springer, Cham. https://doi.org/10.1007/978-3-030-80415-2_10. Gillan, D. J. (2020). Invited Essay: Usability Issues in Human-Robot Interaction.

Journal of Usability Studies, 15(4). Haidegger, T. (2019). Autonomy for surgical robots: Concepts and paradigms. IEEE Transactions on Medical Robotics and Bionics, 1(2), 65–76.

Multidimensional, Intuitive and Augmented Interaction Models for Robotic Surgery 137

Hubert, N., Gilles, M., Desbrosses, K., Meyer, J. P., Felblinger, J. and Hubert, J.

(2013). Ergonomic assessment of the surgeon’s physical workload during stan- dard and robotic assisted laparoscopic procedures. The International Journal of Medical Robotics and Computer Assisted Surgery, 9(2), 142–147. IEC 62061:2021 “Safety of machinery - Functional safety of safety-related control systems”. Inam, R., Raizer, K., Hata, A., Souza, R., Forsman, E., Cao, E., and Wang, S.

(2018, September). Risk assessment for human-robot collaboration in an automa- ted warehouse scenario. In 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA) (Vol. 1, pp. 743–751). IEEE. ISO 10218:2011 “Robots and robotic devices”. ISO 13849-1:2015 “Safety of machinery - Safety-related parts of control systems - Part 1: General principles for design”. ISO 8373:2012 “Robots and robotic devices – Vocabulary” ISO 8373:2021 “Robotics – Vocabulary” ISO/TS 15066:2016 “Robots and robotic devices - Collaborative robots”. Lee, G. I., Lee, M. R., Clanton, T., Sutton, E., Park, A. E. and Marohn, M. R.

(2014). Comparative assessment of physical and cognitive ergonomics associa- ted with robotic and traditional laparoscopic surgeries. Surgical endoscopy, 28(2), 456–465. Schreuder, H. W., Wolswijk, R., Zweemer, R. P., Schijven, M. P. and Verheijen, R.

H. (2012). Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG: An International Journal of Obstetrics & Gynaecology, 119(2), 137–149. Smids, J., Nyholm, S. and Berkers, N. (2019). Robots in the workplace: A threat to - or an opportunity for - meaningful work? Philosophy & Technology. https: //doi.org/10.1007/s13347-019-00377-4. Van der Meijden, O. A., & Schijven, M. P. (2009). The value of haptic feedback

in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surgical endoscopy, 23(6), 1180–1190. Young, J. E., Sung, J., Voida, A., Sharlin, E., Igarashi, T., Christensen, H. I. and Grinter, R. E. (2011). Evaluating human-robot interaction. International Journal of Social Robotics, 3(1), 53–67.

Cover of the proceedings volume Human Factors in Robots, Drones and Unmanned Systems

Explore the complete digital volume

Human Factors in Robots, Drones and Unmanned Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-33-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →