Alejandra Gomez De Cadiz, Ainhoa Rodríguez- De Luis, Iván Martín González, María Carmen Juan Lizandra, Cristina Herrera Ligero (2025). Mixed Reality as a tool for enhancing precision in surgery planning. In: Human Interaction and Emerging Technologies (IHIET-AI 2025): Artificial Intelligence and Future Applications. AHFE Open Access, vol. 161, pp. 438–448. https://doi.org/10.54941/ahfe1005936
Mixed reality (MR) is an emerging technology that combines features of augmented reality (AR) and virtual reality (VR) by overlaying virtual elements onto a natural environment. This fusion of the real with the digital allows users to interact naturally and intuitively with the various aspects, making MR a valuable tool for its application in different fields, including the clinical field. This work aims to present a working methodology for the application of Mixed Reality in different surgical specialities, showcasing scenarios generated for its use in the surgical fields of trauma and vascular surgery. By superimposing images and using 3D anatomical models over the surgeon's field of vision, the aim is to support the surgeon's movement guidance in complex procedures or areas that are difficult to visualise. To achieve this, the development process of the MR scenarios is detailed: firstly, the work of the medical image and extraction of the anatomical models using Materialise’s Mimics software, followed by the importation into the Unity engine for the design and positioning of virtual elements to be displayed, and finally the visualisation and design of the interactions with the digital environment through the use of different devices (tablets, smartphones, headsets). In addition to combining 3D anatomical models with information from the DICOM file of the medical image, the working methodology presented also details the work carried out for the positioning of guiding elements, such as vectors, angles, trajectories or other geometric elements that aid in guiding the surgeon’s movements, using devices that allow professionals to keep their hands free. All of this aims to show a possible use of Mixed Reality by offering greater immersion through anatomical models that faithfully represent the patient's anatomy and the ability to interact with them in real-time, making it technological support for the clinician's training for diagnosis and surgical planning, improving anatomical understanding in complex cases. The potential and application of MR in various surgical fields, especially in surgical planning, could significantly transform medical practice by allowing greater personalization of interventions, optimising precision for better clinical outcomes, and saving time in the operating room by increasing the efficiency and safety of the surgical procedure.
Mixed Reality as a Tool for Enhancing Precision in Surgery Planning 439
INTRODUCTION
The ability to explore the interior of the human body through medical imaging, coupled with the interoperability and structured metadata storage facilitated by the Digital Imaging and Communications in Medicine (DICOM) standard, has revolutionised medicine. These advances, driven by continuous progress in technology and computing, have transformed clinical practice by improving both diagnosis and surgical planning. Mixed reality (MR) is a technology that merges the natural world with digital elements, allowing both to interact in real time. Unlike augmented reality (AR), which overlays digital information onto the physical environment, or virtual reality (VR), which immerses the user entirely in a digital environment, MR more seamlessly integrates virtual objects into the real physical world. This enables users to simultaneously view and manipulate both tangible and virtual objects, utilizing digital elements that react to and respond to the conditions of the real environment.
For complex surgeries, surgical planning provides clinicians with detailed knowledge of the procedure and valuable information that helps to minimize risks in the operating room, enhancing both the safety and efficiency of the process. However, current medical training often relies on two-dimensional, static content, which is frequently characterized by a low level of realism. Furthermore, the use of consultation systems typically requires interaction with screens, keyboards or computers, which can be impractical, particularly when these systems are located outside the surgical area or even outside the operating room (Sánchez-Margallo et al., 2021).
MR demonstrates its potential by offering digitally integrated visual information directly within the clinician’s field of view, realistic data derived from the patient’s medical imaging. By making use of headsets, such as HoloLens2, surgeons can access this data intuitively and manipulate it with their own hands, without the need for external tools. This technology emerges as a valuable asset for optimizing the planning and execution of surgical procedures.
An aortic aneurysm is an abnormal dilation of the aorta, the body’s main artery, caused by weakening of the arterial wall. This condition can lead to life-threatening complications such as rupture or dissection if untreated. Endovascular treatment, known as Endovascular Aneurysm Repair (EVAR), involves inserting a stent-graft through a small incision in the groin. The device is guided to the aneurysm site using imaging and deployed to reinforce the weakened aortic wall, redirecting blood flow and reducing pressure on the aneurysm (England & Mc Williams, 2013). This minimally invasive approach offers shorter recovery times and lower risks compared to open surgery, making it a preferred option for many patients. The integration of MR in the planning of such procedures could enhance surgical planning by providing an improved anatomical understanding through the use of three-dimensional information.
Another surgical procedure of particular interest for the application of MR is spinal arthrodesis. Arthrodesis is a surgical procedure in which two bone segments are permanently fused, eliminating relative motion between
them. It is a widely used method for treating spinal injuries and is indicated in all degenerative or unstable spinal pathologies (De la Torre & Martínez- Quiñones, 1997). In posterior spinal arthrodesis surgery, one of the key aspects for the correct placement of screws is identifying the entry point and determining the direction of insertion (Comin Clavijo, 1995). MR can be highly useful for determining the position and orientation of the pedicle axis during surgical planning to subsequently overlay this trajectory onto the surgical field in question, assisting the surgeon in creating the entry hole for screw insertion through the pedicle.
This article aims to present a methodology for the generation of anatomical models that faithfully represent the patient’s anatomy using medical imaging, integrating these models into game engines to design scenarios that support clinicians in the surgical planning process of the previously mentioned procedures. Accordingly, the proposed methodology includes the design and development of two applications: one for the endovascular treatment of aortic aneurysms, and another for surgical planning of posterior vertebral arthrodesis and the placement of transpedicular screws.
MATERIALS AND METHODS
In this section, the resources, tools, and procedures used for the development of two mixed reality applications are described. These applications are designed to support clinicians in the planning of two different surgical procedures: EVAR and posterior vertebral arthrodesis. The methodological approach includes the generation of three-dimensional anatomical models, the use of specialized software for segmentation and modelling, the configuration of virtual environments in Unity, the implementation of interactive tools using the Mixed Reality Toolkit (MRTK) and the deployment into the headset. The materials utilized and the main stages of the process are detailed below.
Generation of the Digital Models To generate the three-dimensional anatomical model, anonymized DICOM files derived from computer tomography (CT) scans were utilized. Image processing and segmentation were performed using Mimics v17.0 software (Materialise). Masks were generated by applying a thresholding technique based on grayscale levels, which correspond to the density and cellular structure of human body organs. These masks underwent subsequent manual editing and refinement to accurately depict the anatomical structures targeted for intervention.
Aorta With Aneurysms Model The processed medical image is a contrast-enhanced CT scan corresponding to a case of multiple aortic aneurysms in a patient with a renal transplant. For aortic segmentation, a mask was created to include pixels within the grey- level range of [164, 2976] on the Hounsfield scale (HU). Postprocessing of the mesh was performed to create an anatomical model representing the aorta, from the aortic valve to the bifurcation of the femoral arteries. The model
Mixed Reality as a Tool for Enhancing Precision in Surgery Planning 441
also included relevant aortic branches of surgical interest: coronary arteries, brachiocephalic trunk, left common carotid artery, left subclavian artery, celiac trunk, mesenteric artery, renal arteries, and iliac arteries up to the femoral bifurcation. The postprocessing was conducted using Mimics tools, reviewing anatomical slices to clean artifacts, remove irrelevant structures, fill potential gaps, and adjust the length of the branches. The selected threshold represented the contrast agent in the medical image, so that the model depicted the flow within the aorta. Morphological operations were applied to the postprocessed mask, including a 2-pixel dilation to represent the aorta’s wall. Using the “Calculate Part” tool, the 3D object was generated from the previously created mask, resulting in a three-dimensional representation of the targeted anatomy, hollowed out while maintaining a constant wall accurate aortic model with a realistic wall thickness, ensuring the fidelity of the patient’s anatomy (Figure 1A).
Figure 1: Anatomical models segmented and postprocessed in Mimics: A) Aorta B) Spine.
Spinal Model The DICOM file used for generating the spinal column model corresponds to a non-pathological subject. The CT scan comprises 1.377 axial slices. For segmentation, the region of interest (ROI) was adjusted to the lumbar spine area, and Mimics’ predefined threshold for Bone [226, 2802] was applied. Using Mimics’ mask editing tools, other anatomical structures with similar grey density values were removed, artifacts were eliminated, and only the L2, L3, L4, L5 vertebrae and the sacrum were preserved in the segmentation process (Figure 1B). To enable 3D printing of the segmented model as a single piece, the vertebrae were connected through the articulation between the inferior articular processes of each vertebral body and the superior articular processes of the adjacent vertebral body. Additionally, the spinous processes of the L3, L4, and L5 vertebrae were linked to ensure structural integrity.
Both models were then exported in STL format, extracting the polygonal mesh that represent the surface of the structure. Due to the complexity
and heterogeneity of anatomical surfaces, a high number of triangles was required to accurately capture and detail the anatomical features of interest. However, this complexity sometimes resulted in mesh errors. To address these issues, mesh correction was performed using Meshmixer, ensuring a clean and anatomically accurate 3D models. Since Unity does not recognize STL files, both models were imported to Blender and exported in FBX format to allow compatibility with Unity’s asset pipeline.
Project Configuration in Unity To create the MR environment, the Unity game engine was used, specifically by generating a “3D Core” project. Within this project, the Mixed Reality Toolkit (MRTK 2.8.3) package was integrated, providing a set of tools and components designed to facilitate the development of interactions for MR projects, particularly those intended for use with HoloLens 2. The necessary packages for developing the MR application were loaded into the project, and interaction settings were configured following the guidelines of (Larsen, 2023). This configuration enabled the input and controller profiles to support hand interactions with the HoloLens 2. MR environments were developed for surgical planning in two specific cases: one involving endovascular surgery and another focused on the placement of transpedicular screws in spinal surgery.
Endovascular Surgery Once the project was configured and the necessary packages were added, the anatomical model was imported in FBX format. The scale, orientation, and position of the model were adjusted to ensure the digital element remained within the field of view of the HoloLens 2, represented in Unity by the “Main Camera” object.
To enable manipulation of the anatomical model of the aorta with hand gestures, allowing for rotation, translation, and scaling, the ObjectManipulator component from MRTK was added to the parent object. Additionally, a BoxCollider was included and adjusted to define the physical boundaries of the object and facilitate interaction detection. While a MeshCollider could better fit the irregular shape of the aorta, a BoxCollider was chosen due to its lower computational cost, which optimizes application performance without compromising functionality.
Once the model was integrated into Unity, the AI-powered texture generation capability of Polycam Pro was utilized to enhance the realism of the aorta model. Using the text prompt “Realistic smooth aortic tissue” with an image strength of 0.95, a smooth texture with variations in pinkish tones was generated and exported in JPG format. This texture was imported into Unity, where a new material was created and assigned to the child object of the aorta model (Figures 2A and 2B). The designed texture was applied to the Albedo channel of the material, and the “metallic” and “smoothness” properties were adjusted to achieve a more realistic appearance, ensuring that the texture was applied uniformly and without visible distortions. An essential aspect for the accurate visualization of the structure in the
Mixed Reality as a Tool for Enhancing Precision in Surgery Planning 443
MR experience is lighting. Multiple directional light sources were created to ensure the aorta was well-lit from various perspectives, and different parameters were adjusted to enhance visualization while minimizing the dependency on ambient lighting conditions when using the HoloLens 2.
Figure 2: Aorta’s unity configuration: A and B) Texture; C) Lighting configuration.
Spinal Surgery For this MR scenario, tracking was incorporated to position the virtual element, the transpedicular screw, based on the exact position of a real object — in this case, the 3D-printed anatomical model of the lumbar vertebrae L2 to L5. To achieve this, the segmented spine was printed using Selective Laser Sintering (SLS) technology. Once the model was printed, it was scanned using the Polycam mobile application, allowing for the generation of a 3D digital model while preserving information about texture, colour, and patterns, which enhances visual information and facilitates the recognition and tracking of the printed spine. Using Vuforia’s Model Target Generator software, the OBJ file of the scanned and 3D-printed spinal model was processed, and the Model Up vector was selected, defining the vertical orientation of the object for recognition. Various settings were configured to enhance the recognition of the spine model by leveraging the texture and colour information captured during the scan, ensuring that the application could identify the printed spine from any position and angle. Subsequently, the software uses artificial intelligence (AI) to generate multiple views of the model from different angles and perspectives, combined with data such as lighting, scale, and possible occlusions, to improve the robustness of recognition in real-world scenarios. This AI training enhances the system’s ability to recognize the object from complex angles and under challenging conditions. Once the Model Target has been trained, it is saved in a database that can be imported into Unity, containing all the necessary information for the Vuforia-powered application to detect and track the object in real time.
For the development of the application, in addition to the previously mentioned packages, the Vuforia Engine package was added, which is compatible with multiple platforms, including iOS, Android, and HoloLens 2. To begin designing the application, the Model Target of the spine was imported into the Unity Hierarchy scene, its dimensions were adjusted, and the Runtime Occlusion Mesh option was enabled. This ensures that 3D objects are hidden when they come into contact with the mesh defining the spinal model, as will be the case with the virtual transpedicular screws. In posterior vertebral arthrodesis surgery, one of the key aspects for the correct placement of screws is the identification of the entry point. For the design of the application, the clinical criterion of the freehand technique for the placement of transpedicular screws was followed (Bauer et al., 2024). According to this technique, the screw entry point is defined as the intersection point between the sagittal articular plane and the axis along the transverse processes. The hole for the screw is drilled 2–3 mm lateral to the intersection of these lines (Figure 3), and the drilling is performed in an anteromedial direction with a specific angulation relative to the sagittal plane for each vertebra and pedicle (Panjabi et al., 1992). This ensures that the vector is positioned at the screw’s entry point with the recommended angulation.
Figure 3: Drill hole planification. Adapted from (Bauer et al., 2024).
The trajectory for the insertion of the transpedicular screw into the pedicle has been defined using the following anatomical planes created in Unity as references: the sagittal plane, established based on specific anatomical landmarks such as the spinous process of the first vertebra in the model (L2), the lower part of the vertebral body of the last vertebra in the model (L5), and the end of the coccyx; the transverse plane of the vertebrae, parallel to the superior endplate of each vertebra and positioned passing through the pedicles of each vertebra; and the sagittal articular plane, a predominantly sagittal plane formed by the articular facets of the lumbar spine. Using these reference planes, a cylinder is created to act as an axis, positioned parallel to the articular plane while contained within the transverse plane and passing through the entry point or insertion point in the pedicle. In this way, the cylinder is placed to simulate the trajectory vector for screw insertion into the pedicle. This configuration is implemented using C# scripts, which are
Mixed Reality as a Tool for Enhancing Precision in Surgery Planning 445
added as components to the various objects in the Unity hierarchy, with appropriate adjustments for shape, colour, texture, and position. As a result, a vector is obtained in the left and right pedicles of each vertebra, indicating the trajectory for the transpedicular screw insertion to ensure correct positioning. Additionally, a transpedicular screw was designed and implemented in the Unity scene to simulate its insertion into the pedicle. The screw consists of several components, including the screw shaft and the tulip. The screw was aligned with the trajectory vector, ensuring it follows the same orientation and direction, enabling a realistic simulation of its insertion into the pedicle.
To allow the clinician’s criterion to prevail, various buttons were designed and configured to adjust the insertion vector’s angulation relative to the sagittal plane. For this purpose, the MRTK package and the Canvas component were used, enabling the creation and manipulation of user interface elements. These elements are made visible during application use and handle interactions, such as button clicks, performed with the fingers.
Two buttons, “+1
” and “–1
”, were configured to rotate the vector clockwise and counterclockwise, respectively, when pressed. Additionally, a numerical counter text was added to the Canvas and displayed in the user interface alongside the buttons. This counter shows the vector’s rotation angle on the transverse plane, allowing the user to see its initial angulation and any changes applied. This functionality is achieved through a script integrated into the text element, which contains two functions to update the counter’s value when either button is clicked. Two additional buttons, “Insert” and “Remove,” were implemented to control the screw, simulating its insertion and removal from the vertebra’s pedicle. Finally, two more buttons, “Show” and “Hide,” were created to dynamically change the screw’s material, making it appear or disappear (Figure 4).
Figure 4: Unity configuration for spine surgery MR planification.
Deployment to HoloLens 2 After the application was developed in Unity, it was deployed to the HoloLens2 for real-world testing. This step involved building the project into a UWP (Universal Windows Platform) app and deploying it directly to the HoloLens 2 through Visual Studio and through the Device Portal, facilitating
the installation and execution process on the device to ensure its proper functionality in the testing environment. The application was launched on the HoloLens 2 to ensure accurate rendering and interaction of 3D models and real-time tracking.
RESULTS
As a result of the methodology described in the previous section, the first application was developed and tested to assist in the planning of an endovascular procedure through interaction with a model representing the target anatomy. This application enables the visualization of pathological anatomy from multiple angles and perspectives, including its branches and even the internal structure of the aorta (Figure 5), providing a deeper anatomical understanding and enhancing the surgical procedure’s planning process. The interactions developed using Unity proved to be intuitive and user-friendly, relying on commonly used gestures for similar tasks. Notably, the hand-tracking performance and the stability of the holographic model, as well as the interaction with it, were outstanding, allowing for a smooth and realistic MR experience. However, it is worth mentioning that the work invested in achieving a more realistic aortic texture is not fully reflected in the holographic visualization. This limitation arises because the colour representation is not faithfully maintained due to the HoloLens 2’s high dependency on ambient lighting.
Figure 5: Screenshots using arthrodesis MR planning scenario with HoloLens 2.Figure 6: Screenshots using EVAR MR planning scenario with HoloLens 2.
Mixed Reality as a Tool for Enhancing Precision in Surgery Planning 447
Regarding the second application, developed to support and guide surgeons in the planning and placement of transpedicular screws in the lumbar spine, the application was successfully deployed and initialized on the HoloLens 2. Upon viewing the printed model through the HoloLens 2, the virtual objects designed within the application were projected onto the 3D-printed column within seconds. This rapid projection confirms that the application effectively recognizes the printed column as the predefined Model Target, validating the adequacy of the configuration parameters selected during the creation of the Model Target. The holographic model was observed from various angles by moving around the printed column and repositioning it to evaluate the projection from different perspectives. Notably, the parts of the virtual objects in contact with the printed model disappeared seamlessly, demonstrating the functionality of the “Runtime Occlusion Mesh” feature activated in Unity. Additionally, the interactive buttons displayed correctly within the projection (Figure 6), confirming their visibility and usability within the MR environment.
CONCLUSION
This work demonstrates the potential of MR applications as a support tool for clinicians in surgical procedures such as endovascular aneurysm repair and posterior vertebral arthrodesis. Through the integration of holographic anatomical models, advanced tracking techniques, and interactive environments, these applications enable clinicians to intuitively visualize and manipulate complex anatomical structures faithfully extracted from medical imaging. This enhances anatomical understanding and optimizes pre-surgical planning. The presented work highlights the ability to plan the holographic projection of features such as vectors, planes, or measurements to guide the surgeon’s movements during the procedure and accurately position these elements over anatomical structures. While limitations such as precision, physical model tracking, and the need for controlled lighting environments to optimize the visual quality of holograms were identified, MR represents a substantial step forward toward more precise, intuitive, and personalized surgical planning. In the future, it could transform the way surgeons plan and execute surgical procedures, ultimately optimizing outcomes and reducing operative risks.
ACKNOWLEDGMENT
Project (IMDEEA/2023/51) funded by the 2023 program of grants from the Instituto Valenciano de Competitividad Empresarial (IVACE), financed by the European Union. Project (IMDEEA/2024/12), supported by the Conselleria d’Innovació, Indústria, Comerç i Turisme de la Generalitat Valenciana, through IVACE, and co-financed by the European Union via the FEDER Program of the Comunitat Valenciana 2021-2027.
REFERENCES
Bauer, R., Kerschbaumer, F., & Poisel, S. (2004). Cirugía ortopédica—Columna (Vol. 4). Ed. Marbán.
Comin Clavijo, M. (1995). Biomecánica del raquis y sistemas de reparación. Ed.
Instituto de Biomecánica de Valencia. De la Torre, M., & Martínez-Quiñones, J. V. (1997). Artrodesis lumbar posterior. A propósito de 89 casos. Neurocirugía, 8(3), 211–220. https://doi.org/10.1016/S1130-1473(97)71032-7 England, A., & Mc Williams, R. (2013). Endovascular Aortic Aneurysm Repair (EVAR). The Ulster Medical Journal, 82(1), 3–10. Larsen, L. (2023). XR Step-by-Step 2023! Hololens 2: Setting up your Project in Unity 2022 + MRTK 2.8.3 + Visual Studio 2022. Lance Larsen - Microsoft MVP - Hololens / VR / AR. Retrieved from “http://www.lancelarsen.com/xr-step-by-step-2023-hololens-2-setting-up-your- project-in-unity-2022-mrtk-2-8-3-visual-studio-2022/” on November 20, 2024. Panjabi, M. M., Goel, V., Oxland, T., Takata, K., Duranceau, J., Krag, M., &
Price, M. (1992). Human Lumbar Vertebrae: Quantitative Three-Dimensional Anatomy. Spine, 17(3), 299–306. https://doi.org/10.1097/00007632-199203000- 00010 Sánchez-Margallo, J. A., Plaza de Miguel, C., Fernández Anzules, R. A., & Sánchez- Margallo, F. M. (2021). Application of Mixed Reality in Medical Training and Surgical Planning Focused on Minimally Invasive Surgery. Frontiers in Virtual Reality, 2:69264. https://doi.org/10.3389/frvir.2021.692641
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET-AI 2025): Artificial Intelligence and Future Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-37-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional