Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A Participatory Design based Human-Centric Mixed Reality (MR) Simulator for Neonatal Needle Thoracentesis Procedures

1 Department of Industrial and Enterprise Systems Engineering, UIUC, USA
2 Siebel School of Computing and Data Science, UIUC, Champaign, IL 61820, USA ↗
3 University of Illinois College of Medicine, Peoria, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1006968

Abstract

Neonatal needle thoracentesis (NNTP) is a delicate, life-saving procedure performed to remove excess fluid in the pleural space in newborns. Opportunities for hands-on training are limited due to the rarity of such cases and underfunded pediatric infrastructure. To address these challenges, we developed a guided mixed reality (MR) simulator that allows medical trainees to repeatedly and safely practice NNTP in a realistic, immersive environment. By leveraging our Unity application for the Meta Quest 3 and HoloLens 2 headsets, our simulator employs passthrough technology to overlay holographic instruction onto real-world mannequins while using hand tracking to capture precise finger movements required for needle insertion. The application follows a “learn, train, test” learning model inspired by established surgical pedagogy, providing immediate feedback on procedural accuracy. Using a participatory design approach, clinical experts contributed verified procedural data and iteratively reviewed simulation accuracy to ensure medical validity. Future evaluations will compare the MR simulator’s educational effectiveness against traditional mannequin-based training, focusing on accuracy, retention, and user experience. This work demonstrates the potential of mixed reality to bridge the gap between medical theory and procedural practice, offering a scalable, cost-effective solution for neonatal care training.

INTRODUCTION

The neonatal needle thoracentesis procedure (NNTP) is a critical intervention for treating effusions within the pleural cavity. When pathological fluid accumulates in the pleural space, thoracentesis is performed to relieve symptoms and obtain diagnostic samples. Successful completion requires accurate identification of the second or third intercostal space along the midclavicular line and insertion of a needle into the pleural space until

© 2025. Published by AHFE Open Access. All rights reserved. 1577

fluid is aspirated (Wiederhold, Sharma, & O’Rourke). The aspirated fluid is then released by turning the stopcock. Studies have shown that many emergency physicians fail to correctly identify the appropriate intercostal space on human volunteers (Ferrie, Collum, & McGovern, 2005). Accurate identification of surface anatomical landmarks is crucial, as inexperience or inaccuracy during NNTP can lead to complications such as pneumothorax, in which air collects outside the lungs due to a chest wall puncture (Cormier et al., 2018).

Figure 1 and 2: Intercostal space accessed for fluid extraction.

In both adult and neonatal thoracentesis, precision is essential to avoid injury to surrounding thoracic structures. Neonatal procedures are especially delicate due to the small size and fragility of infants. Procedural knowledge alone is insufficient to maintain proficiency, as infrequent clinical exposure contributes to skill deterioration (Patel, Posencheg, & Ades, 2012). Unfortunately, invasive neonatal procedures occur too rarely to allow consistent hands-on practice.

Globally, inadequate neonatal care continues to contribute significantly to child mortality. Nearly 40% of deaths among children under five are attributed to deficiencies in perinatal and neonatal care (Mangiaterra, 2006). Despite increasing rates of facility-based deliveries, the quality of neonatal care remains poor in many low-resource settings (Kaur et al., 2023), compounded by chronic underfunding in pediatric institutions (Velagala & Gupta, 2025). These challenges underscore the urgent need for accessible, scalable training interventions that enhance procedural competency and improve neonatal outcomes, particularly in under-resourced environments.

PREVIOUS WORK Such limitations within neonatal care have prompted the integration of emerging technologies with medical education and practice. With the recent advancements in wearable technology, such as the Meta Quest 3 and the Raybans Meta, surgical experiences for practitioners are continually being improved through extended reality (XR) applications. There is already extensive evidence showing a positive correlation between VR assisted education and neonatal curricula (Jayawardena et al., 2024). Research shows

A Participatory Design Based Human-Centric Mixed Reality (MR) Simulator 1579

that participants felt more engaged with an educational tool when they conducted training of the Neonatal Resuscitation Program through the HTC Vive Pro HMD (Aydin et al., 2025).

Extended reality (XR) presents a cost-effective and scalable way to teach medical students on niche procedures (Pottle, 2019), as the affordances of XR allow for continuous and reproducible training (Gupta et al., 2018, 2022, Cecil et al., 2018). Through robust virtual environments, students can practice without time or spatial constraints until they perfect their skills within the simulation. Augmented reality (AR) and mixed reality (MR) platforms specifically using the Microsoft HoloLens device are effective when applied in clinical settings due to their ability to align with the trainee’s visual-motor axis in the real world. These platforms overlay digital content onto physical environments, allowing for real mannequins, surgical tools, and physical surfaces to be integrated into the user’s virtual training space. Instead of using handheld controllers, this blended interaction tracks hand movements, redirecting the primary interactor to be the user’s hands. This enhances spatial awareness and contextual learning by maintaining the user’s connection to the real world. As a result, mixed reality is a significant evolution in simulation-based medical education, blending low-cost virtual visualizations with physical interaction in a way that traditional VR systems cannot replicate.

In this paper, we describe the design and development of our MR application for simulating neonatal needle thoracentesis. Using a participatory design approach, we gather clinical requirements and iteratively integrate feedback from medical experts.

DESIGN APPROACH Grounded from a participatory design perspective, the prototype was developed in collaboration between involved medical experts and the development team. Information about the procedure was provided by neonatology experts Dr. Nicole Rau and Dr. M. Jawad Javed from the University of Illinois Chicago to align the simulation’s objectives with the standards of a real-world medical environment. Once the concept was established, the experts provided the necessary tools for the procedure, which were replicated virtually as 3D models by the development team. Expert Dr. Nicole Rau was motion-captured as the simulation’s holographic instructor to demonstrate NNTP on a mannequin infant. Through interdisciplinary effort, an animated, life-sized avatar was created as the core presentation system to mimic a real training session with an experienced mentor. Following this groundwork, the prototype was subject to iterative improvement through bi-weekly feedback rounds.

During the prototype’s iteration, the referenced experts dictated a ‘ground truth model’ of the medical procedure from which the simulation engineers attempted to best replicate. As acknowledged in previous works, accurate identification of the intercostal space from surface anatomy is difficult for many practitioners, with a considerable number unable to reliably mark the correct location. This presents a critical bottleneck in NNTP, where precise

needle insertions are required to ensure the patient’s safety. In response to this, the simulation engineers collaborated with the experts to ensure the simulation’s millimeter fault-tolerance and avoid the propagation of future errors. The experts also redirected attention to important procedural steps, such as sterilization, that are already assumed in the virtualized environment. Integrating medical experts as design engineers helped establish the application’s accuracy, reliability, and relevance as an effective teaching tool of NNTP.

DEVELOPMENT APPROACH Building from the conceptual groundwork, the development phase focused on translating the procedural description into a virtual training software. The goal of this virtual training software is to recreate an accurate and precise simulation of this surgical environment, where students have the opportunity to repeat the practice as many times as needed. The environment should uphold the following requirements: (a) the application should be precise on a millimeter scale, (b) the user should be able to interact with both the virtual and physical environment reliably, and (c) the software feedback system should be accurate, where mistakes are immediately penalized and correctness is not falsely flagged.

The prototype was developed using the Unity Engine for release on both the Meta Quest 3 and the Microsoft HoloLens 2. We utilize the MRTK2 toolkit for XR interaction. Both platforms support overlaying virtual animations on the user’s real-world environment; for our procedure, this is expected to be a real medical setting with a mannequin. The HoloLens 2 was considered because of its inherent MR capability, as the HMD projects holograms through a see-through display. In contrast, the Meta Quest is fully-immersive and vision-obscuring, meaning that built MR applications must rely on the Meta Quest’s reconstructed camera feed on the user’s surrounding environment through the Meta Quest Passthrough API. The Meta Quest 3 was specifically chosen for its full-color passthrough and its clear visual fidelity, two fields where it outcompetes the Quest 2 and the Quest Pro respectively. In the simulation, we want the users to practice the procedures with their hands. As NNTP is a delicate surgical procedure, the application encourages exact finger movement from the user and therefore demands robust hand tracking.

Conclusively, our prototype is a Unity application that runs an overlaid simulation on Meta Quest 3 in passthrough mode, with the ‘controllers’ being the user’s tracked hands.

SYSTEM DESIGN The user experiences three levels of involvement within the simulation, each of which is outlined by a Step System model. The simulation’s three stages are inspired by the “learn, see, practice, prove, do, maintain” teaching paradigm from Sawyer et al. In reflection of this paradigm, our simulation is governed by a “Learn, Train, Test” model. Each succeeding task reduces the amount

A Participatory Design Based Human-Centric Mixed Reality (MR) Simulator 1581

of scaffolding supporting the user. Currently, the simulation assumes that the user is familiar with NNTP and requires a review of the steps.

Figure 3: Iterative development of a mixed reality simulator for NNTP.
Original PDF Figure 3:
Figure 4: User view of the NNTP simulation through the HoloLens 2.
Original PDF Figure 4:

The first step “Learn” allows the user to observe from a bystander perspective (Gohman et al., 2024). The virtual avatar will overlay her performance of the procedure on the user’s physical mannequin. The Step System allows the user control over the timeline of a task’s animation, allowing a user to repeat a step, pause a step on certain frames, and move throughout the procedure’s sequence. Through this framework, self-pacing and self-exploration of the practice are encouraged. After observing the procedure from a third-person perspective, the “Train” phase places the user in the position of the practitioner, from which the user will see the virtual instructor’s hands perform the same steps on the overlaid mannequin. The “Train” phase repurposes the user’s role in the simulation from a simple observer to a studying practitioner of NNTP, requiring manual dexterity and attention to detail that is not demanded from the “Learn” phase. Once the user is familiar with the sequence of steps, they will be evaluated in the “Test” phase. A user’s evaluation is based on their correctness in the ordering, completeness, and precision of the performance. Feedback from the simulation is necessary to ensure that the user is accurately trained.

To contribute to virtual reality research, the project’s experimental setup should fulfill two criteria. First, it should compare the effects of natural human touch over conventional controller haptic feedback on memory and learning. Second, our method should highlight the advantages mixed reality

has in scalable learning management and medical simulation reproducibility. For our experiment, our samples will be professionals trained in the NNTP and medical students. Their feedback will be separated to account for differing viewpoints. From trained surgeons, we will collect feedback on our simulation’s correctness and adherence to medical board training. From students, we will collect feedback on the simulation’s perceived effectiveness in teaching the procedure. The experience evaluation will use a questionnaire with comparative multiple choice questions and free-write space for additional comments. The experiment will be designed to be about 2 hours long per person. With these experimental results, the prototype may be iteratively improved upon to align with user expectations.

CONCLUSION

We are currently designing the feedback system and experimental setup for an extended reality (XR) approach to medical education. Testing from a human factors perspective is essential to ensure the application’s effectiveness, as mixed reality (MR) environments inherently present challenges related to human-centered design due to their close interaction with sensory perception. During testing, participants may experience “cybersickness,” with symptoms such as nausea or vertigo. Additionally, achieving comfort and optimal visual alignment when donning a head-mounted display (HMD) can be challenging, particularly for users who wear glasses—an important consideration for accessibility and inclusivity.

From a hardware perspective, we hypothesize that the Meta Quest 3 offers the most suitable platform for MR applications in our context. Historically, medical professionals have expressed greater trust in the HoloLens 2 framework, as its holographic overlay—rather than full immersion—allows users to rely on direct vision instead of a reconstructed camera feed (Palumbo, 2022). However, recent advancements in display fidelity and passthrough technology have significantly improved the reliability of immersive systems. Beyond visual clarity, maintaining a consistent frame rate is essential, as even slight fluctuations can induce motion sickness or visual discomfort. A comparative analysis between the HoloLens 2 and the Meta Quest 3, using the same software as a control variable, will help determine whether immersive HMDs have reached a level of performance and trust appropriate for clinical education. Currently, we are in preparation of conducing exhaustive studies to ensure the validation and efficacy of the proposed MR environments.

Our guided mixed reality simulator for neonatal needle thoracentesis offers a scalable, high-fidelity solution for training in rare and high-risk medical procedures. Through continuous visual feedback, real-time hand tracking, and expert-informed design, the system enhances technical proficiency and learner confidence. Future work will focus on refining user feedback mechanisms and conducting comparative studies to evaluate educational outcomes. Furthermore, integrating these modules into a virtual Learning Management System (LMS) could create a scalable library of medical training resources. Ultimately, this work demonstrates how mixed reality

A Participatory Design Based Human-Centric Mixed Reality (MR) Simulator 1583

can transform neonatal and broader medical education by bridging the gap between simulation-based learning and real-world clinical practice.

ACKNOWLEDGMENT

This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center.

REFERENCES

Aydin et al. (2025) VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program. Available at: https://pmc.ncbi.nlm .nih.gov/articles/PMC11915296/. Cecil, J., Gupta, A. and Pirela-Cruz, M., 2018, April. Design of VR based orthopedic

simulation environments using emerging technologies. In 2018 Annual IEEE International Systems Conference (SysCon) (pp. 1–7). IEEE. Cormier, Chan, Manen & Yaskina (2018) Exploring paediatric residents’ perceptions of competency in neonatal intensive care. Available at: https://pmc.ncbi.nlm.nih.g ov/articles/PMC6376347/. Ferrie, Collum & McGovern (2005) The right place in the right space? Awareness

of site for needle thoracocentesis. Available at: https://pmc.ncbi.nlm.nih.gov/artic les/instance/1726582/pdf/v022p00788.pdf. Gohman, Nisar, Gupta, Javed & Rau (2024) Development and usability of a virtual reality umbilical venous catheter placement simulator. Available at: International Journal of Computer Assisted Radiology and Surgery, pp. 1–9. Springer International Publishing. Gupta, A., Cecil, J., Pirela-Cruz, M. and Ilidan, N., 2018, October. Design of an

immersive simulator for orthopedic surgical training. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 3813–3818). IEEE. Gupta, A., Cecil, J., Pirela-Cruz, M., Shamsuddin, R., Kennison, S. and Crick, C., 2022, April. An Investigation on the role of affordance in the design of extended reality based environments for surgical training. In 2022 IEEE International Systems Conference (SysCon) (pp. 1–7). IEEE. Jayawardena et al. (2024) Relationship of adaptive expertise of health professions

educators with age, experience, academic rank, and their work performance during an altered academic environment. Available at: https://bmcmededuc.bio medcentral.com/articles/10.1186/s\hbox{12909--024}-05838. Kaur et al. (2023) Persistent barriers to achieving quality neonatal care in lowresource settings: perspectives from a unique panel of frontline neonatal health experts. Available at: https://www.joghr.org/article/72089-persistent-barriers-to -achieving-quality-neonatal-care-in-low-resource-settings-perspectives-from-a- unique-panel-of-frontline-neonatal-health-experts. Mangiaterra (2006) Why and how to invest in neonatal health. Available at: https:

//pubmed.ncbi.nlm.nih.gov/16414318/. Palumbo (2022) Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects. Sensors (Basel). 22(20):7709. Available at: https: //pmc.ncbi.nlm.nih.gov/articles/PMC9611914/. Patel, Posencheg & Ades (2012) Proficiency and Retention of Neonatal Resuscitation

Skills by Pediatric Residents. Available at: https://publications.aap.org/pediatrics/ article-abstract/130/3/515/30287/Proficiency-and-Retention-of-Neonatal?redire ctedFrom=fulltext. Pottle (2019) Virtual reality and the transformation of medical education.

Velagala & Gupta (2025) Navigating Gaps Between Pediatric Training and Neonatal Care in Community Practice. Available at: https://publications.aap.org/neorevi ews/article-abstract/26/9/e595/203281/Navigating-Gaps-Between-Pediatric-Trai ning-and?redirectedFrom=fulltext. Wiederhold BD, Sharma S & O’Rourke MC (2024) Thoracentesis. StatPearls.

Available at: https://www.ncbi.nlm.nih.gov/books/NBK441866/.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-75-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →