Mahdi Ebnali, Andrew Goldsmith, Barbara Burian, Byonne Atamna, Nicole Duggan, Chanel Fischetti, Steven Yule, Roger Dias (2022). AR-Coach: Using Augmented Reality (AR) for Real-Time Clinical Guidance During Medical Emergencies on Deep Space Exploration Missions. In: Healthcare and Medical Devices. AHFE Open Access, vol. 51, pp. 67–75. https://doi.org/10.54941/ahfe1002100
1 STRATUS Center for Medical Simulation, Brigham and Women’s Hospital Boston, MA, USA; Department of Emergency Medicine, Brigham and Women’s Hospital Harvard Medical School, Boston, MA, USA ↗↗↗↗↗
2 Department of Emergency Medicine, Brigham and Women’s Hospital Harvard Medical School, Boston, MA, USA ↗↗↗↗
3 Human Systems Integration Division, NASA Ames Research Center, USA ↗
4 San Jose State University-Research Foundation, USA ↗
5 Department of Clinical Surgery, University of Edinburgh, Scotland ↗
Space travel imposes significant risks to crew health due to physiological adaptations, exposure to physical and psychological stressors, and limited capabilities to provide medical care. When medical emergencies occur, appropriate use of diagnostic and procedural guidance tools are crucial countermeasures against the risks of injury and mission failure. Point-of-care ultrasound (POCUS) is the only portable imaging modality available during exploration missions that can provide critical and dynamic medical information. Developing competency in POCUS is time-consuming and it is usually achieved through years of medical residency or clinical fellowship training programs. Due to the amount of astronaut training currently required, it is not feasible to also provide them with in-depth POCUS training. Current cognitive aids for POCUS-based procedures are either paper-based or static electronic checklists, which can be cumbersome to use, non-intuitive, and sometimes distracting; applying their written guidance to real actions can be difficult. To overcome these limitations, we developed a proof of concept of an augmented reality (AR) Coach (AR-Coach) as an Augmented Clinical Tool (ACT): a hands-free virtual coach system that guides the crew in real-time on how to perform POCUS during medical emergencies in space. To better understand the context and design requirements for the proposed ACT, we applied a human-centered design approach as part of our wider space medicine research program. We convened a multidisciplinary expert panel (n=46), including astronauts, flight surgeons, clinicians, XR and AI experts, to identify essential capability requirements. Task analysis with five experts, including clinicians, human factors researchers, and an XR developer, was used to create a process model of a POCUS-guided procedure to diagnose a potentially life-threatening condition (i.e., pneumothorax) that could occur during space travel. An iterative design and prototyping process was conducted. Informed by the expert panel and task analysis, we created a proof of concept of the AR-Coach which includes holographic panels that guide the crew in confirming the diagnosis of pneumothorax using POCUS. The results of this study can be applied in advancing space technologies that support astronauts in managing medical events during space exploration missions, optimizing performance, and improving crew safety.
In recent years, the National Aeronautics and Space Administration (NASA), European Space Agency (ESA), and other governmental agencies along with private companies such as SpaceX, Virgin Galactic, and Blue Origin have expanded their efforts in promoting and optimizing manned missions beyond low Earth orbit (Institute of Medicine et al., 2001). These missions aim to take astronauts and other space travelers beyond the Earth’s orbit to aste- roids, the Moon, and Mars. Considering the rapid growth in commercial launches and the goal of the Artemis program to return humans to the Moon by 2025, exploration missions into deep space are likely to occur in the near future (Robertson et al., 2020).
Accumulated data have demonstrated that spaceflight has detrimental effe- cts on human health, imposing significant and unique risks to crew members due to physiological adaptations, exposure to physical and psychological stressors, and limited capabilities to provide medical care (Norsk, 2020). During deep space missions, such as a trip to Mars which may take up to 34 months, the risks and challenges to the crew astronauts’ health will be tremendous. Therefore, critical measures and mitigation strategies need to be in place in order to ensure crew health and mission success (Bartone et al., 2019). When a medical event occurs in space, appropriate use of diagnostic and procedural guidance tools are crucial countermeasures to mitigate against the risks of injury and mission failure. Currently, ultra- sound is the only medical imaging modality readily available during space missions (Martin et al., 2003). Point-of-care ultrasound (POCUS) provides critical information for medical diagnosis and procedural guidance in real- time. Previous studies have demonstrated the utility and efficacy of POCUS as a portable and versatile diagnostic imaging modality under austere condi- tions during space missions (Jadvar, 2000). Availability of POCUS in a highly compact form enables its use in space travel, where weight and size restricti- ons are strict (Jones et al. 2009). However, POCUS accuracy and effectiveness are largely operator-dependent compared to other imaging modalities, often demanding robust and intensive training prior to use (LoPresti et al., 2019). This is usually achieved through years of medical residency or dedicated clinical fellowship training (Good et al., 2021).
Due to the vast amount of training that astronauts are currently requi- red to complete, it is not feasible to receive additional in-depth medical training. International Space Station (ISS) crew members, for instance, cur- rently receive only 3-hours of training on POCUS, including hardware and software familiarization, ultrasound basics, and specific instruction for any science activities planned for their flight (Sargsyan et al., 2005). These factors limit the ability of space crew members to effectively perform POCUS autonomously without additional support.
Currently, astronauts have access to paper or static electronic cue cards and cognitive aids to support their completion of POCUS-based and other medi- cal procedures. For example, those for POCUS-based procedures provide information on equipment controls, probe placement, and probe manipu- lation. However, these aids can be cumbersome to use, non-intuitive, and are
AR-Coach: Using Augmented Reality (AR) for Real-Time Clinical Guidance 69
Figure 1: Research and design cycle used in creating the AR Coach.
sometimes distracting (Seagull et al., 2007). Further, in a time-sensitive cli- nical scenario, these card-based medical algorithms are not often sufficient to improve clinical outcomes. To address these limitations, remote guida- nce has been investigated as an option for POCUS imaging in low-orbit space missions (Sargsyan et al., 2005). After communication has been establi- shed, a POCUS expert at the mission control center (MCC) guides the crew through real-time or near real-time video transmission of ultrasound images and video, allowing the crew to be guided on probe placement, anatomical landmark identification, and image interpretation. Previous studies suggest that limited training combined with remote guidance by a POCUS expert using a system of visual and verbal cues can effectively facilitate complex medical imaging during space missions (Martin et al., 2012).
While promising, the feasibility of remote guidance by a ground-based expert is limited, particularly in deep space missions, due to communica- tion latency and sometimes lengthy blackout periods. To take the benefit of real-time clinical guidance during POCUS into space, and overcome the limitations related to reliance on communication with MCC, we develo- ped a proof of concept of an augmented reality coach (AR Coach) as an augmented clinical tool (ACT) to support space crews in a progressively Earth-independent manner. The AR Coach is a hands-free system, using AR and highly realistic digital coach avatars, that guides a crew with limited ultrasound training on performing POCUS during medical emergencies in space in real-time.
MATERIALS AND METHODS
To better understand the context and design requirements for the proposed ACT, we applied a human-centered design approach as part of our wider space medicine research program. Figure 1 shows a graphical summary of the approach.
Expert Panel As a part of our ongoing NASA-TRISH (Translational Research Institute for Space Health) project, we first convened a multidisciplinary subject matter
expert (SME) panel, which included astronauts, flight surgeons, clinicians, extended reality (XR) experts, machine learning (ML) and artificial intel- ligence (AI) experts, human factors, engineering, and medical simulation (n = 46). The sessions with the SMEs were completed through 5 monthly video conferences; they also completed Delphi method surveys to identify essential XR capabilities and technical features for medical training and clinical support. Eighty-nine XR capabilities were identified, 76 of which were considered essential for space missions, medical training, and real-time clinical guidance. We also conducted iterative card sorts (open sorting: 8 rese- archers, closed sorting: 3-4 researchers) to categorize these capabilities. We identified 11 categories: XR Features, Assessment and Feedback, Clinical Competence, Clinical Guidance, Environmental Fidelity, Interoperability and Integration, ML/AI, Platform Customization, Adaptive Learning, Training Modalities/Pedagogy, and User Experience and Interface.
Task Analysis A cognitive task analysis (CTA) with five experts (3 clinicians, 2 human factors researchers, and 1 XR developer) was conducted to elucidate the process model of a POCUS-guided procedure to diagnose a potentially life- threatening condition (i.e. pneumothorax) that could occur during space travel. CTA is a qualitative approach of analysis derived from human factors and cognitive psychology (Rasmussen et al., 1994), and has been used in a variety of formats including interviews, observation, talk-aloud/think-aloud protocols, etc. In this study, we performed two sessions of semi-structured interviews—cued by images from different phases of POCUS—with three experts in POCUS. We used these sessions to understand clinicians’ mental representations of activities and work processes they are engaged in POCUS and how those perceptions relate to actions and behavior. The results of the task analysis (online supplemental material at https://tinyurl.com/2p83w687) were used to guide the development process of the AR Coach iteratively.
Development Informed by the identified XR capabilities and results of the card sorting, and the expert panel and task analysis, we created a proof of concept of the AR Coach which includes holographic panels that guide the astronauts during the diagnosis of pneumothorax using POCUS. The panels show (i) patient vital signs monitor, (ii) ultrasound reference images with videos, and (iii) a realistic human-like avatar communicating the procedure process model (steps and sub-steps) via voice and text. These holographic objects overlay the user’s field of view and are easily adjustable by users’ head movements. The voice- based interaction allows users to engage with the AR Coach conversationally. The details of design concepts and components are described (Table 1).
a) AR Coach with a high level of realism: We used the MetaHuman appli- cation from Unreal engine to create a virtual character for the AR Coach. MetaHuman Creator is a cloud-streamed app that creates fully rigged (series of interconnected digital bones) digital humans quickly with high quality (MetaHuman Creator, n.d.), fidelity, and realism. After creating the virtual
AR-Coach: Using Augmented Reality (AR) for Real-Time Clinical Guidance 71
Table 1. AR Coach capabilities based on essential XR capabilities and technical features for real-time clinical guidance identified by an expert panel (n = 46).
Table available in the original PDF paper.
Category Purpose AR Coach capabilities Extended Reality (XR)
Augmented reality application runnable in HoloLens HMD
Support inclusion of XR features (some specific to medicine)/Permit multiple users within the XR environment Assessment and Feedback Structure performance feedback/Provide feedback/monitor progress Adapt to support individualized learning
Monitoring the progress of POCUS procedure Adaptive Learning Clinical Competence
Gain new or refresh existing knowledge/Train procedural, technical and non-technical skills
Helping the user to gain new or refresh existing knowledge of using POCUS
Clinical Guidance
Provide clinical guidance
Guiding the user, step-by-step and interactively, to perform imaging using POCUS
Displaying patient’s vital information on a digital panel
Displaying reference image needed for POCUS in a digital panel Environmental Fidelity
Have environmental realism
AR Coach with a high level of realism (design based on Metahuman and Unreal engine) Interoperability and Integration
Integrate multiple sources of data/information/Integrate with other systems on- and off-board
Integrating voice, hands, head, and gesture
Allowing the user to access a network web page ML/AI Use artificial intelligence
Allowing the user to select interaction modality between voice and hand interaction Adaptive Learning
Adapt to support individualized learning Structure and enable approaches to experiential learning
Not specific capability related to adaptive learning Training Modalities/Pe- dagogy Optimize usability and clinical applicability
Allowing the user to select different options and explore the POCUS procedures using next and back options User Experience and Interface
Makes the user hands free for performing the procedure
Reduces gaze back and forth to the reference image as it is aligned with the user field of view
The user can dynamically change the position of the checklist using head movement
character for our AR Coach, we exported it to the Unreal game engine and used the LiveLink application to stream human face motion data captured via an iPhone 12 Pro equipped with LIDAR technology. Using this method,
Figure 2: Demonstration of the AR Coach being used in a simulated space station during a POCUS procedure: right (AR avatar), middle (US image), left (patient vital signs).
we were able to stream high-quality facial expressions to the virtual Meta- Human avatar, visualizing it with live rendering in the Unreal engine, and recording the AR Coach conversations. The recorded short clips were then imported onto the Unity 3D game engine and embedded into the procedure guidance process model. Figure 2 shows a participant using the AR Coach to perform a POCUS exam in a simulated spacecraft.
b) Voice-Based Interaction: We used Microsoft’s Mixed Reality Toolkit (MRTK) from the AR foundations of Unity to make the AR Coach applica- tion compatible with HoloLens. MRTK is a cross-platform toolkit that has a set of components and features used for AR/VR development. Voice-based interaction also was incorporated into the AR Coach application using Spee- chManager and IMixedRealitySpeechHandler classes provided in the MRTK. Using this capability, astronauts can interact with the AR Coach using voice commands while their hands are free to engage in POCUS procedures.
c) Head-Based Interaction: Although AR has been shown as a promising technology to provide useful information for both training and real-time gui- dance in healthcare (Zhu et al., 2014), inappropriate placement of virtual information can adversely impact the operator’s cognitive load, increasing the chance of human errors, and potentially compromising patient safety (García-Vázquez et al., 2018). To ensure that the 2D panels are properly
AR-Coach: Using Augmented Reality (AR) for Real-Time Clinical Guidance 73
located in the user’s field of view, we implemented head-based interaction in the application, allowing the user to adjust the location of the panels by moving his head up/down and right/left. We used the SloverHandler function to iteratively test the quality of interaction and then selected the head as the tracked target type with a vertical sensitivity of 20 degrees.
DISCUSSION
POCUS is the sole diagnostic imaging modality that has been used to manage medical events during space exploration missions. However, crew training in POCUS is very limited due to the many competing competencies that astro- nauts need to develop before spaceflights. These unique constraints require the development of novel training and clinical guidance systems to support astronauts performing POCUS in a progressively Earth-independent fashion towards fully autonomous medical care. In this study, we created a proof of concept of an AR Coach, a virtual coach system that serves as an unobtru- sive real-time clinical guidance to help astronauts properly use POCUS during in-flight medical emergencies.
The design of the AR Coach is novel from several perspectives. First, its design is based on essential capabilities identified by a multidisciplinary panel of SMEs with expertise in areas directly associated with deep space medicine, training, human factors, advanced technologies, and computing. Second, we used a human-centered design process iteratively guided by the findings from a task analysis conducted through interviews with emergency clinicians and human factors experts. Third, the system is deployed on head-mounted display devices to provide real-time clinical guidance for astronauts with the help of a realistic AR Coach, in a conversational context, using voice and gesture interactions. With the unique capability of supplementing the real world with virtual objects, this technology makes digital information related to POCUS imaging and patient vital signs coexist in the same space as the real environment, enabling astronauts to have ready access to critical medical data as they receive step-by-step guidance in performing POCUS.
This proof of concept provides an interactive clinical guidance experience for astronauts on using POCUS via an AR glass (HoloLens). Integrating AR technology with a realistic and real-time coach in a conversational context has great potential for clinical guidance systems. Future studies can use this proof of concept to experimentally evaluate its usability and effectiveness in optimizing astronauts’ performance in using POCUS. Future research is needed to evaluate the effectiveness of the AR Coach for non-medical per- sonnel of varying science backgrounds and education levels on how they use a portable ultrasound for diagnostic and procedural purposes.
CONCLUSION
In this study, we developed a proof of concept of a conversational AR Coach with a voice- and a gesture-based interface that can be used as an ACT to guide astronauts through clinical tasks, such as POCUS, in space. The fin- dings from this work can be used to advance space technologies that support
astronauts in managing medical events during space exploration missions, optimizing performance, and improving crew safety. The proposed AR Coach can also be used as a decision support and real-time task guidance system for other non-medical space operational situations, such as maintenance tasks and extravehicular activities (i.e. spacewalks). As deep space travel becomes more commonplace, further work exploring the utility and applicability of novel instructional tools such as this will be needed.
ACKNOWLEDGMENT
We would like to thank all the experts who participated in our panel acti- vities and discussions. This work was funded by the Translational Research Institute for Space Health (TRISH) through NASA Cooperative Agreement NNX16AO69A (Grant # T0506).
REFERENCES
Bartone, P. T., Roland, R. R., Bartone, J. V., Krueger, G. P., Sciarretta, A. A., & Johnsen, B. H. (2019). Human Adaptability for Deep Space Missions: An Explo- ratory Study. In Journal of Human Performance in Extreme Environments (Vol. 15, Issue 1). https://doi.org/10.7771/2327-2937.1124 García-Vázquez, V., von Haxthausen, F., Jäckle, S., Schumann, C., Kuhlemann,
I., Bouchagiar, J., Höfer, A.-C., Matysiak, F., Hüttmann, G., Goltz, J. P., Kleemann, M., Ernst, F., & Horn, M. (2018). Navigation and visualisation with HoloLens in endovascular aortic repair. Innovative Surgical Sciences, 3(3), 167–177. Good, R. J., O’Hara, K. L., Ziniel, S. I., Orsborn, J., Cheetham, A., & Rosenberg, A.
(2021). Point-of-Care Ultrasound Training in Pediatric Residency: A National Needs Assessment. Hospital Pediatrics, 11(11), 1246–1252. Institute of Medicine, Board on Health Sciences Policy, & Committee on Creating a Vision for Space Medicine During Travel Beyond Earth Orbit. (2001). Safe Passage: Astronaut Care for Exploration Missions. National Academies Press. Jadvar, H. (2000). Medical imaging in microgravity. Aviation, Space, and Environ-
mental Medicine, 71(6), 640–646. LoPresti, C. M., Schnobrich, D. J., Dversdal, R. K., & Schembri, F. (2019). A road map for point-of-care ultrasound training in internal medicine residency. The Ultrasound Journal, 11(1), 10. Martin, D. S., Caine, T. L., Matz, T., Lee, S. M. C., Stenger, M. B., Sargsyan,
A. E., & Platts, S. H. (2012). Virtual Guidance as a Tool to Obtain Diagnostic Ultrasound for Spaceflight and Remote Environments. In Avia- tion, Space, and Environmental Medicine (Vol. 83, Issue 10, pp. 995–1000). https://doi.org/10.3357/asem.3279.2012 Martin, D. S., South, D. A., Garcia, K. M., & Arbeille, P. (2003). Ultrasound in space.
Ultrasound in Medicine & Biology, 29(1), 1–12. MetaHuman Creator. (n.d.). Retrieved January 21, 2022, from https://metahuman.
unrealengine.com/ Norsk, P. (2020). Adaptation of the cardiovascular system to weightlessness: Surprises, paradoxes and implications for deep space missions. Acta Physiologica , 228(3), e13434. Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems
AR-Coach: Using Augmented Reality (AR) for Real-Time Clinical Guidance 75
Robertson, J. M., Dias, R. D., Gupta, A., Marshburn, T., Lipsitz, S. R., Pozner, C. N., Doyle, T. E., Smink, D. S., Musson, D. M., & Yule, S. (2020). Medical Event Mana- gement for Future Deep Space Exploration Missions to Mars. In Journal of Surgi- cal Research (Vol. 246, pp. 305–314). https://doi.org/10.1016/j.jss.2019.09.065 Sargsyan, A. E., Hamilton, D. R., Jones, J. A., Melton, S., Whitson, P. A., Kirkpa-
trick, A. W., Martin, D., & Dulchavsky, S. A. (2005). FAST at MACH 20: Clinical Ultrasound Aboard the International Space Station. The Journal of Trauma and Acute Care Surgery, 58(1), 35. Seagull, F. J., Ho, D., Radcliffe, J., Xiao, Y., Hu, P., & Mackenzie, C. F. (2007). Justin-time training for medical emergencies: Computer versus paper checklists for a tracheal intubation task. Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting. Human Factors and Ergonomics Society. Meeting, 51(11), 725–729. Zhu, E., Hadadgar, A., Masiello, I., & Zary, N. (2014). Augmented reality in
healthcare education: an integrative review. PeerJ, 2, e469.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Healthcare and Medical Devices
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-27-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional