In this paper a feasibility study of passthrough Extended Reality (XR) with maritime commissioning as a use case is presented. Passthrough XR is a technology designed to implement Augmented Reality (AR) with modern Virtual Reality (VR) devices. The driving force for this research arises from the shipbuilding industry's need to optimise installation and validation processes during critical phases, such as sea trials and larger commissioning process. Prior research into the employment of XR technologies within the shipbuilding industry shows that tools allowing hands-free operation should be favoured, and the use of video passthrough HMDs should be avoided due to the (then) limited capabilities offered by the technology. The research involved the development of an XR environment designed for analysing the passthrough capabilities of modern VR HMDs using the Meta Quest 3 platform. The primary objective of this case study was to assess the maturity of contemporary mobile XR technologies for industrial applications within the shipbuilding sector via a testing session held for participants linked to the shipbuilding industry (n=33). The results revealed potential for the contemporary application of passthrough XR technologies in shipbuilding.
Modern shipbuilding projects are vast and the commissioning of ships is reliant on validating technologies provided by a multitude of manufacturers. For these purposes a solution capable of presenting key data from a selection of complex and often unstructured documents is necessary. Extended Reality (XR) technologies present a new paradigm of human computer interaction, and can be utilized to present data in novel ways.
Passthrough XR is a technology designed to mimic Augmented Reality (AR) with modern Virtual Reality (VR) devices. Passthrough XR enables users to perceive their physical surroundings while immersed in VR through the use of front-facing cameras integrated into their head mounted display (HMD). The resulting experience closely resembles the superimposition of digital content onto the real world, as in optical see-through AR, though with some notable distinctions; the video feed is more controllable, and the precision and field-of-view of the digital overlay is improved. However, the drawbacks of passthrough XR can include slight distortion of the
Passthrough Extended Reality in Maritime Commissioning 407
surrounding environment and potential fluctuations in frame rates. Recent advancements in this technology have been significant, particularly over the past few years. Current market leaders, such as Meta with the Quest 3 (Meta, 2024), and Apple with the Vision Pro (Apple, 2024) now offer full-colour passthrough capabilities with such precision that users can comfortably navigate and interact with their physical environment while wearing the VR headset. This combination of handsfree operability, perception of one’s surroundings, and capability to display information to users makes modern XR a noteworthy technology to explore as a tool for industrial use cases.
This paper outlines the development of an XR pilot implementation designed to showcase the capabilities of modern passthrough XR using the Quest 3 HMD developed by Meta. The primary objective of this pilot was to evaluate the maturity of contemporary XR technologies for industrial applications within the shipbuilding sector. This evaluation was conducted by presenting the pilot to an audience composed of shipbuilding professionals and academic researchers focused on the digitalisation of shipbuilding processes. The demonstration allowed users to install and operate electronic devices under the guidance of an XR solution. Methods used in this research were observation to gather qualitative data and survey to gather quantitative data.
BACKGROUND AND RELATED WORK The driving force for this research arises from the need of the shipbuilding industry to optimise installation and validation processes during critical phases, such as sea trials and the larger commissioning process. The documentation for these processes can be vast and often unfamiliar to the engineers using them (Berndt, Von Lukas and Kuijper, 2015; Peplinski, 2019, pp. 467–484). The aim is to digitise these documents, while enabling technicians to maintain hands-free operation during installation and validation tasks. In this context, the potential of XR was considered highly promising.
The shipbuilding industry has been making XR integrations during the past decade. Most notably, XR technologies have been developed for visualizing design models and for training purposes (Shankhwar et al., 2022; Garza Espinosa et al., 2023). The research into XR in shipbuilding indicates that traditionally AR HMDs like the Hololens by Microsoft (Microsoft, 2025) have been flavored for industrial use cases. AR allows for users to perceive their surroundings, which is deemed crucial for industrial applications. Although video passthrough XR allows perception of the surroundings of a user, traditionally this approach has not been recommended for industrial use cases due to lacking performance (Von Lukas, Vahl and Mesing, 2014; Fraga-Lamas et al., 2018). The viability of video passthrough XR should, however, be re-evaluated due to the recent advancements.
TECHNICAL IMPLEMENTATION In order to introduce the capabilities of modern XR to shipbuilding experts, a pilot implementation of a video passthrough XR tool utilizing
AI enhancements was developed. This development consisted of the following phases: requirements identification, architecture design, technical implementation, and user experience (UX) development.
The identification of requirements was conceived through stakeholder analysis. Shipbuilding professionals in charge of conducting commissioning (and similar) tasks were identified as the primary stakeholders and researchers responsible for scaling the solution were identified as secondary stakeholders. It was assumed that the primary stakeholders do not have extensive prior experience in using XR technologies. The analysis led to the following requirements; an XR solution capable of recognizing or remembering target interfaces, so as not to require extensive setup upon being used. The solution should provide the user information about target interfaces in a manner which does not disrupt the workflow of the user. The solution should also not require novice users to learn complex actions or gestures.
The Quest 3 was chosen as the target HMD for this implementation, as it supports video passthrough, hand tracking and spatial awareness through the use of the device Application Programming Interfaces (APIs). These APIs can be accessed through the Unity3D development framework (Unity, 2024b) and the Meta All-In-One software development kit (SDK) (Unity, 2024a). As an industry standard, XR device manufacturers do not allow developers direct access to passthrough data from HMD hardware. This sets a limitation to the automatic recognition of interfaces. The spatial anchor API from the All- In-One SDK is used instead. With spatial anchors, the solution is capable of remembering the physical environment of the user, and can load content into the same place between multiple sessions. This allows for the creation of a solution, which can be set up by administrators once, and then accessed by future users without requiring any setup. Additionally, as an included benefit, users are not required any kind of controller interaction and can rely solely on their hands. Administrators in charge of setting up the solution use controllers for increased precision (see Figure 1).
Figure 1: Steps to set up the and use the solution and the interaction types.
In terms of the digital content tied to the XR environment, two target tasks were chosen for the solution: an oscilloscope task and a cabling task
Passthrough Extended Reality in Maritime Commissioning 409
(see Figure 2). The oscilloscope represents a complex interface, which users can request information about and where users can interact with purely digital content. The cabling task represents a setup task, during which users are required to follow instructions and perform the physical actions of setting up cables according to the instructions. Combined, these two target interfaces represent interaction with both; physical and digital content. This approach allows the evaluation of the XR technology, while also presenting a wide variety of the capabilities of modern video passthrough XR to the participating testers.
Figure 2: Oscilloscope and cabling task.
The UX development for this pilot implementation followed the Nielsen usability heuristics for XR (Kendrick, 2021). In particular, the highlighting of intractable content as outlined by Nielsen’s heuristic one: “Visibility of System Status” was emphasized as shown in the oscilloscope displayed in Figure 2. Heuristic two: “Match Between System and the Real World” was also deemed crucial and the targets for digital content were selected to mimic real world counterparts. A final stand-out aspect of the UX design involved heuristic five: “Error Prevention”. Interacting with the oscilloscope dials involved activating the wanted dial, after which, a copy of the dial, which could then be turned, appeared above the oscilloscope (instead of attempting to place the interactive dial in the already cramped frontal plate of the oscilloscope). Similarly, when interacting with the computer during the cabling task, proximity to the hand of the user would activate a fading of the digital computer model. This was done, so users would have full vision of the real computer, to which they could place cables to.
RESEARCH SETTING This study involved 33 participants, all of whom were linked with the shipbuilding industry, making them representative of the intended target audience for this research. The participants were observed while testing the implementation, and afterwards they were asked to fill a questionnaire regarding their experience with the pilot implementation.
Participant observation is a method where a participant is conducting activities while a researcher is observing and taking notes without interfering
in the activities (Shull, Singer and Sjøberg, 2008). The purpose of the questionnaire was to find out how the participants experienced the XR environment. The questionnaire was created with open and closed questions regarding the participants previous experience with the technology, user experience and thoughts about using this similar technology in the future. The System Usability Scale (Brooke, 1996) was used as a starting point for the questionnaire, and some of the original questions (such as “I thought the solution was easy to use”) were included in the final questionnaire.
The test sessions were held in a mixed reality laboratory. Participants began with the oscilloscope demo and proceeded to the cabling task (see Figure 2). After completing the tasks, the participants filled out the questionnaire. The average testing time was 7 minutes. The view from the glasses was cast onto a laptop making it possible for test conductors to help the participants in case issues arose during testing. One test conductor was guiding the participants through the testing activities and helping them with the device while another was observing the situation. There was slight variation in the testing conditions between participants due to differing sizes of participant groups. As a result, people waiting for their turn could see others using the device and may have seen the cast from the laptop, making the user interface more familiar to them than to participants who were alone during their test session.
RESULTS
This section begins by introducing the key findings from the observations and continues with questionnaire results. Based on the observations of the testing sessions, the following key details regarding the pilot implementation can be outlined:
1. Participants were generally excited and did not get frustrated when testing the solution. 2. Participants in general had some difficulties locating the digital dials appearing above the oscilloscope. Further difficulties were observed when participants interacted with highlighted dials in the digital oscilloscope, as well as, turned the digital dials above the oscilloscope (see Figure 2 for the aforementioned digital content). Despite these difficulties, all users managed to complete the entire demonstration. 3. Tall participants had problems interacting with the digital content.
Responses to the multiple-choice questions of the questionnaire are presented next. First, respondents were asked about their professional background, as well as, their prior experience using XR systems. Detailed results for the backgrounds of testers are shown in Figure 3.
Passthrough Extended Reality in Maritime Commissioning 411
Figure 3: Participant professional background and prior experience with XR systems.
Following the background questions, testers were asked about their experience with the pilot implementation. Respondents with high prior experience using XR, answered that the quality of the passthrough video was good, working with virtual content while wearing the HMD was somewhat easy (both answering somewhat agree), and that interacting with virtual content was cumbersome (all answers being “somewhat disagree” when asked if virtual interaction were intuitive). Participants with little to no prior experience using XR had otherwise similar answers, but found interacting with virtual content to be generally intuitive. Between the respondents who were very familiar with XR, the interactions with digital content (i.e., activating and turning digital dials or pressing digital buttons using their hands) were unsatisfactory. At the same time, however, both respondents found the solution easy to learn and did not find it frustrating to use. Amongst the 20 respondents with little to no prior experience using XR, there was a similar consensus on spending too much time interacting with the digital content, as well as, finding the solution non-frustrating and easy to learn. The prior experience of users did not play a significant role in how usable the participants found the solution. The results for the multiple-choice questions are illustrated in Figure 4.
In total, testers had a varied background in terms of experience with XR systems and were split evenly between industry professionals and researchers. When asked about their experience with the pilot implementation, users found using the solution to be an easy to learn and non-frustrating experience. They found the virtual environment and the quality of the passthrough video pleasant and on par with their expectations for such a solution. Users found working with physical objects, as well as, working with their hands, while wearing the XR HMD, mostly intuitive and easy, but some interactions required too many attempts or repeat actions.
Figure 4: Participant responses to questionnaire.
Following the multiple-choice questions were open answer questions. The first of the open answer questions addressed the suitability of the pilot for ship building and especially ship commissioning by asking respondents if they could imagine a similar solution being used during ship commissioning. The results showed a large majority of users (26 of 33) believing that the solution was well suited for commissioning. Recommended use cases varied between training, remote presence, verification aid and installation support. Of the remaining respondents, six were not sure and one user did not find the solution suitable for tasks conducted during commissioning.
After questions regarding suitability in commissioning, the users were asked what they would like to see developed further in the experience. The most common request was improved interactions with digital content (8 answers). Second most popular was an inclusion of an interactive assistant users could ask questions from (5 answers). Third most common requests were the inclusion of text-based instructions in addition to the voiced instructions already present (3 answers) and the improvement of the resolution of the experience (3 answers). The fourth and final topic with multiple notes was to remove the default hand tracking visualization overlaid on the hands of users (2 answers).
An analysis of the results from both the questionnaire and observations presents some common themes. Tracking worked, albeit inconsistently at times: the HMD had no problem maintaining pose estimation within the usage area throughout the user tests. The quality of hand tracking varied; after prolonged usage, the HMD began recognizing the left hand of some participants as a controller, forcing them to perform the tasks using only one hand. Additionally, some open answers noted that the default visualization of tracked hands (see Figure 2) performed by the Quest 3 HMD was distracting, and that it got in the way of performing tasks.
Passthrough Extended Reality in Maritime Commissioning 413
No issues with passthrough quality were found: participants responded to questions regarding the video passthrough (how pleasant the virtual environment was & if passthrough quality was satisfactory) positively (see Figure 4). The same could be noted when observing the participants; no signs of nausea (which can be linked with motion sickness caused by low latency video passthrough), getting lost in the virtual environment, or failure to discern details when working with the headset could be detected.
Quality of digital content was lacking: participants who were tall had issues engaging with the virtual content in the solution and would have to crouch down in order to interact with some of the aforementioned content. Some participants had trouble locating where digital content appeared after activating it. Some users also found the highlighted objects in the UI misleading. Engaging with some of the digital content (especially the oscilloscope dials) was challenging to some participants and was seen as “buggy” or finicky in the open question answers for the questionnaire. Similar results can be detected from the multiple-choice question results, where over half of the respondents noted that they had to spend too much time repeating interactions with virtual content (see Figure 4). When observing the participants, the same issues could be detected; participants not finding UI elements or failing to interact with the elements without having to repeat their actions.
General attitude towards the test (and XR in common) was positive: participants found using XR a positive experience as noted by the open answers and the approval of the pilot quality in the questionnaire. Similar positive themes were present when asking about the suitability of XR in the commissioning of ships. The same can be observed from the user tests: working with the HMD did not cause issues for the participants and even the less tech savvy participants who had no prior experience with XR enjoyed the experience. It is possible that the variance in testing conditions had an impact on the interactions and the learnability of the pilot since some of the participants were able to observe what the others were doing while waiting for their turn. However, this was considered a minor issue, since the main goal of this study was to evaluate the maturity of XR technology and not solely to measure how quickly testers could learn to use the pilot implementation.
DISCUSSION
The tracking and passthrough issues are closely related to the suitability of XR HMDs in ship commissioning, since they are directly evaluating the capabilities of the XR technology being used. Based on the user responses and observations: the technology, in terms of passthrough and tracking capabilities, met the expectations of the testers and was considered suitable for use during ship commissioning.
Themes of digital content quality are mostly tied to the pilot implementation application. A few issues related to digital content quality can, however, be associated with XR HMD evaluation: namely the visualization of the hands of a user when the device is performing hand tracking. Allowing for developers to disable the hand tracking visualization
would further enable the Quest 3 HMD to be used as an industrial XR tool. Some general takeaways regarding UX design for an XR tool can also be made. An obvious issue is the height related problems present when interacting with digital content. This design flaw can be rectified by adjusting the position of the intractable content within the digital environment. The other stand out issue was the quality of digital dials. This issue can partly be addressed by reducing the polling rate of the dials: by reducing the polling rate, the dial reacts to the inputs of a user at a more manageable pace, rather than sporadically adjusting whenever a touch interaction is registered. If the dials are deemed as a negative user experience, even after reducing polling rates, another form of interaction with the oscilloscope may be required. Based on the aforementioned issues, it is clear that the pilot implementation in its current state is not fully suitable for daily use in an industrial setting, but could be further developed into an efficient tool.
Issues addressing the attitude towards XR are tied to both: the platform in use, as well as, the application developed for this platform. The capabilities of the platform set constraints for applications developed for said platform. Additionally, the form factor of the HMD (weight, bulk and ergonomics) impacts the user experience of any application run on the platform. Ultimately, however, the experience that a user has with XR is deemed by the solution they are using. Based on questionnaire responses regarding learning to use the solutions and how frustrating of an experience using the solution was, as well as, observations regarding the same issues, it can be deemed that the general attitude towards an XR tool for ship commissioning was very positive. This sentiment is further enforced by the generally positive responses to the open questions asking about the suitability of the technology for commissioning tasks.
CONCLUSION
This paper outlined the development and evaluation of an XR tool for ship commissioning. The motivation for the research stems from the need of the shipbuilding industry to optimise access to information regarding interfaces supplied by the numerous parties involved in shipbuilding projects. An evaluation of a pilot implementation based on the analysis of questionnaire and observational data from a testing session was made. The participants present during the testing session were linked to the shipbuilding industry.
Based on the results of testing, the Quest 3 HMD, and by extension, contemporary video passthrough XR technology, was deemed suitable as a platform for a tool used during ship commissioning. Tester responses to the questionnaire presented to them revealed, that key technical aspects of the Quest 3 HMD (passthrough quality and tracking quality) were suitable for industrial tasks like ship commissioning. Observations of the test setting support the questionnaire results. Furthermore, testers listed multiple potential uses for a similar tool during the commissioning process during the open answer section of the questionnaire.
When developing a commissioning tool for contemporary XR HMDs, certain technical details must be taken into consideration. The HMDs are not
Passthrough Extended Reality in Maritime Commissioning 415
capable of freely conducting object recognition; an anchoring system should be used for remembering usage areas instead. Additionally, when developing solutions with AR components, a diverse set of testers should be utilised in order to account for aspects like user height affecting the user experience.
This study had two main limitations. First, the variations in research setting may have impacted the results regarding learnability. Second, while the participants in this study were linked with the shipbuilding industry, they were not necessarily commissioning engineers. This may have an impact on the accuracy regarding the suitability of the technology in actual commissioning tasks.
The current state of the pilot implementation detailed and tested during this research serves as a demonstration of the capabilities of XR technology imagined for a ship commissioning environment and has served as a starting point for developing a tool. However, in order to provide value for commissioning tasks, further development is required. To address specific needs of the target user demographic, co-development or similar process between the users and application developers should be initiated.
ACKNOWLEDGMENT
This research was done in the Business Finland funded project Virtual Sea Trial under Grant 7316/31/2023.
REFERENCES
Apple (2024) Apple Vision Pro, Apple. Website: https://www.apple.com/apple-visi on-pro/ (Accessed: 22 November 2024). Berndt, O., Von Lukas, U. F. and Kuijper, A. (2015) ‘Functional modelling
and simulation of overall system ship-virtual methods for engineering and commissioning in shipbuilding’, in. Proceedings - 29th European Conference on Modelling and Simulation, ECMS 2015, pp. 347–353. Brooke, J. (1996) ‘SUS: A “Quick and Dirty”Usability Scale’, in Usability Evaluation In Industry. CRC Press. Fraga-Lamas, P. et al. (2018) ‘A Review on Industrial Augmented Reality Systems for
the Industry 4.0 Shipyard’, IEEE ACCESS, 6, pp. 13358–13375. Garza Espinosa, J. A. et al. (2023) ‘Creation of a Virtual Reality Environment for Teaching Applied to a Catamaran-Type Ferry, Detailing the Engine Room’, in. 2023 IEEE International Conference on Engineering Veracruz, ICEV 2023. Kendrick, A. (2021) 10 Usability Heuristics Applied to Virtual Reality, Nielsen
Norman Group. Website: https://www.nngroup.com/articles/usability-heuristics- virtual-reality/ (Accessed: 19 January 2025). Meta (2024) Meta Quest 3: New mixed reality VR headset – Shop now | Meta Store.
Website: https://www.meta.com/fi/en/quest/quest-3/ (Accessed: 22 November 2024). Microsoft (2025) Buy HoloLens 2: Find Specs, Features, Capabilities & More, Microsoft Store. Website: https://www.microsoft.com/en-us/d/hololens-2/91pnz zznzwcp (Accessed: 17 January 2025). Peplinski, H. (2019) Ship and Mobile Offshore Unit Automation - A Practical Guide.
Elsevier.
Shankhwar, K. et al. (2022) ‘A visuo-haptic extended reality–based training system for hands-on manual metal arc welding training’, The International Journal of Advanced Manufacturing Technology, 121.0(1–2). Shull, F., Singer, J. and Sjøberg, D. I. K. (eds) (2008) Guide to Advanced Empirical
Website: https://assetstore.unity.com/packages/tools/integration/meta-xr-all-in-o ne-sdk-269657 (Accessed: 17 January 2025). Unity (2024b) Unity Real-Time Development Platform | 3D, 2D, VR & AR Engine, Unity. Website: https://unity.com (Accessed: 14 January 2025). Von Lukas, U., Vahl, M. and Mesing, B. (2014) ‘Maritime applications of augmented
reality - Experiences and challenges’, in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 465–475.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-71-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional