Roman Bolton, Craig Allison, Alex Smith, Sophie Hallam, Jade Melendez, Katie Plant (2023). An investigation into how an integrated user interface and virtual reality affects operator performance when completing submarine control room tasks. In: Human-Centered Design and User Experience. AHFE Open Access, vol. 114, pp. 456–465. https://doi.org/10.54941/ahfe1004264
Submarines use a plethora of sensors crucial for above-water surveillance. For example, the optronics mast utilises sensors that collect optical data on the surrounding environment. Whereas the Radar Electronic Support Measures (RESM) mast provides electromagnetic surveillance that focuses on avoiding counter-detection. Paradoxically though, every time the mast is up it increases the risk of counter-detection. To reduce exposure time, submarine masts are integrating multiple sensors; like optronics and RESM, to collect data simultaneously. Traditionally, different operators complete optronics and RESM tasks. However, an integrated optronics and RESM mast, would likely require an integrated operator role. Therefore, optimising the Human-Machine Interface would enable optimal operator performance. One suggestion is to present both optronics and RESM data on a single user interface and explore different ways of presenting this information, using more emerging technologies. Therefore, the aim of this study is to investigate how an interface, which supports the presentation of both optronics and RESM data, affects operator performance compared to an interface that presents optronics data only. The study will also explore the effects of presenting such information using current and novel display methods, specifically computer monitors and virtual reality (VR). To test this, four experimental conditions were devised: (1) no additional data using a conventional display, (2) additional RESM data using a conventional display, (3) no additional data using a VR display, and (4) additional RESM data using a VR display. To assess operator performance, participants will complete simulations in each condition, and data will be collected on task accuracy, task completion time, operator workload, situation awareness, and system usability. A detailed account of the research findings will be presented.
An Investigation Into How an Integrated User Interface and Virtual Reality 457
INTRODUCTION
Submarine sensors gather information about the surrounding environment, enabling the submarine to operate safely, covertly, and effectively. The optronics mast utilises sensors that collect optical data on the surrounding environment (Stevenson, 2005), while the Radar Electronic Support Mea- sures (RESM) mast utilises sensors that collect electromagnetic data (Moir, 2019). Both masts are critical to above-water surveillance. However, while above-water surveillance is conducive to data gathering, it is not conducive to covertness. One way to combat counter detection is to reduce mast exposure time (Vieira, 2016). To do this, submarine masts are becoming increasingly modular, allowing them to integrate multiple sensors, like optronics and RESM, and collect data simultaneously.
Traditionally, different operators complete optronics and RESM tasks. However, an integrated optronics and RESM mast, would likely require an integrated optronics and RESM operator role. So the question then, is how would the Human-Machine Interface (HMI) enable optimal perfor- mance of a single operator? Understanding the impact of HMI is important to ensure that the operator receives sufficient and relevant information that helps, rather than impedes, them in task completion (Houghton et al., 2006; Salmon et al., 2017; Stanton et al., 2006). The optimisation of HMI requires the ‘translation’ of sensor awareness to operator awareness. Optimising this will enable the human operator to make better decisions that align with their ability to perceive their environment (Salmon et al., 2017; Stanton et al., 2006). One way in which HMI can optimise operator performance is through the introduction of novel user interfaces that integrate data from disparate sources (Fay et al., 2017). Research has shown that an integrated subma- rine interface can outperform a conventional interface in several measures of subjective performance, when completing tasks related to tactical picture generation (Bolton et al., 2022).
An alternate way in which HMI can optimise operator performance is through the utilisation of novel display types. Emerging technologies, like Virtual Reality (VR) are becoming increasingly ubiquitous as their bene- fits become more apparent. VR displays synthetic sensory information that enables a user to perceive a particular environment (Blascovich et al., 2002). A VR environment provides a more immersive experience by physically surrounding the user with synthetic information. Augmenting the level of immersion can have an effect on how the user perceives their reality (Smith, 2015). An optronics operator relies on visual data to perceive reality, whereas a RESM operator relies on electromagnetic data to perceive reality. If an oper- ator’s perception could be augmented, through immersive technology, then it is feasible that they would become more effective in their tasks. One approach of improving task performance is enhancing an operator’s situation aware- ness (SA), which encompasses perceiving environmental elements and events in relation to time and space, comprehending their significance, and project- ing their future status (Endsley, 2017). Given that submarines operate in a safety-critical domain, increasing SA could empower the command team to make superior tactical decisions. In the current context, VR has the potential
to enhance an operator’s task performance by augmenting their perception of reality and, consequently, improving their SA. This, in turn, could enable them to make more informed tactical decisions and mitigate risks that may jeopardize the safety of the submarine and its crew.
The current research aims to explore the use of a novel interface that sup- ports the presentation of data from multiple sensors, aspects of which are traditionally displayed separately to different operators. More specifically, it will seek to compare operator task performance between an interface that only displays optronics data, and an interface that displays both optron- ics and RESM data. The research will also explore the effects of presenting optronics and RESM data using conventional display methods (i.e. computer monitors) and novel display methods (i.e. a VR environment) on operator task performance. It is anticipated that a better understanding of individual performance, together with the effects of interventions, will help to enhance understanding and the overall performance of future command teams (Klein & Miller, 1999).
METHOD
Design The study will employ a 2x2 within-subjects experimental design. The two independent variables are the interface and the display type. The four conditions are: (1) no additional data using a conventional display, (2) additional RESM data using a conventional display, (3) no additional RESM data using a VR display, and (4) additional RESM data using a VR display.
Participants The study aims to collect data from 52 participants. All participants will be non-military novices, with little to no prior experience of submarine con- trol room operations. They will be recruited opportunistically using posters, online advertisements, and directly contacting organisations with a mili- tary interest. The study was granted ethical approval by the faculty’s ethical committee (ID: 86442) and Ministry of Defence Research Ethics Committee (ID: 2235/MODREC/23).
Operator Role Traditionally, there are separate Optronics and RESM operators onboard a submarine. However, for this study, participants will complete tasks related to both optronics and RESM, depending on which interface they are using. The Optronics operator is required to operate the optronics mast. They are responsible for conducting visual sweeps, detecting and designating visual contacts (vessels and aircrafts), classifying contacts, estimating range of con- tacts, calculating a contact’s course, bearing, and determining a contact’s level of hostility. While using the integrated interface, the operator will have access to electromagnetic data too. The RESM operator classifies sources of electromagnetic radiation to determine contact classifications and the level
An Investigation Into How an Integrated User Interface and Virtual Reality 459
of a contact’s hostility. They do this by matching the characteristics of a received signal with the characteristics of a known signal (typically stored in an electronic database). The system will partially automate this filtering process.
Scenario One 20-minute long operationally relevant scenario (with four iterations) will be used for the present study. The objective of the scenario will be to identify and build solutions (i.e. calculate bearing, speed, range, course, classification, and hostility) for all contacts present, with priority going towards enemy ves- sels and aircrafts. To avoid practice effects, there will be four iterations of the same scenario. In each iteration, the bearings of all vessels and aircrafts will shift by 45
, while all other parameters will remain constant. Changing the bearing only was deemed a suitable way to ensure participants do not become familiar with the scenario, while simultaneously avoiding confounding vari- ables. To avoid order effects, the iterations will be counterbalanced across the participant pool.
MATERIALS
User Interface There are two interfaces to test as part of the current work: a baseline interface, with no additional data, and an integrated interface, with addi- tional RESM data. The Simulation Engine II (SEII) simulation game will be used for both interfaces and display types. SEII is a consumer-off-the- shelf product developed by Sonalysts Inc. While not identical to interfaces used onboard currently active submarines, SEII was chosen as a simula- tion engine as it allowed the creation of an environment with high task fidelity (Roberts et al., 2015; Stanton & Roberts, 2017a). The current version of SEII is a bespoke build which was developed with the input from human factors and end-user subject-matter experts. Figure 1-2 are conceptual representations of how the two interfaces will fundamentally differ.
Figure 1: No additional data.Figure 2: Additional RESM data.
Display Type There are two display types to test as part of the current work: a conven- tional display and a novel display. For the conventional display condition, the participant will be sat at a dual-monitor workstation (Figure 3). The worksta- tion will include two monitors, a keyboard, a mouse, an audio headset and a whiteboard. Depending on the interface condition, the top monitor will either display visual environmental data or combined visual and RESM envi- ronmental data. The bottom monitor will provide an additional workspace for building contact solutions. For the novel display condition, the participant will be sat at a workstation inside a VR environment (Figure 4). The VR envi- ronment is an immersive space capable of projecting dynamic images on four adjacent walls. Similar to the conventional display, the interface condition will determine what’s projected on the walls (i.e. either visual environmental data or combined visual and RESM environmental data). The workstation monitor will provide an additional space for building contact solutions.
An Investigation Into How an Integrated User Interface and Virtual Reality 461
Objective Data Data logged by the simulation engine will provide an objective measure of task accuracy (i.e. solutions entered into the system). An operator creates a contact solution by calculating its bearing, range, course, speed, hostility, and classification. However, submarines operate in ambiguous and dynamic environments, meaning solutions are not always accurate and require contin- uous updates. Therefore, this analysis will look at the error between solution data and truth data. Moreover, SEII timestamps data entered, which will be used to objectively measure task completion time (e.g. the time taken from identifying a contact to classifying it). Both measures will enable an objective assessment on operator performance.
Usability The System Usability Scale (SUS) is a subjective measure of usability (Brooke, 1996). Using a five-point likert scale, the user determines whether they strongly agree or disagree with 10 different statements (i.e. sub-scales). A score on a scale from 0–100 is then calculated. The 10 statements are: ‘I think that I would like to use this system frequently’, ‘I found the system unnecessarily complex’, ‘I thought the system was easy to use’, ‘I think that I would need the support of a technical person to be able to use this system’, ‘I found the various functions in this system were well integrated’, ‘I thought that there was too much inconsistency with the system’, ‘I would imagine that most people would learn to use this system very quickly’, ‘I found the system very cumbersome to use’, ‘I felt very confident using the system’, and ‘I needed to learn a lot of things before I could get going with the system’.
Workload The NASA-Task Load Index (NASA-TLX) is a subjective measure of a par- ticipant’s workload. The measure consists of six 7-point sub-scales: mental demand, physical demand, temporal demand, performance, effort, and frus- tration. Each sub-scale is completed and then given a weighting; the result of which, is a workload score from 0 (low demand) to 100 (high demand) (Hart & Staveland, 1988).
Situation Awareness The SA questionnaire offers a subjective measure of the participant’s SA on various aspects of the scenario (Table 1) and follows the Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1988). This questionnaire, along with the NASA-TLX scales, were completed digitally and have been used in previous studies (Roberts & Stanton, 2018; Roberts et al., 2017; Roberts et al., 2019; Stanton & Roberts, 2017a, 2017b, 2020; Stanton et al., 2017).
Procedure Data collection will take place early next year at the University of Southamp- ton. All participants will undergo training to familiarise themselves with domain specific concepts, role-specific tasks, the interfaces, and the display
types. Participants will then complete four counterbalanced scenario sim- ulations (i.e. one for each experimental condition). After each simulation, participants will complete a SUS survey, the NASA-TLX scale, and an SA questionnaire. Once the testing phase has ended, participants will fill out a qualitative survey where they can share any final feedback.
Table 1. SA questionnaire measures.
Table available in the original PDF paper.
Measure Description Scale Ownship’s Parameters Participant’s SA of ownship’s speed, course, and depth.
Scale from 1–6 (1 = having extremely low awareness, 6 = having extremely high awareness). Contact Parameters
Participant’s SA of all contacts’ bearing, classification, speed, course, and range.
Numerical value given.
Scale from 1–6 (1 = having extremely low awareness, 6 = having extremely high awareness). Total Contacts
Estimation of how many vessels the command team encountered during the scenario.
Estimation of how long the scenario lasted. Numerical value given in minutes.
Scenario Length
Equation reproduced from the original PDF.
Mission Effective- ness
Rating of how effectively the participant felt they completed the primary objective of the scenario.
Contact Priority
The level of priority allocated to a vessel. Scale from 1–6 (1 = very low priority, 6 = very high priority).
RESULTS
Performance data will be analysed using IBM SPSS Statistics version 29 (for Windows). All data will be examined to see if parametric assumptions are met. Suitable post-hoc tests will be carried out on all significant results, to understand where differences are found. To account for multiple post-hoc comparisons, the Bonferroni correction method will be used (α = .05/number of comparisons).
To assess differences in task performance (i.e. solution accuracy and task completion time), two mixed Analyses of Variance (ANOVAs) will be cal- culated to consider overall accuracy for the two interfaces and the two display types. Independent ANOVAs will be calculated for (1) bearing error, (2) course error, and (3) range error using data collected for each vessel within the scenario. 2 (interface) x 2 (display type) Mixed ANOVA will allow for consideration of both IVs. Should significance be identified, appropriate post- hoc tests will be calculated, using paired samples t-tests to explore where differences are found. Chi squared analysis will be calculated for classifica- tion error (4), and hostility error (5) using data collected for each vessel within the scenario.
To consider subjective differences in participants rating of the usability of each interface on each display type, as measured using the SUS, a total SUS score will be calculated. These will then be compared using a 2 (interface) x
An Investigation Into How an Integrated User Interface and Virtual Reality 463
2 (display type) repeated ANOVA. This process will be repeated for each SUS sub-scale.
To consider subjective differences in participants’ rating of perceived workload for each interface on each display type, as measured using the NASA-TLX, a total workload score will be calculated. These will then be compared using a 2 (interface) x 2 (display type) repeated measures ANOVA. This process will be repeated for each NASA-TLX sub-scale.
To consider subjective differences in participants rating of perceived SA while using each interface on each display type, as measured using the SA questionnaire, a total SA score will be calculated. These will then be compared using a 2 (interface) x 2 (display type) repeated measures ANOVA.
DISCUSSION
The impetus of this research is to understand how HMI affects operator performance, however, the direction of the effect is unclear. Integrating information from disparate sources could improve an operators’ capacity to perceive the world, but, it could also cognitively overload an operator. Coupling information integration with automation will likely have a positive impact on performance, particularly if an operator becomes task saturated. Previous research has shown that integrated displays, with increased automa- tion, can have a positive effect on workload, usability, and SA (Bolton et al., 2022). While automation has the capacity to liberate the operator from cer- tain tasks, it is important to note that excessive automation may result in cognitive underload. The impact of HMI has also been observed in other domains such as the medical (Ahmed et al., 2011) and nuclear industries (Burns et al., 2008). These studies showed that improved display configura- tions can reduce task load, cognition errors, and enhance SA. Alternatively, complex interfaces can increase visual searching tasks and degrade user experience (Wu et al., 2016).
Elsewhere, research conducted in the education industry has indicated that VR applications can augment information retention and improve the overall learning experience (Allcoat & von Mühlenen, 2018). Moreover, the number of clinical task errors in surgery have been shown to reduce, when initial training is administered using VR rather than more standard training methods (Seymour, 2008). Conversely, a meta-analysis showed that extended reality, like VR, does not outperform traditional instructional techniques as a training tool (Kaplan et al., 2021).
So, it is evident that both HMI and novel display methods can have a varying level of influence on different performance measures. What is unclear, is the potential effect that combining novel HMI with VR displays can have on operator performance.
FUTURE WORK
The next steps for this research is to analysis data collected from all the participants. Future work could examine the effects of display types on other members of the command team, such as sonar operators. Additionally, research could look at more practical applications of novel technology in a
submarine control room, such as augmented reality (AR) headsets. Similar to VR, AR enhances a user’s perception of reality by combining the real world environment with computer generated graphics (Bimber & Raskar, 2005), and so, similar to VR, it has the potential to affect operator task performance.
ACKNOWLEDGMENT
This work was funded by the Defence Science and Technology Laboratory (Dstl) on behalf of the Ministry of Defence (MOD). All opinions expressed are those of the authors.
REFERENCES
Ahmed, A., Chandra, S., Herasevich, V., Gajic, O., & Pickering, B. W. (2011). The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. In. Allcoat, D., & von Mühlenen, A. (2018). Learning in virtual reality: Effects on
performance, emotion and engagement. Research in Learning Technology, 26. Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N.
(2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological inquiry, 13(2), 103–124. Bolton, R., Smith, A. F., Hallam, S. M., Melendez, J., Hutchins-Lander, A., Allison, C., & Plant, K. L. (2022). The Effects of Novel User Interfaces on Operator Task Performance in a Submarine Control Room Simulator. Dstl. Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in
industry, 189(194), 4–7. Burns, C. M., Skraaning Jr, G., Jamieson, G. A., Lau, N., Kwok, J., Welch, R., & Andresen, G. (2008). Evaluation of ecological interface design for nuclear process control: situation awareness effects. Human Factors, 50(4), 663–679. Endsley, M. R. (1988). Situation awareness global assessment technique (SAGAT).
Proceedings of the IEEE 1988 national aerospace and electronics conference, Endsley, M. R. (2017). Toward a theory of situation awareness in dynamic systems.
In Situational awareness (pp. 9–42). Routledge. Fay, D., Stanton, N. A., & Roberts, A. P. (2017). Designing new interfaces for submarines: from cognitive work analysis to ecological interface design. In Advances in Human Aspects of Transportation (pp. 413–425). Springer. Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load
Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier. Houghton, R. J., Baber, C., McMaster, R., Stanton, N. A., Salmon, P., Stewart, R., & Walker, G. (2006). Command and control in emergency services operations: a social network analysis. Ergonomics, 49(12–13), 1204–1225. Kaplan, A. D., Cruit, J., Endsley, M., Beers, S. M., Sawyer, B. D., & Hancock, P.
A. (2021). The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: A meta-analysis. Human Factors, 63(4), 706–726. Klein, G., & Miller, T. E. (1999). Distributed planning teams. International Journal of Cognitive Ergonomics, 3(3), 203–222. Moir, I. (2019). Military avionics systems. John Wiley & Sons.
An Investigation Into How an Integrated User Interface and Virtual Reality 465
Roberts, A. P., & Stanton, N. A. (2018). Macrocognition in submarine command and control: a comparison of three simulated operational scenarios. Journal of applied research in memory and cognition, 7(1), 92–105. Roberts, A. P., Stanton, N. A., & Fay, D. (2015). The command team experimental
test-bed stage 1: design and build of a submarine command room simulator. Pro- cedia Manufacturing, 3, 2800–2807. https://doi.org/http://dx.doi.org/10.1016/j. promfg.2015.07.745 Roberts, A. P., Stanton, N. A., & Fay, D. (2017). The command team experimental test-bed phase two: assessing cognitive load and situation awareness in a subma- rine control room. In Advances in human aspects of transportation (pp. 427–437). Springer. Roberts, A. P., Stanton, N. A., Fay, D., & Pope, K. A. (2019). The effects of team
co-location and reduced crewing on team communication characteristics. Appl Ergon, 81, 102875. https://doi.org/10.1016/j.apergo.2019.102875 Salmon, P. M., Stanton, N. A., & Jenkins, D. P. (2017). Distributed situation awareness: Theory, measurement and application to teamwork. Seymour, N. E. (2008). VR to OR: a review of the evidence that virtual reality
simulation improves operating room performance. World journal of surgery, 32, 182–188. Smith, J. W. (2015). Immersive virtual environment technology to supplement environmental perception, preference and behavior research: a review with appli- cations. International journal of environmental research and public health, 12(9), 11486–11505. Stanton, N. A., & Roberts, A. P. (2017a). Examining social, information, and task
networks in submarine command and control. IEEE Transactions on Human- Machine Systems, 48(3), 252–265. https://doi.org/https://doi.org/10.1109/THMS .2017.2720659 Stanton, N. A., & Roberts, A. P. (2017b). The command team Experimental Testbed (ComTET): Understanding Submarine command team Functionality from a Sociotechnical Perspective. Stanton, N. A., & Roberts, A. P. (2020). Better together? Investigating new control
room configurations and reduced crew size in submarine command and control. Ergonomics, 63(3), 307–323. https://doi.org/10.1080/00140139.2019.1654137 Stanton, N. A., Roberts, A. P., & Fay, D. (2017). Up periscope: understanding submarine command and control teamwork during a simulated return to periscope depth. Cognition, Technology & Work, 19(2), 399–417. Stanton, N. A., Stewart, R., Harris, D., Houghton, R. J., Baber, C., McMaster, R.,
Salmon, P., Hoyle, G., Walker, G., & Young, M. S. (2006). Distributed situa- tion awareness in dynamic systems: theoretical development and application of an ergonomics methodology. Ergonomics, 49(12-13), 1288–1311. Stevenson, A. (2005). Evolution from periscopes to optronic mast systems. Proceedings of the International Conference WARSHIP’05: Naval Submerines 8, London, Royal Institution of Naval Architects, RINA, ISBN: 1-905040-14-8, Paper: P2005-5 Proceedings., Vieira, R. d. S. (2016). Towards a game-based periscope simulator for submarine
officers tactical training Monterey, California: Naval Postgraduate School]. Wu, L., Zhu, Z., Cao, H., & Li, B. (2016). Influence of information overload on operator’s user experience of human–machine interface in LED manufacturing systems. Cognition, Technology & Work, 18, 161–173.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human-Centered Design and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-90-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional