Hui Xue, Tae-eun Kim (2024). Multi-modal data analysis for enhanced nautical skill development. In: Advances in Human Factors of Transportation. AHFE Open Access, vol. 148, pp. 749–761. https://doi.org/10.54941/ahfe1005267
This paper explores the innovative application of multi-modal data analysis for nautical skill development process. By examining the potential of integrating diverse data sources e.g., ship motion, visual, auditory and sensor data, the possibility of obtaining performance insights could be enhanced, and the decision-making processes in complex navigational environments can be made more transparent. The findings from this systematic literature review revealed a significant gap in the current body of knowledge concerning the analysis of multi-modal data in maritime domain. Through a case study on a sailing route involving navigation under a bridge and interactions with two other vessels, we evaluated the potential of a multi-modal data analysis approach to enhance future nautical training. This work aims to catalyse system development and prompt future research endeavours that align with the intersection of multi-modal data analysis and nautical skill advancement.
Developing ship navigation skills frequently employs simulations, which pro- vide a safe and cost-effective method for enhancing nautical capabilities. This training process incorporates the use of ship simulators, sometimes eye trackers, and other wearable technologies to assess situation aware- ness and the competence of learners. Such processes could generate a vast array of information from multiple sources, including commanded actions, ship movement, visual patterns, verbal communication, movements around the simulation console and physical interactions. Although this varied data provides valuable insights and yields more comprehensive and informa- tive perspectives on student performance and learning progress, its utiliza- tion, whether individually or collectively, remains very minimal in practical training processes.
overlook. Giannakos et al. (2019) explored how multi-modal data could enhance personalized learning experiences by adapting teaching methods to student interactions and feedback in digital environments. These studies highlight the broad applicability and potential of multi-modal data analysis to improve educational practices by providing deeper insights into learner behaviors and preferences. Key methods and approaches within multi-modal data analysis include data fusion, cross-modal retrieval, sentiment analysis, and topic modelling techniques (Gandhi et al., 2023). Moreover, leverag- ing advanced deep learning architectures like convolutional neural networks (CNNs) and recurrent neural networks (RNNs) enhances analysis capabil- ities. Visualization techniques aid in comprehending intricate relationships across modalities, while transfer learning and domain adaptation techniques enable knowledge transfer across different data types (Ray et al., 2023). These methods synergize to reveal insights and uncover patterns that would be challenging to discern through analysis of individual modalities alone, advancing our understanding of multi-faceted datasets.
Multi-modal data analysis methods and approaches have the potential to be applied in the context of nautical skill development to improve per- formance assessment and training effectiveness (Kim et al., 2023). During simulation-based maritime training, diverse modalities such as visual (virtual environments), auditory (simulated sounds of the ocean, engine noise), and haptic (force feedback from controls) can be integrated to provide a realistic training experience. Real-time navigation guidance and hazard detection can be facilitated through the integration of GPS data, radar images, and sonar data. Additionally, the analysis of weather forecasts, vessel traffic informa- tion, and engine performance indicators can offer recommendations for route planning, collision avoidance, and emergency response. By analysing data generated from these various modalities, trainers can assess the performance of trainees across different areas and provide targeted feedback for improve- ment. This analytical approach enables seafarers to make more informed decisions during navigation, ultimately leading to safer and more efficient maritime operations.
The primary objective of this study is to review the methods and approaches in existing studies on multi-modal data analysis in the mar- itime domain and to assess the potential of multi-modal data analysis to enhance nautical training, through a case study focusing on a sailing route that includes navigation beneath a bridge and interactions with two other vessels. To this end, we address the following research questions:
1. What are the research trends in multi-modal data analysis in the maritime domain? 2. What methods and approaches are commonly employed in multi-modal data analysis in this field, and what tools are used for data collection? 3. How can multi-modal data analysis contribute to the development and enhancement of nautical skills?
Through this investigation, we aim to identify and analyze the potential benefits and applications of multi-modal data analysis techniques in maritime training and provide a basis for future advancements in practice.
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 751
Systematic Literature Review: Methodology and Analysis Process The initial phase of the Systematic Literature Review (SLR) involves con- ducting a comprehensive search for pertinent academic literature through a methodical process. In this study, we used Web of Science (WOS) database due to its extensive coverage, encompassing over 211 million records com- prising journal articles, conference proceedings, and books (Clarivate, 2023). Aligned with the focus of the study, the Boolean expression initiated with an all-fields search, encompassing “multimodal data” or “multi-modal data” to account for variations in expression. We extracted literature related to “mar- itime” or “nautical skills” related topics. The search criteria were confined to titles, abstracts, and keywords, concentrating on English-language journal articles, proceeding papers and review articles, based on the research focus and precision requirements, resulting in an initial pool of 77 potential records. The detailed Boolean expression is outlined in Table 1. The data collection method and analytical approaches are shown in Figure 1. The parameters of the query are:
1. Date range: all-time years. 2. Publications of type journal articles, proceeding papers and review articles are used. 3. Limited to only English-language. 4. The query was performed on 14th December 2023.
Figure 1: Flow diagram of the research procedure including the identification of studies and data analysis.
STATE OF THE FIELD Existing Application of Multi-Modal Data Analysis in Nautical Training
A total of 76 publications related to nautical science and multi-modal data analysis were identified in the WOS database and are incorporated in this study. The findings of the data analysis are presented through both biblio- metric analysis, which involves network visualization to highlight prolific research contributors, and descriptive content analysis aimed at discerning research trends.
Overlay visualization of co-occurrence mapping (Fig. 2) of all keywords using VOS viewer shows that navigation and localization are the most used keywords and also presents the distribution of the most used keywords by year.
Figure 2: Overlay visualization of co-occurrence of all keywords presents the distribu- tion of the most used keywords by year.
The introduction of digitalization and big-data analysis in recent years has facilitated the development of multi-modal data analysis in this field and the trend is moving towards more data science related keywords (e.g., deep learning, computer vision).
Recognized Advantages and Challenges of Implementing Multi-Modal Data Analysis for Enhanced Nautical Skill Development There are several advantages of multi-modal data analysis for nautical train- ing as recognized in the literature. Firstly, the use of multi-modal analysis offers a comprehensive assessment of the students’ performance than solely relying on a single source of data. Further, it offers an objective perspec- tive compared to subjective judgement from the nautical instructors which could be used as a supplementary source of performance assessment aiding better insights and evaluation. By incorporating multiple data modalities,
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 753
instructors can gain a deeper understanding of student performance and tailor training interventions more effectively to address individual learning needs.
The challenge of implementing multi-modal data analysis for enhanced nautical skill development is primarily attributed to the absence of a stan- dardized framework for establishing a data collection platform. Researchers often need to devise custom data collection frameworks tailored to their specific requirements, encompassing various sensors such as those gather- ing information from ships and human factors. Each sensor device needs calibration for both intrinsic and extrinsic parameters to ensure the preci- sion of data acquisition, which poses challenges in managing the substantial volume of collected data. Further, the analytical process involves not only individual assessment of each sensor’s data but also integration and cross- validation with data from other sensors to uphold accuracy. The complexities are exacerbated by the need of ensuring correct orientation and time synchro- nization across all modalities, while the acquisition equipment itself often incurs expenses. Thus, addressing these challenges is crucial to harnessing the full potential of multi-modal data analysis in advancing nautical skill development.
In addition to research studies, multiple research and innovation projects have investigated how to maximize training effectiveness and situation awareness in nautical settings. i-MASTER project is developing AI-based intelligent learning systems for seafarer training. In BEYOND project (Build- ing an Experience to Yield Optimal Nautical Displays), Fraunhofer CML and Fraunhofer FKIE (CML, 2022) have developed an experimental setup to evaluate maritime Human-Machine Interfaces (HMI) using the Situa- tion Awareness Global Assessment Technique (SAGAT) in simulated ship navigation scenarios.
CASE PRESENTATION: SCENARIO AND LEARNING GOALS Scenario Description
The study was conducted utilizing a full mission simulator bridge equipped with K-sim navigation software from Kongsberg and featured a 360
view. The scenario is derived from one of the training scenarios within the i- MASTER project (i-MASTER, 2023). The designated sailing route involved navigating northbound through the Lillebelt buoyed channel, passing the Øresund Bridge, and continuing towards Gothenburg. The route spans around 6.5 km and requires around 20 minutes to traverse, depending on the speed of the vessel. Upon the start of the exercise, the vessel operated on autopilot with a course set at 040
and a speed of approximately 10 knots. During the exercise, two other vessels were encountered: a southbound ves- sel, identified as the “target” vessel named Sune, positioned at N55
37.37 E012
53.76, maintaining an average speed of 10 knots, and a dredger named Aarts, also southbound, positioned at N55
36.00 E012
51.90, with a speed of 2.5 knots. Navtex message issued was “The dredger requests a mini- mum 70 m berth for safe passage and a maximum of 5 knots for passage speed”. Weather conditions remained stable with clear skies, no current and excellent visibility. The vessel model represents a general cargo ship with a
length between perpendiculars of 132.7 meters and a width of 19.6 meters. The dredger involved represents a heavy lift pipelay barge named Lewek Champion with a length between perpendiculars of 142 meters and a width of 40 meters. All participants were well acquainted with the vessels employed in the simulation.
Learning Objectives and Performance Evaluation The learning objective of the exercise is to execute a safe passage while con- sidering all relevant sources of information. Participants are expected to promptly recognize potential close encounters with other vessels and criti- cal congestion areas, as well as identify navigation lights and adhere to the COLREGs. The COLREG rules that should be considered include rule 9, rule 16, rule 17, rule 13, rule 8, rule 3(g) and rule 18. To evaluate the achieve- ment of these learning objectives, performance was assessed using specific metrics. The following metrics serve as quantitative indicators for evaluating participants’ competence in safely navigating in confined waters.
Proximity between the own ship and the dredger during passage should be as long as possible and more than 70 meters and indicative of safe manoeuvring.
Speed of the own ship when passing by the dredger should be as slow as possible and less than 5 knots signifies adequate adherence to safety protocols.
In this study, three participants with varying levels of sea experience navigated the scenario. Participant 1 has over 10 years of sea experience, Participant 2 has less than 5 years, and Participant 3 is a novice. Situational awareness (SA) and decision-making skills emerged as pivotal factors influ- encing performance effectiveness. The challenge lies in the own ship’s need to make critical decisions based on limited information regarding the restric- tion of meeting the dredger. The scenario presents a situation where clarity may not be fully attained, particularly regarding the own ship’s encounter with the target ship and the dredger. The timing and nature of decisions, such as adjusting speed and course, significantly impact the outcome of the encounter with the dredger. The objective is to navigate in a manner that avoids meeting vessels under the bridge and simultaneously avoids encounter- ing multiple vessels. This necessitates ensuring adequate manoeuvring space while maintaining sufficient speed to safely pass the dredger.
To analyze the participants’ performance, the distances between the own ship and both the target ship and the dredger are calculated using the coor- dinates provided in the log data. This computation is carried out using the formula:
sin latA sin latB + cos
latA cos latB cos lonA −lonB
D = 3440.1 cos−1 where D is the distance in meters, latA is the latitude of point A expressed in radians, latB is the latitude of point B expressed in radians, lonA is the longitude of point A expressed in radians, lonB is the longitude of point B
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 755
expressed in radians, 3440.1 is the radius of the earth in nautical miles (NM) (Xue et al., 2024).
To analyze the participants’ decision-making process, a factor relating the speed of the own ship to the sum of the distances is proposed. This factor is calculated using the following formula:
Equation reproduced from the original PDF.
(1) Dt + Dd where F represents the factor, V denotes the speed of the own ship in knots, Dt signifies the distance between the own ship and the target ship, and Dd stands for the distance between the own ship and the dredger.
To assess the participants’ SA skills, the analysis of eye-tracking data proves invaluable. Understanding how participants allocate their visual attention between the radar display and the external surroundings provides insights. This analysis sheds light on the effectiveness of participants’ SA strategies and their ability to gather and process relevant information during critical decision-making moments.
Data Analysis Results and Discussion Among the three participants, two chose to pass the dredger from the star- board side, while the other one opted for the port side passage. This choice is depicted in Figure 3 illustrating the distances between the own ship and the dredger, as well as the respective speeds during passage. The participant pass- ing from the port side had the shortest distance. In Figure 4, which presents the plot of distances, it is evident that two participants encountered the Tar- get ship Sune after encountering the dredger Aarts. The figure also provides insights into the timing of these encounters. Figure 5 displays the factor of the speed order to the sum of distances, revealing that all participants initially maintained lower speeds. Participant 1, who has the most sea experience, promptly reduced speed to a very slow pace at the outset and sustained this pace until encountering the Target ship Sune. Conversely, the remaining two participants did not reduce the speed low enough at the beginning of the voy- age therefore they had to endeavour to lower their speeds when getting closer to the dredger sufficiently to ensure a safe distance when approaching other vessels. Participant 1 accelerated approximately 16 minutes after encoun- tering the Target ship Sune, experienced a minor speed reduction around the 19-minute mark during this encounter, and subsequently increased speed around the 23-minute mark upon encountering the dredger Aarts. This deci- sion made him have too high speed when passing the dredger which is not recommended. Participant 2 made several attempts to reduce speed within the first 7 minutes of the journey, further reducing speed around the 10-minute mark, before increasing speed around 19 minutes after encountering the Tar- get ship Sune, maintaining this pace thereafter. Participant 3 reduced speed twice, initially around the 4-minute mark and again at the 11-minute mark of the voyage.
In this study, the key factors influencing outcomes and performance include adjustments in speed and course, as well as decisions regarding the
passage of the target ship and dredger. The three outcomes observed among the participants suggest that participants’ levels of experience may contribute to these variations. Experience is known to impact SA and decision-making skills, ultimately influencing the decisions made and subsequent performance grades (Chauvin et al., 2009). However, the students can perform better than experts in the simulation training because they are more familiar with the ship model and used to be asked to strictly follow the instructions. Par- ticipant 3, representing the least sea-experienced participant, had a greater ability to manoeuvre when encountering the dredger, indicative of the influ- ence of training on simulation outcomes. Figure 6 screenshots extracted from the simulation scenario depict various scenarios as the own ship nears the bridge and encounters the dredger. Both Participant 2 and 3 encountered sim- ilar situations, necessitating a more substantial reduction in speed to avoid meeting the two vessels simultaneously. To understand how experienced seafarers navigate complex situations, it is helpful to observe their decision- making processes. Improving this observation with auxiliary data analysis can provide valuable insights. For instance, analyzing eye-tracking data using heatmaps can reveal how individuals allocate their visual attention. Combining this with simulation log data offers a comprehensive understand- ing, aiding in the development of learning plans and the design of training scenarios for students.
Figure 3: Plot of the sailing route for the 3 participants, the target ship Sune and the dredger Aarts for 25 minutes.Table 1. Performance of the three participants.
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 757
Figure 4: Plot of the distance between the own ship to the target ship Sune and dredger Aarts for the three participants.Figure 5: Factor of the speed order to the sum of the distances from the own ship to the target ship Sune and dredger Aarts for the three participants.Figure 6: Comparison of the various decision-making processes and their respec- tive outcomes across the three participants. Figures (a), (b), and (c) depict the own ship nearing the bridge across the three participants, whereas figures (d), (e), and (f) illustrate encounters with the dredger across the same participants.
APPLICATION OF MULTI-MODAL DATA ANALYSIS APPROACH The case analysed above mainly used the ship log data as the data input for performance assessment. The simulation log data offers performance metrics e.g., speed, course, location data, and inter-ship distance, enabling a quantita- tive evaluation of adherence to navigational principles and safety protocols. This only provides one perspective on the performance, because log data can be a “bridge” connected to other types of data giving a deep understanding of the decision-making results from the trainee.
The acquisition of diverse data types from varied sources has the potential to enrich this analysis. Human factor-related data, including auditory data, eye-tracking data, text data, and biosignal data, are gathered to elucidate learners’ performance. Auditory data, comprising communication record- ings, facilitates insights into verbal interactions among learners, instructors, or team members within simulated scenarios. Eye-tracking data reveals gaze patterns, furnishing essential information regarding situational awareness and decision-making mechanisms. biosignal data, comprising metrics such as heart rate, body temperature, skin conductance response (SCR), and electroencephalogram (EEG) data, indirectly assess sympathetic autonomic activity linked with emotion and attention, thereby providing a comprehen- sive understanding of learners’ performance. The schematic representation of the multi-modal data framework is depicted in Figure 7 for clarity.
Figure 7: Simplified schematic representation of the multi-modal data framework in a simulation-based training.
The multi-modal data analysis approach represents an innovative strategy for advancing nautical skills within simulation-based maritime training. A fundamental aspect of its novelty is its holistic framework, as demonstrated in the presented case study, which integrates a variety of human factors alongside traditional performance metrics. Specifically, in the case study, the analysis of simulation log data contributes to performance assessment by revealing the distance between the own ship and the dredger, serving as a key criterion. The proposed formula of the factor of the speed order to the sum of the distance provides a novel method to analyse navigation data about speed
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 759
adjustments, providing a basis for analysing dynamic decisions. By plotting this factor, distinct patterns of various decision-making outcomes become discernible, which can be used as a feature when using machine learning algorithms in large-scale data analysis. Incorporating auditory data analysis, such as evaluating communication efficiency, further elucidates how com- munication effectiveness influences decision-making dynamics. Moreover, by integrating biosignal data analysis results, valuable insights into learners’ physiological responses to training stimuli are gained. Monitoring physio- logical indicators such as heart rate and skin conductance response allows trainers to detect signs of stress or fatigue that may jeopardize safety. This comprehensive approach, encompassing multiple modalities of data analy- sis, holds promise for optimizing maritime training effectiveness and safety protocols.
The framework enables real-time analysis of multi-modal data, empow- ering trainers to provide immediate personalized feedback to learners. This feedback loop, based on individual performance metrics and human factors data, supports adaptive training strategies tailored to address specific skill gaps or challenges as they emerge, thereby expediting the learning process and enhancing skill acquisition efficiency.
By leveraging technologies and methodologies from disciplines such as data science, human-computer interaction, and cognitive psychology, this multi-modal data analysis could represent a shift in maritime education and training. It offers a more comprehensive and data-driven approach to nauti- cal skills development, with implications for enhancing safety, efficiency, and effectiveness in maritime operations.
CONCLUSION
This study investigated the use of multi-modal data analysis to enhance nau- tical skills development. Through a systematic review and case study, it underscored the potential of various data types to improve maritime training and performance evaluation. A notable knowledge gap persists in effectively utilizing multiple data sources to interpret and enhance nautical performance, which underscores the need for continued research. To advance this field, the paper proposed a multi-modal data analysis approach, and we hope this will address existing gaps and stimulate further advancements in multi-modal data analysis for nautical skill development.
ACKNOWLEDGMENT
The authors would like to acknowledge the participants from UiT-The Arctic University of Norway for their participation and consent to use the simulator log data in this research.
REFERENCES
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238.
Chauvin, C., Clostermann, J.-P., Hoc, J.-M., 2009. Impact of training programs on decision-making and situation awareness of trainee watch officers. Saf. Sci., Research in Ergonomic Psychology in the Transportation Field in France 47, 1222–1231. https://doi.org/10.1016/j.ssci.2009.03.008 Clarivate, 2023. LibGuides: Resources for Librarians: Web of Science Coverage
Details [WWW Document]. URL https://clarivate.libguides.com/librarianresourc es/coverage (accessed 12.14.23). CML, F. (2022). Retrieved from https://www.cml.fraunhofer.de/en/research-projects /BEYOND.html (accessed 04.29.23). Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., Lim, W. M., 2021. How to
conduct a bibliometric analysis: An overview and guidelines. J. Bus. Res. 133, 285–296. https://doi.org/10.1016/j.jbusres.2021.04.070 Endsley, M. R. (1995). Measurement of Situation Awareness in Dynamic Systems.
Human factors(1), 65–84. Endsley, M. R. (2000). Direct measurement of situation awareness: Validity and use of SAGAT. In M. R. Endsley, & D. J. Garland (Eds.), Situation awareness analysis and measurement (pp. 147–174). Mahwah, NJ: Lawrence Erlbaum. Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., Hussain, A., 2023. Multimodal
sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Inf. Fusion 91, 424–444. https://doi.org/10.1016/j.inffus.2022.09.025 Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019).
Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108–119. Grundmann, R., & Kang, D. (2020). Global linked simulations - A key to the evaluation of future transport concepts and navigation. IOP Conference Series: Materials Science and Engineering. Haavardtun, P., Nyairo, F., Bustgaard, M., Munim, Z. H., Thorvaldsen, H., (2023) i-
MASTER Diliverable. D3.1 Consolidation of learning resources for the developed train- ing scenario. i-MASTER (2023). D3.1 Integrating Adaptive Learning in Maritime Simulator-Based Education and Training with Intelligent Learning System. IEEE. (1996). IEEE Standard for Distributed Interactive Simulation - Application
Protocols. IEEE Std 1278.1–1995. Kim, T. E., Munim, Z. H., & Schramm, H. J. (2023). Exploring alternative performance evaluation method in nautical simulations. Training, Education, and Learning Sciences, 109(109). Melet, A., Teatini, P., Le Cozannet, G., Jamet, C., Conversi, A., Benveniste, J.,
Almar, R., 2020. Earth Observations for Monitoring Marine Coastal Hazards and Their Drivers. Surv. Geophys. 41, 1489–1534. https://doi.org/10.1007/s10712- 020-09594-5 Moradi, M. H., Brutsche, M., Wenig, M., Wagner, U., Koch, T., 2022. Marine route optimization using reinforcement learning approach to reduce fuel consumption and consequently minimize CO2 emissions. Ocean Eng. 259, 111882. https://do i.org/10.1016/j.oceaneng.2022.111882 Ray, A., Kolekar, M. H., Balasubramanian, R., Hafiane, A., 2023. Transfer Learning
Enhanced Vision-based Human Activity Recognition: A Decade-long Analysis. Int. J. Inf. Manag. Data Insights 3, 100142. https://doi.org/10.1016/j.jjimei.2022. 100142
Multi-Modal Data Analysis for Enhanced Nautical Skill Development 761
Ray, C., Goralski, R., Claramunt, C., Gold, C., 2011. Real-Time 3D Monitoring of Marine Navigation, in: Popovich, V. V., Claramunt, C., Devogele, T., Schrenk, M., Korolenko, K. (Eds.), Information Fusion and Geographic Information Systems: Towards the Digital Ocean. Springer, Berlin, Heidelberg, pp. 161–175. https://do i.org/10.1007/978-3-642-19766-6_14 Vachon, B., LeBlanc, J., 2011. Effectiveness of past and current critical incident
analysis on reflective learning and practice change. Med. Educ. 45, 894–904. https://doi.org/10.1111/j.1365-2923.2011.04042.x Xue, H., Haugseggen, Ø., Røds, J.-F., Batalden, B.-M., Prasad, D. K., 2024.
Assessment of stress levels based on biosignal during the simulator-based mar- itime navigation training and its impact on sailing route reliability. Transp. Res. Interdiscip. Perspect. 24, 101047. https://doi.org/10.1016/j.trip.2024.101047
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Human Factors of Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-24-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional