Jennifer Smith, Reza Zeinali Torbati, Randy Billard, Bruno Emond, Brian Veitch (2023). Developing an Adaptive Instructional System for Simulation-Based Lifeboat Training Using Instructor Feedback. In: Human Factors in Transportation. AHFE Open Access, vol. 95, pp. 675–685. https://doi.org/10.54941/ahfe1003853
Integrating adaptive instructional systems (AIS) into simulation-based lifeboat training creates the opportunity to customize instruction and practice to meet the individual needs of trainees. Embedding an AIS into lifeboat simulation could make training more accessible for on-demand and remote learning applications such as practicing lifeboat coxswain duties while onboard vessels. An AIS evaluates performance, tailors instruction, and delivers practice exercises using four key models: the domain knowledge, the student/learner, the instructional framework, and the user interface. This paper applies a human-centred approach to developing the instructional model for an AIS to ensure the adaptive simulation-based training is attuned to the learning strategies of seafarers. Specifically, we gathered information from users of the lifeboat simulators by conducting semi-structured interviews virtually with three lifeboat instructors. The instructors were asked questions about how they provided instruction, assessment, and feedback and to comment on video examples of lifeboat operations in a simulator. The videos depicted a trainee launching the lifeboat, clearing away from an offshore installation, and manoeuvring the lifeboat in the simulator. The instructors explained how they use simulation-based training to help trainees practice lifeboat operations and build the trainees’ confidence and leadership skills as coxswains. Information from the interviews was used to develop a conceptual instructional model for an AIS. Future work will integrate the instructional model into the AIS and test its functionality for providing adequate instruction, deliberate practice opportunities, and corrective feedback to trainee lifeboat coxswains.
Seafarers are required to complete multi-day lifeboat training courses at onsh- ore facilities and perform regularly scheduled abandonment drills onboard to maintain coxswain skills in compliance with the International Maritime Organization’s (IMO) International Convention on Standards of Training, Certification and Watchkeeping (STCW) (IMO, 2009; IMO, 2017a) and
similar national regulations (Transport Canada, 2007). However, harsh wea- ther conditions and safety concerns limit how often abandonment drills can be conducted using lifeboats (IMO, 2017b). Simulation-based lifeboat training is increasingly used to supplement conventional shore-based trai- ning and provide seafarers with opportunities to practice safety critical skills. Advances in data mining, machine learning, and adaptive instructio- nal systems (AIS) are transforming simulation-based training; enabling the technology to customize instruction to meet the individual needs of trai- nees. Additionally, the integration of an AIS into simulation technology has the potential to make training more accessible for on-demand and remote learning applications such as practicing lifeboat operations while onboard vessels.
This research applies a human-centred approach to AIS development to ensure the technology is adequately assessing and responding to the vari- ety of learning strategies of seafarers. Here we present a pilot study to gather information from instructors to inform the instructional model of an AIS for simulation-based lifeboat training. Semi-structured interviews were administered virtually with three lifeboat instructors, representing dif- ferent training organizations. The interviewers asked questions about when and how instructors provided guidance, assessment, and feedback during simulation exercises to train lifeboat launching and slow speed manoeuvring skills. The instructors were also asked to comment on four video examples of lifeboat operations in a simulator. This information was used to deve- lop a conceptual model of the instructional framework for the AIS. This paper presents the instructions and corrective feedback approaches that the instructors use to address skill gaps and how these approaches can be inte- grated into the instructional model of an AIS for simulation-based lifeboat training.
AIS DEVELOPMENT FOR LIFEBOAT SIMULATION TRAINING Conceptually, an AIS manages the instruction and evaluation of the learning process by evaluating performance and individually tailoring instruction, delivering practice exercises, and providing feedback to meet the needs of the learner (Sottilare, 2018). The AIS employs four interacting models (depicted in Figure 1): the domain knowledge, the student/learner, the instructional framework, and the user interface. As explained by Pavlik et al. (2013), the domain model contains the relevant knowledge and skill sets for the context. The student model consists of different states of the learner. The instructional model compares the student model to the domain model and applies tutoring strategies to close the gap. Finally, the user interface interprets the inputs and provides outputs based on the interactions between the three other models (Pavlik et al., 2013).
The AIS model can inform how to collect and integrate domain kno- wledge, student performance data, and instructional/learning frameworks to develop adaptive instruction for simulation-based training. Ideally, to develop each of the AIS models, information should be gathered from all
Developing an Adaptive Instructional System 677
Figure 1: Adaptive instructional system (adapted from Goenka et al., 2021; Nwana, 1990).
users of the lifeboat simulator (e.g., students entering the field, experie- nced seafarers receiving refresher training, and the instructors who provide simulation-based training).
The domain model can be populated using knowledge gathered from interviews with domain experts and industry standards for competency requi- rements. For example, the STCW proficiency requirements in survival craft and rescue boats, other than fast rescue boats, as set out in Table A-VI/2-1 (IMO, 2017a) and the marine emergency duties, as described in TP4957E (Transport Canada, 2007).
The student/learner model can be informed by data collected from empiri- cal studies in simulation-based lifeboat training (Magee et al., 2016; Billard et al., 2020a) and statistical models, such as Bayesian networks (BN) (Billard et al., 2020b). Specifically, as the learner completes tasks in the scenario, evi- dence collected in the simulator is used to model and evaluate the learner’s performance using an evidence-centered design approach. Data collected through simulation-based assessments (both in the current scenario as well as long-term collection of performance data throughout the training) inform the student/learner model and provide insights on the learner’s overall performa- nce which can be used to direct the appropriate training pathway (Mislevy, 2004). The instructional model can be informed by learning theories to recommend practice and feedback such as mastery learning (Bloom, 1971; Guskey, 2007) and challenge point framework (Guadagnoli & Lee, 2004); as well as transformative adult learning theories related to simulation-based training (Clapper, 2010). Further, interviews with subject matter experts can identify important factors for the delivery and assessment of lifeboat training to implement into the instructional model.
This work is part of a larger study aimed at developing an AIS for simulation-based lifeboat training (Emond et al., 2022). This paper presents a pilot study to gather information to develop the instructional model for an AIS. Specifically, semi-structured interviews with lifeboat instructors to gain insights into their instructional practices and approaches for providing
corrective feedback in simulation-based training. This information was used to conceptualize an early-stage instructional model. The co-development of the student/learner model using simulation data and Bayesian networks is described in Zeinali-Torbati et al. (2023).
METHODS
Semi-structured interviews were conducted virtually to gather information on how lifeboat instructors provide instruction and feedback during simulation- based training to develop the instructional model of an AIS. The interviews consisted of two parts. First, the instructors were asked questions on their instructional approaches. Then, they were asked to watch and comment on video-recorded examples of lifeboat training in a simulator.
Ethics approval for this pilot study was granted by the Memorial University’s Interdisciplinary Committee on Ethics in Human Research (ICEHR) following the Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans (ICEHR number 20222445-MI).
Participants Three lifeboat instructors (denoted LI01, LI02, LI03) participated in this pilot study, each representing different training organizations. All partici- pants were seafarers with prior careers at sea. Each had experience as a lifeboat instructor and taught lifeboat training using both live-equipment and simulators. The instructors had a range of experience teaching with live lifeboat equipment (1.5 to 12 years) and simulator-based training (0.5 to 8 years).
Interview Format The interview questions asked instructors about their instructional appro- aches and how they characterize performance differences between novices and experienced coxswains. Specifically, the interview elicited information concerning when and how the instructor’s provided guidance, assessment, and feedback during simulation exercises to train lifeboat launching and slow speed manoeuvring skills.
Commenting on Video Examples The instructors commented on four video examples of lifeboat training ope- rations completed by four trainees in a simulator, provided by Virtual Marine. The videos depicted an individual launching the lifeboat in the simulator, cle- aring away from an offshore petroleum installation, and manoeuvring the lifeboat in a series of subtasks. For example, Figure 2 depicts one of the subtasks the instructors reviewed in the videos in which the trainee is mano- euvring the lifeboat to approach and stop at an offshore supply vessel. Each video represented different levels of performance in two situations: a basic lifeboat drill and emergency launch condition. For the basic lifeboat drill, instructors watched two trainee performance examples (denoted D1 and D2). The basic scenario was designed for calm weather conditions, clear visibility, a Beaufort 3 sea state (large wavelets), with no current and light wind of
Developing an Adaptive Instructional System 679
Figure 2: Screen capture of the video configuration from the virtual marine lifeboat simulator showing an approach to the recovery vessel.
5 knots. The trainees performed the lifeboat drill from an offshore installa- tion. Specifically, the trainees launched the lifeboat and cleared away from the platform. To demonstrate their navigation skills, the basic scenario also involved a series of slow speed manoeuvring tasks. The trainees were requi- red to steer by a compass heading towards a smoke float, recover a person in the water, retrieve and tow a life raft, come alongside a recovery vessel for passenger transfer, and practice deploying the sea anchor.
For the emergency scenario involving the abandonment of the installa- tion, the instructors also watched two trainee performance examples (deno- ted E1 and E2). The emergency scenario was designed for moderate weather conditions, reduced visibility (e.g., 1000ft), a Beaufort 4 sea state (small wavelets), with no current and winds of 13 knots. The trainees performed the emergency lifeboat launch from the same offshore installation. Specifically, the trainees launched the lifeboat and cleared away from the platform. The trainees also practiced a series of slow speed manoeuvring tasks that are cri- tical operations for lifesaving during emergencies. The trainees were required to recover two persons in the water, approach a life raft for inspection, and to steer by a compass heading towards another offshore platform as indicated by a helicopter search light. The scenario ended when the trainee manoeuvred the lifeboat into the safe zone of the second platform.
For each video example, the instructors scored the trainee performance and provided a commentary on the videos. The instructors commented on how they would provide guidance and feedback during the exercise (i.e., speaking to the temporal, aggregation, modality, performance, and result, Cockburn et al., 2015). Following the interview, the transcripts were analyzed using NVivo software and this information was used to develop an early-stage conceptual model of the instruction framework for the AIS.
FINDINGS AND DISCUSSION This section describes some of the insights gained from the interviews with the lifeboat instructors and begins to conceptualize how this information could be used to develop an instructional model.
Insights From Lifeboat Instructors This pilot study captured how instructors assessed performance, ranked the skill level of trainees, and provided corrective feedback. The instructors provided a performance scored using 5-point scale, where a score of one represented not very successful, three was somewhat successful, and five represented very successful performance. The instructors’ overall performa- nce scoring and skill level rankings for the four video examples are provided in Tables 1 and 2.
Throughout the study, the instructors explained how they used simulation- based training to help trainees practice necessary technical skills of lifeboat operations, to build the trainees’ confidence and leadership skills as coxsw- ains, and to discuss other ‘what if’ considerations when performing coxswain duties. The instructors commented on the correct procedures for each task in the launch and manoeuvring exercises in the simulator. They contextuali- zed the reasons why the launch protocols and the manoeuvring approaches were important to consider given how much the environmental conditions could change and impact the trainee’s ability to control the lifeboat. They also emphasized the importance of manoeuvring at a reduced speed and drew particular attention to the real-world considerations when planning different approaches to people in the water and other vessels.
The instructors also described factors (or high-level skills) that they inclu- ded in their assessments, such as control of the vessel (i.e., technical skills and duties of the helm during the launch and manoeuvring of the lifeboat), situation awareness (i.e., real-world implications of the tasks and the associ- ated risks and safety issues), and command (i.e., coxswain leadership duties, decisions, communications, and the individual’s confidence in the tasks).
Conceptual Instructional Model The interviews revealed several themes related to the complexity of the lifeboat coxswain duties and that performance assessment (diagnosing the
Table 1. Instructor scoring of the overall performance.
Table available in the original PDF paper.
Video Examples Instructor Rating Score of Overall Performance LI01 LI02 LI03
D1 Drill 2 4 2 D2 -* 3 4.5 E1 Emergency 2 3 3.5 E2 2 3 1.5 *LI01 did not provide a score for example D2.
Table 2. Instructor rankings of the trainee example skill levels.
Table available in the original PDF paper.
Video Examples Instructor Ranking Example Skill Level LI01 LI02 LI03
D1 Drill Novice to Some Experience Some Experience Novice D2 Novice Novice Experienced E1 Emergency Novice Novice Experienced E2 Novice Novice Novice to Some Experience
Developing an Adaptive Instructional System 681
student performance and classifying their skill level) goes beyond demonstra- ting a good understanding of how to launch and steer the lifeboat. From the interviews it was observed that there are factors (or assumptions) embedded into how instructors assess performance in simulation-based lifeboat trai- ning. These factors influence how instructors score the trainee’s performance in each task and their overall judgment of the trainee’s skill level. The factors (or high-level skills) involved in assessing the lifeboat coxswain duties include whether the individual can demonstrate i) control of the vessel, ii) awareness of the surroundings and associated risks/consequences of their actions, and iii) command of the vessel including decision-making, communications, and confidence.
Figure 3 shows how the factors build from the fundamental skills of ope- rating and controlling the vessel (e.g., the launch procedures and operations of the vessel for slow speed manoeuvring tasks) to the more complex skills of commanding the vessel (e.g., thorough decision-making and communications with the crew, passengers onboard, and the emergency response personnel coordinating the installation abandonment; as well as the overall proficiency in the integration of the control, awareness, and command).
This information can be used to develop a conceptual instructor model for the assessment of performance and classification of skill levels. Different modelling approaches can be taken; we use decision trees here to show the interconnectivity of these factors (control, situation awareness, and com- mand) and the influence they can have on the student’s success in the task, the scenario, and the overall training session or program. An early- stage representation is shown in Figure 4. The instructor model would collect the diagnostics from the student/learner model (e.g., the rubric score of the student performance in each task, in this case, a simple yes or no) and use that to classify the skill level in the three factors. Depen- ding on the skill-level classification, different learning pathways would be recommended.
From an instructor model perspective, the decision trees may not neces- sarily follow the linear path or the order and combination in Figure 4. Due to individual differences, trainee performance in a given scenario could
Figure 3: Factors that are included in instructor assessment and feedback.Figure 4: Simplified decision tree illustrating instructor classification of skill-level.Table 3. Proposed learning pathways for each factor.
Table available in the original PDF paper.
Factor Skill Instruction Practice Pathway Command Demonstrate proficiency and confidence in decisions and communications.
Offer solution sharing and ‘what-if’ reminders (e.g., ask the trainee to reflect on what might influence their control, awareness, and decisions).
Provide opportunities for deliberate practice to integrate control, situation awareness and command practice (e.g., foster student intentions and build confidence). Situation Awareness
Demonstrate consciousness of surroundings and possible safety issues.
Contextualize the real-world implications of safe/ unsafe behaviours and emphasize the situational aspects that must be considered (i.e., beyond mechanics of operating vessel).
Recommend a variety of different practice scenarios by employing the challenge point framework (e.g., scenarios specific to the trainee’s skill level) and varying the task complexity and environmental conditions to provide enough challenge to produce engagement and effort for trainee learning and skill improvement. Control Demonstrate understanding of launch procedures and vessel operations.
Assign task specific practice (e.g., maintain safe speed, steer a heading, and safely approach a person in the water, small boats, and recovery vessels).
Scaffold or demonstrate how to do the task correctly and explain the consequences of errors (e.g., storytelling to explain plausible tangle/catch points, and risks of physical injury).
demonstrate strengths in control and command, but weaknesses in aware- ness. In this case, the trainee would need to practice the deficit skill (i.e., improve their awareness of their surroundings and possible safety issues).
Developing an Adaptive Instructional System 683
The instructor model would need to facilitate different combinations of trainee strengths and weaknesses for each factor and adapt the instruction and practice pathways based on the trainee’s score in each of these factors. Table 3 outlines the corresponding instruction (in the form of guidance and feedback) and assigned practice format for each factor.
CONCLUSION AND FUTURE WORK A human-centred approach to adaptive instructional systems (AIS) develo- pment of simulation-based lifeboat training is important to produce adaptive training that is informed by the learning strategies of seafarers (i.e., incor- porating the insights from students entering the field, experienced seafarers receiving refresher training, and the instructors that provide simulation-based training).
Our focus in this pilot study was on gathering insights from simulator instructors to inform the development of an instructional model for lifeboat training. Overall, the interviews revealed that the instructors’ teaching appro- aches and corrective feedback often go beyond teaching the technical skills of lifeboat operations. The instructors explained how they used simulation- based training to help trainees learn non-technical skills that are essential to the real-world command of a lifeboat (i.e., leadership and confidence as a coxswain). Specifically, three factors (or high-level skills) were emphasized in the instructors’ performance assessment and comments during the video examples (e.g., overall control of the vessel, awareness of the surroundings, and command of the vessel). We discussed how these factors could be inte- grated into a conceptual instructional model for an AIS in simulation-based lifeboat training. The early-stage conceptual model in this paper will be vali- dated in follow-on research studies. Future work will integrate the conceptual instructor model into the AIS and test its functionality for providing adequ- ate instruction, deliberate practice opportunities, and corrective feedback to lifeboat coxswain trainees.
The overall implication of this conceptual model is that it is informed by the feedback of lifeboat instructors. Embedding these lessons into an automated form of simulation-based training could help with the mainte- nance of essential skills for emergencies. Further, the methodologies used in this pilot study are not specific to the context of lifeboat operations or simulation-based training. Developing an AIS by eliciting the knowledge from domain experts, like lifeboat instructors, brings us closer to a simulation tool that can automate key aspects of the instructor; thereby reducing instructor workload and potentially making simulation technology more equipped to provide meaningful instruction and practice in remote applications such as onboard vessels.
ACKNOWLEDGMENT
This project was supported in part by collaborative research funding from the National Research Council of Canada’s Artificial Intelligence for Logistics
Program. The authors also acknowledge with gratitude the participation of the instructors in this pilot study.
REFERENCES
Billard, R., Musharraf, M., Veitch, B., & Smith, J. (2020b). Using Bayesian methods and simulator data to model lifeboat coxswain performance. WMU Journal of Maritime Affairs, 19, 295–312. Billard, R., Smith, J., & Veitch, B. (2020a). Assessing lifeboat coxswain training
alternatives using a simulator. Journal of Navigation, 73 (2), 455–470. Bloom, B. S (1971). Mastery Learning. In J. H. Block (Ed.), Mastery Learning: Theory and practice (pp.47–63). New York: Holt, Rinehart & Winston. Clapper, T. (2010). Beyond Knowles: What those conducting simulation need to
know about adult learning theory. Clinical Simulation in Nursing, 6(1), e7–e14. Cockburn, A., Gutwin, C., Scarr, J., & Malacria, S. (2015). Supporting novice to expert transitions in user interfaces. ACM Computing Surveys, 47(2), 1–36 (01 2015). Emond, B., Smith, J., Musharraf, M., Zeinali-Torbati, R., Billard, R., Barnes, J.,
and Veitch, B. (2022). Development of AIS Using Simulated Learners, Bayesian Networks and Knowledge Elicitation Methods. In: Sottilare, R. A., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2022. Lecture Notes in Computer Science, 13332. Springer, Cham. Retrieved from: https://doi.org/10.1007/978-3- 031-05887-5_11. Goenka, N., Choudhury, T., & Ahuja, N. J. (2021). Enhancing number sense of dyscalculia learners by pedagogical agents: A review. Emerging Technologies in Data Mining and Information Security, 1286, 827–839. Guadagnoli, M. A., & Lee, T. D. (2004). Challenge point: a framework for conce-
ptualizing the effects of various practice conditions in motor learning. Journal of Motor Behavior, 36(2), 212–224. Guskey, T. R. (2007). Closing achievement gaps: revisiting Benjamin S. Bloom’s “Learning for Mastery”. Journal of Advanced Academics, 19(1), 8–31. IMO. (2009). Measure to prevent accidents with lifeboats MSC.1/Circ.1206/Rev.1,
International Maritime Organization, London, UK. IMO. (2017a). International Convention on Standards of Training, Certification and Watchkeeping for Seafarers (STCW) 1978, as amended in 1995/2010. Internatio- nal Maritime Organization, London, UK. IMO. (2017b). Guidelines on safety during abandon ship drills using lifeboats,
MSC.1/Circ.1578, International Maritime Organization, London, UK. Magee, L., Smith, J., Billard, R., & Patterson, A. (2016). Simulator training for lifeboat manoeuvres. Interservice/Industry Training, Simulation & Education I/ITSEC Conference, 16030. Mislevy, R. J., Almond, R. G., & Lukas, J. (2004). A brief introduction to evidence-
centered design. CSE Technical Report. Los Angeles: The National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Retrieved from https://www.cse.ucla.edu/products/reports/r632.pdf. Nwana, H. S. (1990). Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4(4), 251–277. Pavlik, P., Brawner, K., Olney, A., & Mitrovic, A. (2013). A review of learner models
used in intelligent tutoring systems. In Design Recommendations for Intelligent Tutoring Systems (Volume 1 - Learner Modeling), Orlando, FL, United States. Sottilare, R. (2018). A comprehensive review of design goals and emerging solutions for adaptive instructional systems. Tech., Inst., Cognition and Learning, 11, 5–38.
Developing an Adaptive Instructional System 685
Transport Canada. (2007). Marine Emergency Duties Training Courses. TP4957E.
Retrieved from Transport Canada Marine Safety website: https://tc.canada.ca/sit es/default/files/migrated/tp4957e.pdf. Zeinali-Torbati, R., Smith, J., Billard, R., Emond, B., Barnes, J., & Veitch, B. (2023, July) Towards application of adaptive instructional systems in lifeboat training simulations using Bayesian networks. Accepted for presentation at the Human Computer Interaction International (HCII) 2023 Conference, July 23-28, 2023.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-71-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional