Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Maritime Simulator Training: Eye-Trackers to Improve Training Experience

1 Department of Systems and Interface Design Institute for Energy technology Halden, 1777, Norway ↗ ↗
2 Department of Maritime Technology and Operations, Aalesund University College, Aalesund, 6025, Norway ↗
Publication year
Digital Object Identifier10.54941/ahfe100724

Abstract

The purpose of the present study was to explore the utility of eye tracking technology as a feedback tool in maritime simulator training to assess the trainee performance and to provide more valuable feedback for enhancing trainee-trainer experience. In maritime domain, advanced simulators are used to prepare students for extremely demanding and safety critical operations. However, very little study has been done to develop tools to improve learning as well as measure the learning outcomes. Current training assessment methods are mostly subjective and mainly rely on instructor’s verbal feedback. This paper researches new training method based on eye tracking technology for simulator training, and discusses their application to offshore maritime training. Dynamic positioning (DP) training has been chosen for this study since it is a mandatory system to have installed onboard all modern offshore vessels. A study was conducted with 10 first year nautical students to evaluate this new training method. An assessment checklist for the training outcome was developed based on best practices compliant with NWEA guidelines and used to quantify the student’s performance and to contrast the effectiveness of the new training method with the older one. Results from the study suggest that instructors were able to follow students more closely and were 43% more accurate in assessing their performance in simulator when eye trackers were used. Also using eye tracker, instructors managed to point out some bad practices of students such as too much attention on DP GUIs, which was otherwise not possible to monitor.

INTRODUCTION

Ships are very complex machines, which have several intricate systems combined. The people operating ships from the ship bridges are totally responsible for the safety of the ship and rest of the crew. The bridge crew has to deal with a cluster of displays, consoles and alarms to operate the ship safely. This requires a high level of competence and vigilance from the crew. Human error accounts for approximately 80% of the accidents at the sea. Operator error is the single most important cause of major loss of position incidents in dynamic positioning vessels (Tjallema et al., 2007). Wagenaar and Groenewag (1987) listed that 35% of the accidents were due to improper training and 46% due to bad habits, which could most likely be influenced by procedural training. So training plays a crucial role in maritime accident prevention.

Simulators are standard fixtures for maritime training since 1970s. Even though they are no replacements for real life experience, their capability to create navigational environment and ability to repeat, make them a vital tool for training professional mariners. One general complaint on the younger generation officers is that their focus is often tunneled into the information displays and eventually they fail to look outside environment when required. Results from a recent eye tracking study conducted by Muczyński et al. (2013) confirm the above. In that study inexperienced officers spent more time looking at the controls, radar and displays than the experienced officers (Muczyński et al., 2013). Sometimes this can lead to a potentially hazardous situation. In aviation industry the FAA and other organizations have systematic visual search scan recommendation to pilots for air traffic. Unfortunately there are no such recommendations and practices available for ship officers to follow in the maritime industry. So it is interesting to study the visual search behaviors of experienced & novice navigators and try to use it in training to advise on the ‘rights’ & ‘wrongs’ in the visual search during navigation.

BACKGROUND

Dynamic positioning (DP) is a computer-controlled system to automatically maintain a vessel's position and heading by using its own propellers and thrusters. DP system is very important for any offshore vessel that operates close to offshore installations such as oil platforms. When maneuvering close to an oil platform in DP, if something goes wrong, the DP operator has only few seconds to prevent an accident. This involves visually checking information, handling alarms, decision making and executing it. In a situation like this all three cognitive, visual and motor loads are higher for the DP operator. The operator has to have the most efficient visual search to keep these loads at minimum and respond to the situation correctly. Looking for the right information at the right time can reduce these loads considerably. However, visual search is a perceptual skill that is not immediately obvious and often unobservable during training. Further, most of the time it is a challenge for the instructor to directly measure what visual cues the trainees are examining at a particular time. So it is safe to say that in the current DP simulator- training instructors evaluate student’s performance by the end result of the given task. So chances of imposing good practices in the visual search of students are often missed in such situations. Using eye tracking during training can rectify this problem and provide a mean by which to effectively assess and correct the visual search skills.

A study conducted in laparoscopic surgery training found out that experts were quicker and more accurate than novices because of their visual search strategy. In that study experts target gaze strategy was found to be more successful than the tool following strategy of novices (Law et al., 2004). Also Law et al. (2004) suggested that with these eye movement differences it may be possible to assess the skills of surgeons as part of a battery of tests, and it could be used to assess the progress of training surgeons. Another study in the medical domain also shows the difference in search strategies between novice and expert radiologists (Nodine et al., 2000). An eye tracking study conducted in aviation industry to check pilot’s monitoring performance reveals that most of the participated pilots did not use the recommended scan pattern from FAA. The pilots looked outside less often than the recommendation (Colvin et al., 2005). Another study conducted in flight simulator concluded that expert-pilots’ eye movement patterns were better defined and the dwell times were significantly shorter than those of novices (Kasarskis et al., 2001).

Bednarik et al. (2005) studied the gaze difference between novice and intermediate programmers during program animation. They found out that novice programmers spend significantly more time on extracting the features of animated concepts. A common finding in the above and other studies is that domain knowledge and experience affect performance and eye movements on a related task. Also another study on US Marine Corps training found out that eye tracking based feedback significantly improved the search strategies of soldiers compared to the traditional feedback method (Carroll et al., 2013). So this study was formed on the basis that if the gaze video of student is shown to instructor during training, the instructor could more closely follow the performance of the student. The hypothesis for the study is, by using eye trackers instructors can more actively intervene during simulator training and can also more accurately assess the student’s performance as he/she can clearly follow where the student’s attention is while operating in simulators. Also using more performance evaluation tools such as checklists will improve and standardize the simulator training. Using eye trackers and checklists will increase the accuracy of feedback and hence the training outcome.

METHODS

Subjects 10 first year nautical students from Aalesund University College were chosen as subjects, 9 of them in the 20-32 (mean = 22.67) years age group. One participant was 47 years old. The older participant was not filtered during hiring process because it was an opportunity to see how different age group people perceive simulator training. All participants had normal or corrected to normal vision. These nautical students were already introduced to dynamic positioning systems in their course study and were familiarized with the controls and equipment. They also had experience training with navigational simulators. However they have not had any DP simulator training before. So they were considered ideal candidates for this kind of training, as less familiarization was required for them. A waiver of consent was applied and approved by Regional ethical committee (REK), Norway before commencing the study since eye trackers are used on human subjects.

Experimental procedure In the current simulator training method students operate the simulator and the instructors stand next to them. The instructors intervene during operation when they think it is necessary or when the student asks to do so. To evaluate the training the student is usually asked to repeat the task independently this time and the instructor monitors him/her from the simulator control station. DP graphical interface in the simulator is mirrored in the control station. So the instructor can follow the changes that student make in the simulator. Currently instructors use this mirrored information to follow the students.

In the new training method the student wears eye-tracker. Choosing the right eye tracker is really important for this study. Generally eye trackers are fixed to a screen or particular interested visual area. This is not suitable for a ship bridge where the subject need to move around. Also the gaze video has to be streamed to instructor in real time. Based on the above requirements ASL mobile eye XG eye-tracking glasses (Fig.1) was selected for this study. Like the current training method the instructor stand next to the student while he/she operate the simulator. However this time the instructor is provided with the real time gaze video of the student. Also during evaluation an additional screen is added in the control station that shows the gaze video of the student in the simulator. In addition to the above a DP Operator assessment checklist was created based on NWEA guidelines (“NWEA guidelines”, 2013), DPO best practices in Norwegian water and inputs from expert DPOs. The purpose of this checklist is to standardize the training feedback from instructors and to help evaluating the effectiveness of eye trackers as support tool in the training.

Figure 1. ASL mobile eye XG eye-tracker
Original PDF Figure 1.

So the new condition is eye-tracking support in addition of DPO assessment checklist. In order to test our hypothesis a case-control experiment was used for the study. In a case-control study participants are divided into two groups, the experimental group and the control group, and then a change is introduced for the experimental group and not the control group. Also to reduce the error variance associated with individual differences, repeated measures design was adapted. So same participants were used in both control and case groups. Another advantage of within subject study design is it requires fewer participants than independent measures study design. Procedure for the study is as follows:

Control group: Standard simulator training Case group: Simulator training + Gaze video of student provided to instructor (in real time)

Both control and case groups had two trials each. Same scenario was repeated during these trials.

Trial 1: Simulator training with instructor’s intervention Trial 2: Independent run by student for training evaluation. Since within subject design was used for this study, counter balancing was introduced by systematic change of order of case and control trials for different participants.

Scenario in the simulator: a Platform supply vessel on DP (auto Pos) was positioned 250 m west of an oil platform, aft of the vessel pointing towards the platform. The main scope was to move the vessel with the side of the vessel in parallel with the crane side of the platform. In order to accomplish this safely the student has to perform a set of sub tasks according to the procedure and recommendations. Initially the scenario and tasks to be performed were explained to the participants in a black board by the instructor. After that participants were allowed to familiarize themselves with the simulator for 10 minutes. Then the trials were conducted. 10 minutes interval was given in between each trial in order to avoid boredom and fatigue of the students. During trials 2 in both case and control study instructors were asked to fill out the DPO assessment checklist. After each study instructors and students were also asked to fill out the simulator training evaluation questionnaire to know their personal opinion about the training. 3 different instructors were used for the above study in order to increase the validity of the findings about eye tracking in training.

Figure 2. Illustration of the new training assessment
Original PDF Figure 2.

RESULTS

Time taken In the following figures (3&4) time taken for trial 1 and trial 2 in both case and control study are compared.

7 out of 10 participants took more time in case group than in control group during trial 1. The average times taken by participants in case and control groups are 18.03 and 16.38 minutes respectively. So, on average trial 1 in case group takes ‘one and half minutes’ longer than in control group.

Figure 3&4. Comparison of time taken

Contrary to trial 1, 8 out of 10 participants took less time in case group than in control group during trial 2. The average times taken by participants in case and control groups are 17.01 and 18.14 minutes respectively. So on average trial 2 in case group takes 1.13 minute less than control group.

DPO assessment checklist The DPO assessment checklist had 33 steps to assess the student’s performance according to the best practice for the particular scenario. Each of the steps had 3 options namely correct, incorrect and inconclusive. The instructors filled this out from the control station during students perform trial 2 in simulator.

Figure 5&6. Results from DPO assessment checklist

From the above figures, it is evident that instructors could be conclusive most of the time in the case group. On average instructors could conclude in 95% of steps that whether the student performed the task correctly or not. Only 17 out 330 steps for 10 participants were mentioned inconclusive by the instructors when eye trackers were used. Also the result from DPO assessment checklist for control group clearly shows that instructors could not come to a decision about students’ performance for almost half of the steps.

DISCUSSION

The main objective of this study was to find out the effect of eye trackers in DP simulator training especially during intervention and evaluation stages of training. The results from DPO assessment checklist and instructors comments on eye trackers in training are the main measures for evaluating the use of eye trackers. The results from DPO assessment checklist clearly reveal that the checklist can only be effectively used when the students’ gazes are tracked. Interestingly with the current training method, for 47% steps in the checklist instructors could not conclude whether the student did it correctly or not. This is a significantly big gap in communication between simulator and the control station from where the instructor monitoring student. This is not ideal for any kind of training evaluation.

For almost half of the total steps the instructors had to rely upon their experience and the overall task outcome to evaluate the student performance. The instructors might be successful in assuming many of the inconclusive steps with their vast experience with the task however it is likely that instructors missing out to notice some bad habits of the students.

In the same task the instructors were able to follow students’ performance for almost 95% of the steps in DPO assessment checklist. This is 43% more than what instructors could achieve with the current training method. Also instructors could more accurately find out the bad practices of students such as not looking outside often and more attention on displays and controls. This was also evident from the training evaluation questionnaire and comments from instructors. Furthermore from our observation during the study, instructors gave more detailed and accurate feedback to students after the session when eye trackers were employed. However the subjective feedback from students does not reveal much difference between the training methods. This might be due to the fact that the questionnaire used was not sensitive enough to gauge the difference between the training experiences or simply because of the fact that the students did not feel any difference. Both the above arguments are possible because the questionnaire used here was not a validated one and also it is important to notice that only instructors were exposed to additional support tools not students.

Usually time taken for trials is not a good performance measure in DPO training because it is difficult to determine how much time should be taken to complete a task in DP operation. Time taken for a DP operation varies according to the overall vessel maneuvering strategy adapted by the DPO. For example some DPO might turn the vessel first then move it closer to oilrig and some might turn the vessel while moving closer to the oilrig. It is difficult to determine which strategy is good or bad as long as it is within the accepted best practices and NWEA guidelines. However as within subject design was used for this study time taken for different trials by the same participant could be compared. Some meaningful patterns and correlations in the overall time taken for different trials were found among the participants. For example 70% of students took more time for trial 1 in case group than in control. This is because the instructors were frequently intervening with the help of eye trackers during trial 1 in case group and intervention usually takes time. This is also supported by the subjective feedback from instructors as they said they could accurately follow the students and more actively intervene with the help of eye trackers. In simulator training, instructors normally intervene to correct mistakes and impose good practice. This was clearly evident in trial 2 as 8 out of 10 students in case group managed to complete quicker than in control group. Active and accurate intervention of the instructors in trial 1 could be the reason for better performance in trial 2 in case group.

All the students who participated in the study showed high level of satisfaction about the simulator training. This does not necessarily mean that the current training method has no room for improvement. Because the students were very excited with the simulator and they might have overlooked the flaws in it or simply could be happy with what they had received. So we cannot draw any conclusion from the subjective feedback from students. We can just take into account that the overall training experience was positive from students’ side.

All three instructors shared positive feedbacks about the use of eye-trackers in simulator training. Enough measures were taken during the trials to make sure that the instructors were not forced to use gaze video. The instructors were informed before the case trials that the gaze video of student was just additional information and they could prefer to use it or not to use it. Nevertheless still there is a possibility that the instructors were attracted by the advanced technology of eye tracking and deliberately used it. However this argument is inconclusive.

CONCLUSIONS

The conclusions of the above DP simulator study can be summarized as follows,

1. Using eye trackers in simulator training, aids instructors to follow students more accurately. This is evident from the differences in the results from DPO assessment checklist among control and case group.

2. Standardization of training evaluation was achieved to some extent by using DPO assessment checklist.

3. DPO assessment checklist combined with eye tracking helped evaluating student’s training outcome more effectively than used without eye tracking. Also the feedbacks from instructors were more detailed with DPO assessment checklist and eye tracking.

4. The results of the subjective evaluation questionnaires and eye tracking of instructors showed that intervention during coaching was more precise and detailed when eye-tracker was used on students.

5. From the subjective feedback of instructors, observation and eye tracking analysis of instructors, eye-trackers were not found to be distractive.

With all above it could be concluded that eye trackers are very helpful in coaching and evaluation phases of DP simulator training. The results are specific to DP simulator training. However, these results can be generalized for other maritime simulator training that has similar setup and training goals as DP simulator. Example: Anchor handling training. One critical thing that should be remembered is developing and customizing the assessment checklist according to different training scenarios. These assessment checklists have to be in detail and should have some validation in order to effectively gauge the training outcome.

ACKNOWLEDGEMENT

We would like to thank Prof. Hans Petter Hildre for his wholehearted financial and moral support for this research work. Also our sincere thanks go to the Maritime Course specialists at Aalesund University College Arnt Håkon Barmen and Tron Richard Resnes for their continuous support and eagerness to help. Their enthusiastic support made it possible to complete the work on time. This project was funded by MARKOM2020 a Norwegian maritime initiative.

REFERENCES

Bednarik, R., Myller, N., Sutinen, E., Tukiainen, M. (2005), “Effects of Experience on Gaze Behavior during Program Animation” in: 17th Workshop of the Psychology of Programming Interest Group, Sussex University Carroll, M., Kokini, C., Moss, J. (2013), “Training effectiveness of eye tracking-based feedback at improving visual search

skills”, in: International Journal of Learning Technology, Vol. 8, No. 2 Colvin, K., Dodhia, R., Dismukes, K.R. (2005), “Is pilots’ visual scanning adequate to avoid mid-air collisions?” Kasarskis, P., Stehwien, J., Hickox, J., Aretz, A., Wickens, C. (2001), “Comparison of expert and novice scan behaviors during VFR flight” in: The 11th International Symposium on Aviation Psychology. Law, B., Atkins, M.S., Kirkpatrick, A.E., Lomax, J.A., Mackenzie, L.C. (2004), “Eye Gaze Patterns Differentiate Novice and

Experts in a Virtual Laparoscopic Surgery Training Environment” Muczyński, B., Gucma, M., Bilewski, M., Zalewski, P. (2013), “Using eye tracking data for evaluation and improvementof training process on ship’s navigational bridge simulator”, In: Scientific journals of maritime university of Szczecin pp.75- 78 Nodine, C., Mello-Thoms, C. (2000), “The nature of expertise in radiology”, In: Beutel, J., Kundel, H., Metter, R.V., eds.:

Handbook of Medical Imaging. SPIE Press NWEA Guidelines for the safe management of offshore supply and rig move. (2013). Retrieved December 12, 2013, from http://www.nwea.info/postmann/dbase/bilder/100308-TX-16021-NWEA-retningslinjer-endelig-no-v3%20mlf.pdf Tjallema, A., Grimmelius, H., van der Nat., Stapersma, D. (2007), “The Road to Eliminating Operator Related Dynamic

Positioning Incidents”, in: MTS Dynamic Positioning Conference, Houston Wagenaar, A.W., Groenewag, J. (1987), “Accidents at sea: Multiple causes and impossible consequences” In: International journal of Man-machine studies pp. 587-598

Cover of the proceedings volume Advances in Human Aspects of Transportation: Part III

Explore the complete digital volume

Advances in Human Aspects of Transportation: Part III

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2099-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →