Chidera Amazu, John Mcgrory, Maria Chiara Leva, Gabriele Baldissone, Davide Fissore, Micaela Demichela (2024). Human Factors in Alarm Response Procedures: an Empirical Analysis of Paper versus Digital support. In: Human Interaction and Emerging Technologies (IHIET 2024). AHFE Open Access, vol. 157, pp. 116–125. https://doi.org/10.54941/ahfe1005470
The objective of a Human Machine Interface (HMI) is to communicate process monitoring information, data, metrics and graphics to an operator through a screen or dashboard and offer an opportunity to control equipment and processes in factories and plants. But after the annunciation of an alarm, how effective is the supporting documentation. The existence of a refined set of instructions and procedures in the form of checklists has been a major factor contributing to the improved safety outcomes observed in the nuclear and aviation sectors. The use of paper-based checklists has been the norm, however, trials of digitized instruction systems have been on the rise in these sectors. The focus of the paper is to analyse an operator on their behaviour and situational awareness from when an alarm is annunciated to the completion of the intervention process using either paper or digitised procedures. The participants(n=46) were split equally into two groups, each testing three tasks with increasing levels of complexity. Results showed that those who were presented with the procedures on paper had slightly better situational awareness and preferred to use paper procedures when compared to those using the digitised procedures. The rationale for this outcome and recommendation for subsequent redesign of the HMI are presented in this paper.
As commercial factories and process plants are becoming increasingly complex in the equipment they use and techniques they perform, it is fair to suggest that the control and support systems they rely on are also becoming progressively multifaceted. We can contend that the function of an HMI is to present process monitoring information, data, metrics and graphics to an operator through a series of screens or dashboards to provide the operator with decision-support material. The HMI should also offer an opportunity for the operator to control equipment and processes. With the increase in plant system complexity, and safety responsibilities are being spread over different departments and vendors, there is now a
greater necessity to support the operator in resolving process faults according to an agreed interdepartmental process. In various safety-critical domains such as process and nuclear industries, documentation supporting alarm response, routine operations, and emergency protocols is paramount for ensuring safety. These documents, through a series of outlined task steps, serve as indispensable guides for operators, aiding them in maintaining plant stability, adjusting process parameters, monitoring, diagnosing issues, making informed decisions, and effectively controlling plant operations. They are an operator support tool used to highlight when to intervene and provide steps on how to rectify abnormal process conditions and have the potential to impact their performance (Barnes and Radford, 1987). Over time, significant advancements have been incorporated into the types and format of support procedures provided to control room operators for executing these activities.
Although considerable effort has been made to normalise HMI components through the introduction of the ISA101 Human Machine Interfaces standard in 2015 to produce high-performance interfaces, there is a lack of clarity on the structure of successful procedures after the annunciation of an alarm.
RELATED WORK
In the nuclear energy domain, there are already migrations afoot from paper to computerised support procedures at various levels based on functionality; Level 1 – Electronic Procedures, Level 2 – Computer-based procedure systems (CBP), and Level 3 – CBPs with Procedure-based automation. Depending on the level, operator supports can range from a primitive level; select and display procedures on a computer screen through to performing sophisticated integrating soft controls to conduct multiple synergised steps on the command of a single operator. (Naser, 2007, IEEE, 2011). In some cases, these supports have been formatted as flowcharts, checklists or numeric task steps. In the aviation sector, checklists have proven to be beneficial for the pilots before take-off, landing or upon the need for troubleshooting during an upset. However, the process industry is still heavily reliant on paper- based procedures for start-ups, shutdowns, troubleshooting or emergency situations.
Generally, procedures in paper form have been criticised for their short comings and contribution to many accidents in the process and nuclear sectors. These shortcomings include having to scan and transition between steps or arrange the papers while monitoring and controlling at same time, incompleteness, lack of updates, complexity to follow...etc (Yang et al., 2012, Gao et al., 2013). Also, in an earlier nuclear sector study focusing on different procedure classification schemes and procedure interfaces, it was reported that one of the key issues with procedures is a lack of human factor consideration which leads to several usability challenges that impact safety (Barnes and Radford, 1987). To resolve these issues, the nuclear sector has transitioned from paper to digitised paper to computerised procedures. It is important to make a distinction at this juncture between digitization and digitalization. Digitization refers to digitising information for electronic
storage and retrieval (e.g. paper compared to PDF, or photograph to JPG image), whereas digitalization refers to the inclusion of business rules that aid the person using the electronically stored media (e.g. searching the document for a keyword/link or pattern match an image content).
To understand the effectiveness of these new systems researchers in the nuclear domain have experimentally evaluated their impact on situational awareness, workload and performance. Yang et al. (2012), in their experimental study, compared the computerised vs paper-based procedures on their impact on workload and situational awareness using subjective methods like Nasa-TLX and SART respectively. Their results reported that CBPs reduced mental workload and improved situational awareness compared to paper-based procedures. Other studies have comparing different computerised procedures in flow chart styles (Xu et al., 2008, Gao et al., 2013). This work aims to analyse an operator on his behaviour and situational awareness from the time an alarm is annunciated through to the completion of the intervention process using either paper or digitised procedures. The rationale for this outcome and recommendation for subsequent redesign of the HMI are presented in this paper.
METHOD
The Experiment Design This study was a between-subject experiment in which individual participants were divided into two separate groups and tasked to perform operator type process resolution instructions and procedures using either a paper- based procedure (G1) or digitised screen-based procedure (G2). Each group comprised of 23 participants. All the participants used in the experiment were equivalent to junior process engineers drawn from the chemical engineering master’s students and staff members of the university. The age range of the participants was broad, spanning from 21 to 61 years, with a standard deviation of SD = 5.4 years.
The Experiment Setup The test rig was a computer-based simulator encoded using MATLAB, primarily based on the four levels of the ISA101 Human-Machine Interfaces standard, as illustrated in Figure 1. The experiment was a single operator environment where the operator was in a seated position with standard room lighting and all three screens are of similar make, model, age with comparable luminance and contrast settings. Screen A, the centre screen, was located directly in front of the operator showing level 1 overall process situation awareness content. Screen A also displayed level 2 windows containing additional selectable detail content. Screen B was tasked with Level 3 content showing specific equipment details. Finally, screen C showed the diagnostics screen information which included the alarm annunciation.
Human Factors in Alarm Response Procedures 119
Figure 1: Test rig layout.
The alarm annunciation component was located in the top left corner of screen C, which was in line with the F-Shaped pattern layout as identified by Nielsen (Nielsen, 2006). The content was in keeping with the ethos of high performing HMI’s incorporating symbols and the limited inclusion of colour. There were additional stimuli components to attract the attention of the operator to the annunciation of an alarm using colour coded alarm banners with black text. A minor alarm appeared as a yellow banner whilst a major alarm appeared as a red banner. There were two types of support mechanism to aid the operator in resolving alarms as illustrated in Figure 2. Mechanism A was where the support documents were provided in a paper format, whereas Mechanism B’s support documents were selected from a hierarchal menu located on the diagnostic screen to the right of the alarm annunciation.
Both paper and on-screen support procedures were presented in a similar font, format and order. The paper-based support was presented on a stack of ordered, single-sided printed pages with a table of contents at the beginning and coloured tab dividers used to separate section areas. The screen-based support was accessed via a tiered navigation menu on the left pane to enable the operator to easily traverse through the sections and respective tag headings to retrieve the appropriate support procedure, as shown in Figure 3. A vertical scroll bar to the right of the procedure allows the operator to move the text down if the procedure bridged two or more pages. The screen-based tiered menu of Mechanism B was not interconnected to the alarm and had no additional stimuli components included, such as colour mapping, jump or hyperlinking to a specific support procedure. Also, no digitalisation using supplementary User Experience UX enhancements, such as, initiative-taking or reactive assistance logic was incorporated. In this experiment, the support procedure was a digitised version of the paper media using the tiered menu. There was no variation of the appearance, timing, order or location of the alarm annunciation for each test participant.
The Experiment Tasks The participants were invited to operate a simulated process undergoing three carefully orchestrated events with escalated levels of complexity and supervisor interactivity. The case study is of a formaldehyde production
plant with identified safety-related events. The events were realistic in nature and completed sequentially for each simulation with 5-minute break between them. Each task required the operator to monitor the process, detect alarms, prioritise alarms, perform a situation assessment, plan a response, and implement the response to manage the event following a standardised procedure. Essentially, each task required specific procedures for the troubleshooting, control and evaluation. An example of the screen navigation menu and procedure tasks for tag FAL01 is shown in Figure 3.
Figure 2: Support mechanism orientation, (A) paper (B) screen.Figure 3: Digital procedure display.
RESULTS
Due to missing data on the responses recorded by the participants and supervisors during the experiments, some participants were omitted in the reporting of certain scenarios. This was done to ensure not to bias the results in favour of a particular outcome. The participants rated their preference using a standard 5-point rating scale (5– Outstanding descending
Human Factors in Alarm Response Procedures 121
to 1– Unacceptable). The following sections divide the research questions into (a) preference on the perceived level of support and (b) situational awareness where the results are presented, followed by a discussion of the findings.
Preference on Perceived Level of Support The participants’ rating of their perceived levels of support provided by paper versus digital formats by each group during the three tasks is presented in Table 1.
Table 1. Rating of the perceived level of support of paper vs. digital by each group during the three tasks.
Table available in the original PDF paper.
PERCEIVED LEVEL OF SUPPORT Procedure type (Paper: G1), (Digital: G2)
Scenario Group M SD MED p (Mann-Whitney) T1 G1 4.33 1.11 5 0.74 G2 4.23 0.97 4.5 T2 G1 4.57 0.98 5 0.18 G2 4.15 0.99 4 T3 G1 3.48 1.12 4 0.70 G2 3.32 1.52 3.5
Impact on Situational Awareness When assessing the carefully orchestrated events from the HMI alarm annunciation to end of intervention using the procedures certain scripted questions were asked by the supervisors to evaluate the 3 levels of operator situational awareness;
I. Perception (Question 1): Which of these alarms, in your opinion, must be verified first? and why? II. Understanding (Question 2): Why do you think the PAL01 alarm is activated? and what do you intend to do? III. Projection (Question 3): Now that you have done this, what do you think will change in the system? Why?
The questions, which were particularly phrased and scheduled to evaluate the participants’ perception, comprehension and projection, were asked using a think-aloud approach at three stages during the test without interruption to the tasks. Specifically, a minute after the alarm annunciation, upon starting to read from the procedure, and after the task is observed to have been completed. The alarm-procedure design, participant process knowledge or participant training/experience and combinations of them can influence the outcomes on these three levels. The results of a comparison between paper- based and screen-based support are presented in Table 2 (Perception), Table 3 (Comprehension) and Table 4 (Projection).
Table 2. Situational awareness level 1 (perception) comparison between paper and digital support during tasks 1, 2 and 3. Scoring: 1 – very low situational awareness, 5 – very high situational awareness.
Table available in the original PDF paper.
PERCEPTION Alarm-Procedure Design Support (Paper: G1), (Digital: G2). M – mean, SD – standard deviation, MED – median
Tasks Group No. of responses M SD MED P (Mann-Whitney) T1 G1 21 4.86 0.36 5.00 0.40 G2 22 4.55 1.01 5.00 T2 G1 21 4.81 0.51 5.00 0.36 G2 20 4.60 0.75 5.00 T3 G1 21 3.90 1.41 5.00 0.35 G2 22 4.32 1.09 5.00
Table 3. Situational awareness level 2 (comprehension) comparison between paper and digital support during tasks 1, 2 and 3. Scoring: 1 – very low situational awareness, 5 – very high situational awareness.
Table available in the original PDF paper.
COMPREHENSION Alarm-Procedure Design Support (Paper: G1), (Digitised: G2). M – mean, SD – standard deviation, MED – median
Tasks Group No. of responses M SD MED P (Mann-Whitney) T1 G1 21 4.29 0.85 4.00 0.15 G2 20 3.73 1.24 4.00 T2 G1 21 4.00 0.95 4.00 0.72 G2 20 4.05 1.15 5.50 T3 G1 21 3.19 1.69 3.00 0.51 G2 22 2.86 1.64 3.00
Table 4. Situational awareness level 3 (projection) comparison between paper and digital support during tasks 1, 2 and 3. Scoring: 1 – very low situational awareness, 5 - very high situational awareness.
Table available in the original PDF paper.
PROJECTION Alarm-Procedure Design Support (Paper: G1), (Digitised: G2). M – mean, SD – standard deviation, MED – median
Tasks Group No. of responses M SD MED P (Mann-Whitney) T1 G1 21 4.33 1.15 5.00 0.20 G2 22 3.68 1.67 4.50 T2 G1 21 4.48 1.12 5.00 0.51 G2 20 4.05 1.57 5.00 T3 G1 21 2.90 1.61 3.00 0.89 G2 22 3.00 1.63 3.00
DISCUSSION
Although the statistical differences of these results are not substantial, there is a clear pattern emerging that there was a higher score in favour of paper-based support procedures compared to screen-based support. Based on related studies by Yang et al. (2012), our original hypothesis that the design of these procedures using screen-based formats would improve performance and score better on the rates of preference and level of support was proven to be false. On the contrary, the data shows that participants in group 2 (G1) with the paper-based procedures consistently rated their preference and level
Human Factors in Alarm Response Procedures 123
of support from their procedures higher than those in group 3 (G2) using screen-based procedures across the tasks.
Likewise, those participants with paper-based procedures reported better overall situational awareness, under the lenses of Perception, Comprehension and Projection, than those who used the screen-based support. This outcome on situational awareness is evident, even as the task complexity increased, except for the most complex task in the perception and projection dimensions. Indicating that the screen procedures are likely to be of better benefit for more complex task scenarios where multiple cues such as alarms are present. The navigation design of the procedures on screen could have been more significant in this type of task, hence the observed outcome. Thus, on the first two tasks, with less alarms or complexity, the print version would be preferred.
Learning From Paper vs. Digital Some people harbour the opinion that reading is simply reading and there should not be a difference if the reading is using paper or an equivalent digitised onscreen content. But the research evidence paints a more nuanced narrative specifically when additional factors are included, for example, the purpose of the reading activity. If the purpose of the reading is for learning, the scale tips in favour of paper. The ability to read and engage with the material by underlining, highlighting, and pointing a finger as a placeholder while reading elsewhere is very helpful to the learning process. The addition of summarising notes by creating simplified visual cues aids in cognitively compacting the material and helps in the encoding process. Deep reading of this type fosters text comprehension, memory, and critical thinking. Whereas, if the purpose of the reading is to get the general idea of a story gist, or for checking texts, social media posts and skim reading the scales are tipped in favour of electronic formats. But the support material used in this experiment is read for neither learning nor providing a gist. It is for the express purpose of following a procedure, in particular operator support.
Two areas that have turned around their safety records are the nuclear and aviation sectors and they have specifically concentrated on the following of a procedure with the use of checklists. This has eliminated the dangerous, unreliable, and unpredictable outcomes associated with trusting one’s memory response or habit. The research evidence also indicates there are two specific moderators where paper-based activities performed better: (1) Time Frame: time-constrained activity and (2) Text Genre: where there is a mix of informational and narrative texts. (Delgado, Vargas, & Ackerm, 2018). The activity of following a process-specific support mechanism is constrained by time and contains information that is a mix of information and narrative text, which supports the reported findings from the experiment and that swings the preferred preference of the operator to paper-based support documents.
Stepping back from the experiment results, an additional three human factor anomalies were identified during the experiment; (1) On two occasions there was an operator error mix-up when a PAL01 Pressure Alarm Low 01 annunciation prompted the operator to follow the support procedure for the PAH01 Pressure Alarm High 01. Although it’s a low instance percentage, on closer examination this operator error may be the result of
two Human Factor characteristics. The first contributing factor being the colour contrast between the black foreground text on a red background. The WCAG Criterion 1.4.3 (AA) relates to contrast for the visual presentation of text and images and sets the minimum acceptable contrast ratio of 4.5:1. The contrast ratio during the test was 4.91:1 which is marginally higher than the WCAG AA criterion. However, the AAA WCAG Criterion 1.4.6. for Contrast (Enhanced) raises the threshold ratio to 7:1. (W3C, 2023). This is a significant finding as red banners are commonly used in HMI design for drawing the attention of an operator. But in doing so the clarity of the text becomes obscure. (2) The second contributing factor being the significance of letter position in word recognition. When reading paragraphs a person does not read every character of each word. The beginning two and end two characters have a greater significance for performing pattern matching when reading where the word is mentally guessed based on the syntax and flow of the sentence (Rawlinson, 1976; Goto, Shirato, & Uda, 2014). On this occasion PAX01 the only difference was the middle character. This observation is also significant as the tag ID using similar abbreviations is common in the industry. (3) Thirdly, when a person is reading a series of titles to search in a document, it is good practice to keep the sequence intact. For example, in Figure 4(1) the alarm annunciation shows the tag ID first (leftmost) followed by the section to the right in a natural language sequence order for reading. Whereas, in the online procedure, used in this experiment the tiered menu for the support document the order is reversed, using the section first (leftmost) followed by the tag ID to the right. This requires some cognitive processing. An improvement would be to invert the banner to Figure 4(2) where the flow of reading of the alarm banner is section then tag ID to align similar to that used in the online support documents, thus offering some cognitive off-loading.
Figure 4: Inversion of alarm banner order of area and tag.
Human Factors in Alarm Response Procedures 125
CONCLUSION
The study compares the effectiveness of paper-based and digitized procedures in supporting operator responses to alarms in industrial environments. Findings indicate a slight preference for paper procedures and better situational awareness, particularly in less complex scenarios contrary to expectations. This aligns with the published research evidence that indicates there are two specific moderators where paper-based activities performed better: (1) Time Frame: time-constrained activity and (2) Text Genre: where there is a mix of informational and narrative texts. The step-back review of the experiment uncovered banner colour, TagID and sequence of labels as being areas for future exploration.
In addition to the observations and recommendations presented, the author looks to further redesign the digital procedure to include missing human factor elements that can potentially achieve the expected result.
REFERENCES
Barnes, V. and Radford, L. (1987). Evaluation of Nuclear Power Plant Operating Procedures Classifications and Interfaces Problems and Techniques for Improvement Prepared for U. S. Nuclear Regulatory Commission. Available at: https://www.nrc.gov/docs/ml1025/ML102560008.pdf. Delgado, P., Vargas, C. & Ackerm, R., 2018. Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, Nov, Volume 25, pp. 23–38 Gao, Q., Wang, Y., Song, F., Li, Z. and Dong, X. (2013). Mental workload
measurement for emergency operating procedures in digital nuclear power plants. Ergonomics, 56(7), pp. 1070–1085. doi: https://doi.org/10.1080/00140139. 2013.790483. Goto, M., Shirato, T. & Uda, R., 2014. Text-Based CAPTCHA Using Phonemic Restoration Effect and Similar Sounds. Vasteras, Sweden, s.n. IEEE Power & Energy Society (2011). IEEE Guide for Human Factors Applications
of Computerized Operating Procedure Systems (COPS) at Nuclear Power Generating Stations and Other Nuclear Facilities. IEEE Standards 1786-2011, vol., no., pp. 1–38. doi: https://doi.org/10.1109/ieeestd.2011.6029264. Naser, J. (2007). EPRI Project Manager Computerized Procedures Design and Implementation Guidance for Procedures, Associated Automation and Soft Controls. [online] Available at: https://www.nrc.gov/docs/ML0800/ML 080080077.pdf [Accessed 14 May 2024]. Nielsen, J., 2006. F-Shaped Pattern For Reading Web Content (original study).
[Online] Available at: https://www.nngroup.com/articles/f-shaped-pattern-readin g-web-content-discovered/?lm=how-people-read-web-eyetracking-evidence&pt =report Rawlinson, G., 1976. PhD Thesis: The Significance of Letter Position in Word Recognition, s.l.: s.n. W3C, 2023. Web Content Accessibility Guidelines (WCAG) 2.2. [Online]. Available
at: https://www.w3.org/TR/WCAG12/ Xu, S., Wells, J. W., Li, Z., Zhao, Q., Li, W., He, X. and Gavriel Salvendy (2008). An ergonomics study of computerized emergency operating procedures: Presentation style, task complexity, and training level. Reliability Engineering & System Safety, 93(10), pp. 1500–1511. doi: https://doi.org/10.1016/j.ress.2007.09.006. Yang, C.-W., Yang, L.-C., Cheng, T.-C., Jou, Y.-T. and Chiou, S.-W. (2012). Assessing
mental workload and situation awareness in the evaluation of computerized procedures in the main control room. Nuclear Engineering and Design, 250, pp. 713–719. doi: https://doi.org/10.1016/j.nucengdes.2012.05.038.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2024)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-33-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional