Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Dynamic Human Error Simulation of Work as Done – Modeling Procedure Deviations With Empirically Derived Failure Mechanisms

1 Idaho National Laboratory, Idaho Falls, ID 83415, USA ↗
2 University of Idaho, ID 83844, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1004455

Abstract

HUNTER is an Idaho National Laboratory software tool developed to support dynamic human reliability analysis. The software performs Monte Carlo simulations of a virtual operator performing procedurally prescribed tasks within the context of a dynamic and coupled nuclear power plant model such that changes in the plant state impact what tasks the operator must perform and the operator’s actions impact the plant state. HUNTER supports a limited suite of scenarios with models containing procedures and corresponding human performance context parameters for a loss of feedwater and steam generator tube rupture scenario. The procedure models contain a single path of steps to mitigate the faults within the two scenarios. Failures occur from the dynamically calculated HEP value exceeding a random generated HEP value for any given task or if the elapsed time to complete a task exceeds the allowed time for that task. In the real world, Operators may error by proceeding along the wrong path and then exceed the allowed time to mitigate a fault as a consequence. To improve realism, HUNTER needs to be able to allow the virtual operator to incorrectly deviate along the wrong procedure path due to a diagnostic or understanding errors in addition to the existing HEP and time failures. Procedure deviations are errors of commission. These are quite challenging to model since there are theoretically infinite errors of commissions that could be made at any point in the simulation. Empirical data collected from a recently performed study evaluating computer-based procedures and failures to adhere to the prescribed procedure steps was used to derive failure mechanisms to realistically constrain the possible deviations to a manageable set that could be modelled within the HUNTER simulation. The process to analyze the empirical procedure adherence data and develop generalized forms of the empirically observed failure mechanisms are described along with their implementation within the HUNTER simulation. Future work aims to continue to validate these failure mechanisms outside of the specific context of the loss of feedwater and steam generator tube rupture contexts to understand their generalizability to other scenarios to more accurately model work as performed within nuclear process control.

© 2023. Published by AHFE Open Access. All rights reserved. 207

INTRODUCTION

Increasing nuclear industry interest in digital control rooms is driving increas- ing adoption of automation in tandem, which requires research on how these digital technologies impact human performance. Computer-based procedures (CBPs) are a unique form of automation that encompasses many of the types of relevant control room automation features (Boring, Ulrich, and Lew, 2023). Furthermore, these individual automation features must be integrated appropriately within the CBP as it becomes the central human-machine inter- face of the new digital control rooms. Traditional human factors research has investigated design principles; however, CBPs have only briefly begun to penetrate the main control room, at least within the U.S. commercial nuclear industry. As such, there is much more research needed on nuclear domain CBPs to ensure utilities have the proper guidance to develop effec- tive site-specific deployments of CPBs with the NUREG-0711 Human Factors Engineering Program Review Model regulatory framework (O’Hara, Hig- gins, & Fleger, 2012). Dynamic human reliability analysis (dHRA) focusing on CBPs provides a unique and synergistic approach for advancing our under- standing of human factors within future digital control rooms. This paper presents a novel approach to evaluate CBPs while simultaneously developing new dHRA methods, specifically the Human Unimodel for Nuclear Technol- ogy Enhanced Reliability (HUNTER; Boring et al., 2022) that can evaluate the digital control rooms including new forms of automation and their impact on the operator roles, responsibilities, and reliability.

DIGITAL CONTROL ROOMS Current nuclear control rooms for the existing fleet of U.S. light water reac- tors have avoided digitizing their instrumentation and controls for the simple reason that there is little economic incentive to do so. Control room mod- ernization itself does not afford greater efficiency, at least in the short term. Furthermore, it is expensive, logistically challenging, and has significant reg- ulatory risk for the licensee. However, equipment obsolescence and rapidly evolving grid infrastructure are driving plants to reconsider adopting digital control room technologies. Antiquated analog instrumentation is no longer available and must be replaced by a digital counterpart. Renewable energy sources like wind and solar have induced greater grid dynamics that increase volatility of whole sale electricity markets such that base load power gen- eration is much less economically viable than when the plants were first commissioned. The industry has begun to look towards alternative revenue streams such as supplying process heat for industrial applications which requires plant modifications and modern digital control systems (Ulrich & Boring, 2022; Ulrich et al., 2021). Perhaps most notably, the emergence of advanced reactors is a primary driver promoting the shift towards the adop- tion of digital technologies for nuclear power. In both the conversion to and initial implementations of digital control rooms for existing or advanced reac- tors, the digital technologies necessitate new regulatory methods to support vendors selecting and deploying control rooms for subsequent review by the U.S. Nuclear Regulatory Commission. To establish safety cases, one of the

Dynamic Human Error Simulation of Work as Done – Modeling Procedure Deviations 209

areas that must be addressed is establishing robust and reliable probabilis- tic risk analysis methods that can model and evaluate human error within a digital control room context.

DIGITAL CONTROL ROOMS AND AUTOMATION Automation used in digital control rooms can broadly be delineated as con- trol and information automation (Boring, Ulrich, and Mortenson, 2019). Control automation describes activities or tasks performed by the system on behalf of the operator. Information automation describes the gathering of data and synthesis into actionable information for an operator. Each defini- tion defines the function performed by the system in relation to the operator to reveal an associated spectrum or level of automation defined as the rela- tive task division between the human and the system (Sheridan & Verplank, 1978). This can ran range from the operator performing all tasks manually to a fully autonomous system performing all tasks without any operator inter- vention. The relative allocation of tasks for any system should be determined based on the concept of HABA-MABA or “Humans are better at/Machines are better at” in which the optimal agent is assigned the task (Fitts, 1951).

Within the digital control room context, distributed control systems (DCS) integrate sensor and controller data for analysis and presentation to the oper- ator typically via a hierarchical multi-windowed display embedded within a vertical control panel or a desktop workstation. Control automation is employed to unburden operators from tedious, repetitive, and error prone tasks in lieu of supervisory control, in which the operator selects a system goal and allows the system to perform the nuanced control actuations to achieve the goal state. The role of operator shifts towards selecting the opti- mal goal states based on information automation. Information automation can make use of the burgeoning field of artificial intelligence to extract pat- terns with machine learning algorithms to characterize the state of the system and even predict future states of the system. Desktop workstations eliminate the need for operators to walk around the control room to acquire indicator values from the various control panels. Instead, the indications are arranged into a series of windowed displays. This multi-windowed arrangement no longer requires physical movement around the control room, but instead requires the operators to identify and then navigate to the appropriate display to find the information. Automation can ease some of this burden by cleverly synthesizing information and intelligently cuing the operator to the appropri- ate display; however, imprudent implementation of automation represents a significant failure point that could induce significant human error.

DYNAMIC HRA FOR DIGITAL CONTROL ROOMS Human error modelling is fundamental to a successful adoption of digital control rooms for both existing and advanced reactors. Models of human error can directly inform the human-machine interface (HMI) design process as vendors approach deployment and begin developing digitally based con- cept of operations. Many advanced reactor vendors are pursing designs with

passive safety features that preclude the need for much of the existing man- ual actions performed by operators in existing nuclear power plants. These designs also aim to leverage autonomy to further reduce any required human intervention, with the lofty goal of full autonomy eliminating the operator role entirely. HRA serves a critical function for either outcome and is particu- larly important in the near term to benchmark potential automation risk and reliability against the extensive established reliability observed for current operations. The methods support the progression towards greater autonomy in advanced reactors, but existing plants without passive safety capabilities will still rely on human operations within their digital control rooms.

In regard to the path for eliminating the human operator, high automa- tion levels are rarely if ever achieved at the initial system deployment, as evidenced in surrogate domains such as aviation and surface transportation. The prudent approach entails incrementally introducing additional types and increased levels of automation in the digital control rooms as confidence in the systems grow. Indeed, existing plants reported this as their intended approach when surveyed about their strategy for control room modernization (Joe, Boring, & Persensky, 2012).

HRA serves the role of qualitatively and quantitatively benchmarking human performance to establish reliability targets the autonomous systems must achieve or exceed during each of these incremental progressions. Fur- thermore, initial prototypes performing function allocation can use dHRA to make risk-informed human or automation function allocations. Lastly, human oversight and intervention during each progressive stage must be analysed to ensure the configuration meets risk requirements. This hybrid configuration is similar to what existing nuclear power plant control rooms are experiencing as they begin to pursue their piecemeal adoption of a digtal control room.

Achieving true autonomy may not actually be a realistic goal, and that is why the term highly automated has been used to describe the goal state for advanced reactors. Indeed, it is quite difficult to envision a scenario where there is no role for a human operator, and more than likely humans will pro- vide oversight of advanced reactors, albeit from a fundamentally different paradigm. Existing methods may not be adequate to model the new oversight role. For example, future digital operations centers will likely incorporate advanced digital supervisory control concepts such as human-robot auton- omy and artificial intelligence. HRA methods must begin considering these new concepts. Efforts to digitize existing control rooms as well as initial efforts to develop advanced reactor control rooms provide some insights into the types of automation HRA methods must consider.

COMPUTER-BASED PROCEDURES FOR DHRA CBPs represent a special form of automation within the digital control room context. NUREG/CR-6634 classifies different levels of CBPs (O’Hara et al., 2000) based on supported automation features. The most automated variant includes live data values of process parameters and soft controls to manipu- late the system organized within a procedure structure that allows the crew to

Dynamic Human Error Simulation of Work as Done – Modeling Procedure Deviations 211

navigate through complicated sequences of tasks and governing logic that dic- tates the path the crew should follow. As such, CBPs represent the bulk of the existing analog control room elements along with the task structures in which those elements are used. This holistic representation makes them a useful tool for HRA methods. The remainder of this paper focuses on computer-based procedures and how they can be used by HRA to capture human performance data and improve our ability to model virtual human operators for dynamic HRA. A prototype empirical computer-based procedure (eCBP) developed to capture human performance data within the context of a dynamic plant state was developed. The rationale for eCBP, implementation details, and the proposed data collection strategy are discussed.

DATA COLLECTION FOR DHRA Traditional static HRA performs time-invariant analysis of human failure events to derive human error probabilities (HEPs; Liinasuo, Karanta, & Kling, 2020). These HEPs are then used within a larger probabilistic risk assessment to characterise overall nuclear power plant risk profiles. Dynamic HRA provides a method to derive HEPs while reducing the subjectivity asso- ciated with the static methods. To perform a dHRA, Monte Carlo simulations are used of a virtual nuclear power plant and virtual operator(s) interact- ing with the plant while resolving some initiating event in the form of a malfunction within the plant systems.

Idaho National Laboratory has been developing a dynamic HRA method called the Human Unimodel for Nuclear Technology to Enhance Reliability (HUNTER; Boring et al., 2022; Ulrich et al., 2022). The unimodel term sig- nifies the simplistic approach to rely on procedures as the driving mechanism for virtual operator tasks performed within the simulations. Performance shaping factors and GOMS-HRA primitives (Boring et al., 2017), repre- senting the most basic task elements used as a dictionary to decompose procedures into basic elements, capture the HEPs and task time respectively. The plant state acts as context to manipulate these values to alter the error rate and speed of execution. Lastly, actions taken by the operator and the natural progression of the physical process within the plant model interact with the procedure logic to govern the potential activities the virtual operator performs as the simulation ensues.

Recent HUNTER development activities aimed to incorporate dependency mechanisms. Dependency is an elusive and nebulous concept to operationally define within HRA; but the notation of dependency is intuitive to understand, with a very simple conceptual definition of “error begets error” (Blackman and Boring, 2018). Toward the goal of deriving a HUNTER simulation mech- anism to model dependency, previously collected empirical data (Boring et al., 2023) collected from the Rancor Microworld Simulator (Ulrich et al., 2017). Rancor was modified with an initial CBP that both guided student “opera- tors”and logged plant process parameters, operator actions on the plant, and operator interactions with the CBP (Ulrich et al., 2022). The analysis found evidence of dependency within the empirical trials representing single scenar- ios developed to emulate a potential human failure event sequences. It was

unable to uncover the specific mechanisms due to a lack of data; however, it revealed the significant potential for CBPs as a tool to record empirical data for HRA.

Scenario-based data collection has precluded the collection of data sam- ples of sufficient size to support the development of dHRA models. Historical approaches rely on full scope training simulators with expert observers cod- ing operator actions in relation to know goals of the scenario to evaluate HRA human performance metrics. As a result, the data are challenging and tedious to convert to units of analysis of sufficient resolution to extract gen- eralizable error metrics that can be applied outside of the specific context evaluated in the scenario. HUNTER, by using GOMS-HRA, provides the method to quantify small units of analysis from scenarios to generalizable tasks that can be used to characterize other contexts. GOMs-HRA provides a collection of fundamental operator tasks called GOMS-HRA task level primitives. Each of these primitives has a nominal human error and time dis- tribution representing the length of time required for execution of the task. The CBP collected data provides the context and task objectives within step level units of analysis, i.e., step epochs, which provide the ability to evalu- ate the appropriateness of the operator actions or lack of actions at a high level of detail to quantify GOMS-HRA primitives that can then be used by HUNTER in dHRA simulations.

Significant post hoc analysis was required to create step epochs from the raw data. In this context, epochs were created for step and procedure units of analysis. The post processing compiled the relevant plant planters and oper- ator actions and evaluated those against the procedure logic to output HRA error categories as described in Table 1 below. Initial classification attempted to use the traditional HRA metrics of correct, error of omission, and error of commission. However, the traditional categories proved insufficient to char- acterise the types of operator actions or lack of actions observed within the context of the step epochs.

Table 1. HRA human error categories used to characterize operators actions or absence of actions while following procedures governing operator response tasks to the plant state.

Table available in the original PDF paper.

Dimensions Level Tradition Metrics Prescribed Ordinal Executed Step Procedure

Correct ✓ ✓ ✓ ✓ ✓ Error of Omission ✓ ✓ ✓ ✓ Error of Commission ✓ ✓ ✓ Proposed Metrics Error of Sequence ✓ ✓ ✓ Discretionary Commission ✓ ✓ ✓ ✓ Discretionary Omission ✓ ✓ ✓

The unit of analysis for epochs is a crucial element to this analysis and the rationale for additional HRA error metrics proposed here. Within each epoch, procedure-prescribed actions were defined as actions explicitly stated

Dynamic Human Error Simulation of Work as Done – Modeling Procedure Deviations 213

within the step. Evaluating the operator response is straightforward within step epochs, and these can be characterised by the traditional HRA metrics. It is possible to use a strict interpretation of the traditional metrics and apply those at the procedure epoch level, but this overlooks much of the nuance revealed while analysing the data for dependency relationships. Procedure epochs require additional categories to characterise deviations that are appro- priate based on the plant conditions and prior operator actions but are not explicitly defined as such within the procedure, which can be defined as the contrast between “procedure completion as prescribed versus as performed.”

To improve accuracy, HUNTER needs to be able to allow the virtual operator to deviate along the wrong procedure path due to diagnostic or understanding errors in addition to the existing HEP and time failures. Proce- dure deviations are errors of commission that are quite challenging to model since there are theoretically infinite errors of commissions that could be made at any point in the simulation. The deviations from the prescribed procedures can greatly expand the fidelity of our dynamic HRA models since they now provide potential error of commission mechanisms that also support recovery and consequently dependency modelling.

A set of error mechanism algorithms must be developed based on empiri- cally derived probability distributions. Errors of omission are not challenging to model, since a nominal probability for missing a particular step can be estimated. Errors of commission cannot be as easily derived since there is no bound to what could be erroneously performed by the operator. Instead the error of sequence, discretionary commission and discretionary omis- sion serve as error mechanisms analogous to the collective category of error of commission. Additional research is needed to collect the data to define the probabilities and time distributions for specific instantiations of these observed error mechanisms within the GOMS-HRA framework. An exper- imental CBP can serve as tool to collect these data, but it is not within the feature set of typical CBPs to collect such data.

PROPOSED COMPUTER-BASED PROCEDURE FOR HRA DATA COLLECTION Commercially deployed CBPs are typically configured for the application without any or with highly limited customization. This is intentionally done to ensure consistency across operators. Commercial CBPs include an approved suite of automation features collectively implemented to enhance performance. As a result, a commercially configured CBP has limited utility to understand the nuances of individual automation features as they impact operator performance. A system evaluation of these individual features and combinations of features requires a configurable CBP explicitly designed to evaluate human performance impacts. Figure 1 shows the proposed CBP system coupled with the Rancor Microworld Simulator during a simulated startup procedure.

For illustrative purposes, one automation feature demonstrates how auto- matic features eliminate the ability to evaluate human performance and the subsequent challenges of isolating and evaluating individual feature impacts

on performance. As can be seen under Step 1 within Figure 1, each substep provides a button containing a descriptive link to the next required proce- dure item based on live plant status and procedure logic. Selecting this button automatically moves the active element to this prescribed item. As such, it is not possible to deviate from the prescribed path of the procedure, and errors cannot be evaluated for navigation within procedure elements. There have been paper based procedures that can identify navigation errors, however these are limited and the applicability to a digital system is unknown. Log- ically, the error prone navigation of paper based procedures is eliminated; however, the specific contribution of navigation errors to overall CBP-based human error is not understood, since the digital format of the procedure and the linked automation features mask easy identification of other errors in procedure following.

Figure 1: Computer-based procedure developed to systematically evaluate human per- formance at a resolution unit of analysis for populating HUNTER model elements with timing distributions and human error probabilities.
Original PDF Figure 1:

Teasing apart individual error probabilities for automation features that eliminate those errors may initially appears superfluous. After all, the error mode has been eliminated in the solution. Nonetheless, automation fea- tures often come at a cost to the operator skills and knowledge in regard to the system as a result of the out-of-the-loop performance decrement. Automated navigation to the next element eliminates the need for the oper- ator to track the logic of the procedure. As such, the operator loses their understanding of why each step may be performed. Additionally, locking operators into a prescribed path provides a consistent task execution; how- ever, there may be novel instances when operators should deviate from the procedure. The ability for operators to override the procedure has been con- sidered; however, the trade-offs associated with automatic navigation require additional investigation, since it is not possible to foresee all instances to provide varying levels of override capabilities. The customizable CBP thus provides a systematic method to evaluate this impact alone. Once individ- ual automation elements can be understood in isolated experiments targeting their performance impacts, the next phase of experiments can then evaluate the interactions between the individual elements with systematic integrated

Dynamic Human Error Simulation of Work as Done – Modeling Procedure Deviations 215

automation feature experiments. These experiments also provide the highly related benefit of providing the means to obtain high resolution task and subtask level unit of analysis of human error across these individual automa- tion features, which can then be used to populate GOMS-HRA primitives for dHRA modelling within HUNTER. This modeling capability can then be flexibly applied to advanced reactors with combinations of these features in their CBPs as well as more traditional control rooms, since the manual exe- cution timing and errors would be captured as baseline performance when evaluating individual automation features.

CONCLUSION

CBPs serve a dual role within near-term HRA research. First, the CBP imple- mentation itself is complex and requires a better understanding of how this relatively new technology impacts the operator task. Second, the CBP itself can be used as a tool to collect human performance data due to the context- providing capability the procedure provides, in which the procedure steps serve as epochs for high resolution units of analysis. Future work with a revised CBP is underway. The new CBP leverages the lessons learned from this initial dependency analysis. Mainly, the new CBP records the data and calculates many of the metrics that were manually calculated post hoc for the previous analysis. The customizability of the CBP was also enhanced to allow features that would typically never be included in a CBP deployed in a commercial setting. This is done to provide better experimental control over the manifestation of errors based on individual procedure features presence. For example, place keeping is one notable advantage of CBPs, but this may unduly restrict the possible types of errors. Manipulating place keeping pro- vides a means to understand the utility of this feature as well as represent more traditional procedure place keeping in line with existing paper based procedures.

ACKNOWLEDGMENT

This work of authorship was prepared as an account of work sponsored by Idaho National Laboratory, an agency of the United States Government. Nei- ther the United States Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal lia- bility or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately-owned rights. Idaho National Laboratory is a multi-program laboratory operated by Battelle Energy Alliance LLC, for the United States Department of Energy under Contract DE-AC07-05ID14517.

REFERENCES

Blackman, H. S., & Boring, R. L. (2018). Assessing dependency in SPAR-H: some practical considerations. In Advances in Human Error, Reliability, Resilience, and Performance: Proceedings of the AHFE 2017 International Conference on Human Error, Reliability, Resilience, and Performance, July 17–21, 2017, The Westin Bonaventure Hotel, Los Angeles, California, USA 8 (pp. 13–19). Springer International Publishing.

Boring, R. L., Rasmussen, M., Ulrich, T., Ewing, S., & Mandelli, D. (2017). Task and procedure level primitives for modeling human error. Advances in Intelligent Systems and Computing, 589, 30–40. Boring, R. L., Ulrich, T. A., & Lew, R. (2023). Levels of digitization, digitalization,

and automation for advanced reactors. Proceedings of the 67th Human Factors and Ergonomics Society International Annual Meeting. Boring, R., Ulrich, T., Lew, R., Hall, A., Park, J., & Kim, J. (2023). Preliminary empirical findings on dependency from a simulator study. Proceedings of the American Nuclear Society Joint Topical Meeting on Nuclear Power Instrumen- tation and Control and Human-Machine Interface Technology and Probabilistic Safety Assessment, pp. 195–204. Boring, R., Ulrich, T., Park, J., Heo, Y., & Ahn, J. (2022). The HUNTER Dynamic

Human Reliability Analysis Tool: Overview of the Enhanced Framework for Mod- eling Human Digital Twins. In Proceedings of the Probabilistic Safety Assessment and Management (PSAM 16) Conference. Joe, J. C., Boring, R. L., and Persensky, J. J. (2012). Commercial utility perspectives on nuclear power plant control room modernization. 8th International Topical Meeting on Nuclear Power Plant Instrumentation, Control, and Human-Machine Interface Technologies (NPIC&HMIT), pp. 2039–2046. Liinasuo, M., Karanta, I., & Kling, T. (2020). Dynamic human reliability analysis

(HRA): A literature review. O’Hara, J., Higgins, J., & Fleger, S. (2012). Human factors engineering program review model (NUREG-0711) revision 3: update methodology and key revisions. In proceedings of 8th international topical meeting on nuclear plant instrumentation and control and human-machine interface technologies. O’Hara, J., Higgens, J. C., Stubler, J., & Kramer, J. (2000). Computer-based Proce-

dure Systems: Technical Basis and Human Factors Review Guidance (Technical Report No. NUREG/CR-6634). Sheridan, T. B., & Verplank, W. L. (1978). Human and computer control of undersea teleoperators. (Technical Report). Cambridge, MA: Man Machine Systems Laboratory, MIT. Ulrich, T. A., Boring, R. L., Lew, R., & Whiting, T. A. (2023). Rancor computer-based

procedures – A framework for task level human performance data collection. In Proceedings of the 67th Human Factors and Ergonomics Society International Annual Meeting Ulrich, T. A., Boring, R. L., & Lew, R. (2022). Beyond control room modernizationnuclear concept of operations development for novel systems using operator-in- the-loop control room simulations. Proceedings of the 66th Human Factors and Ergonomics Society Annual Meeting, pp. 1737–1741. Ulrich, T., Boring, R., Park, J., Heo, Y., & Ahn, J. (2022). The HUNTER Dynamic

Human Reliability Analysis Tool: Procedurally Driven Operator Simulation. In Proceedings of the Probabilistic Safety Assessment and Management (PSAM 16) Conference. Ulrich, T. A., Lew, R., Hancock, S. G., Westover, T. L., Medema, H. D., Boring, R. L., & Minard, N. (2021). Dynamic Human-in-the-Loop Simulated Nuclear Power Plant Thermal Dispatch System Demonstration and Evaluation Study (No. INL/EXT-21-64329-Rev000). Idaho National Lab. (INL), Idaho Falls, ID (United States).

Cover of the proceedings volume Augmented, Virtual and Mixed Reality Simulation

Explore the complete digital volume

Augmented, Virtual and Mixed Reality Simulation

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-94-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →