Jari Laarni, Hanna Koskinen (2018). Method for Fidelity Evaluation of Nuclear Power Plant Simulators from the Human Factors Point of View. In: Advances in Physical Ergonomics and Human Factors: Part I. AHFE Open Access, vol. 1. https://doi.org/10.54941/10025
In order to support operator skill acquisition, training simulator environments have to be realistic and represent the real-world counterpart. There is a lot of research on simulator fidelity on various domains, and even though it is also an important topic in the nuclear energy field, there is little research and guidance on how the fidelity of nuclear power plant training simulator should be evaluated. Simulator fidelity evaluation is challenging in the nuclear domain: to be able to use the simulator for training purposes, fidelity evaluation is required before commissioning; on the other hand, it is difficult to compare the simulator to a control room that does not yet exist or that has not yet been upgraded.Our solution to this challenge is a stepwise iterative approach to fidelity assessment that is based on a well-defined fidelity assessment plan, according to which different components of simulator fidelity are evaluated successively in different instants of time. First, we evaluate the physical fidelity including environmental and equipment fidelity; second, we evaluate task and functional fidelity. We will apply our approach in the fidelity evaluation of a training simulator of a Finnish nuclear power plant whose automation and control room systems are digitalized.
Training simulators play an important role in training and licensing of new operators, in refresher training and in periodic re-qualification of licensed operators in nuclear industry. In addition to operator training, simulators are also used in the Integrated System Validation (ISV) of control room systems in modernization and new build projects.
In order to support operator training and ISV of control room systems, the training simulator must resemble to a sufficient degree the referent control room. This paper focusses on simulator fidelity, which can be defined as the degree of similarity between a training simulator and a reference (that is, the main control room, MCR), and more specifically, on the evaluation of its physical fidelity from the human factors (HF) point of view. We will also touch another topic which typically has not been so much dealt with in the discussion of simulator fidelity, that is, how and to what degree the training simulator supports the work of trainers. New simulators equipped with new features, such as advanced data acquisition and analysis tools, apparently provide new opportunities for training; on the other hand, they may also have flaws that make them less usable for their purposes than the old simulator.
In nuclear power plant (NPP) modernization projects, a special problem in the evaluation of simulator fidelity is the fact that you need information of the fidelity of the training simulator before the reconstruction of the MCR has been completed. At this phase, however, you cannot directly compare the fidelity of the simulator to the referent CR. This kind of direct comparison does not become possible until the commissioning of the new MCR. Therefore, we propose that the assessment of simulator fidelity has to be conducted in an iterative fashion, and it should consist of several sub-phases.
The proposed simulator fidelity assessment procedure is one element in a more integrated approach to Human Factors Engineering (HFE) approach that is tightly integrated into the engineering design process from the beginning. This kind of integrated HFE process is characterized, e.g., by the proper timing of implementation of HFE activities, the use of screening and graded approach to HFE evaluation, the importance of skills of participating stakeholders, early integration of operators into development, and appropriate sharing of knowledge among designers, HFE experts and management (Laarni and Savioja, 2014).
CHARACTERISTICS OF SIMULATOR FIDELITY Simulator fidelity is a multidimensional concept, and there are different classifications of simulator fidelity (e.g., Dahl et al., 2010). A common classification is based on the distinction between physical and psychological fidelity. Physical fidelity is the degree to which the real-world operational equipment and environment are replicated in the simulator, and to what degree the simulation duplicates the physical elements of the real-world baseline. Psychological fidelity concerns the degree to which the end-user perceives the training environment as being identical of the real-world system, and the degree to which he/she is psychologically engaged in the simulated environment. Physical fidelity can, in turn, be divided into equipment and environmental fidelity. Equipment fidelity refers to the extent to which the simulator replicates the appearance and feeling of the real-world systems and to the extent to which their components is replicated in the simulation; and environmental fidelity refers to the extent to which the simulated environment replicates the physical characteristics and sensory stimulation of the real-world system. Psychological fidelity can be further divided into task fidelity and functional fidelity. Task fidelity refers to the degree to which operational conditions are considered as realistic in the simulation. Functional fidelity concerns the degree to which the simulation provides realistic stimuli to the end-users, and the degree to which the response options of the simulator environment are realistic.
Actually, a simulation cannot be directly compared to the real-world referent but only to a representation of the real- world system. Schricker et al. (2001) have, thus, differentiated the concept of the referent from the model (simulation), and according to them, simulator fidelity can be defined as a measurement of to what degree a simulation represents a referent which is a representation of the corresponding real-world system.
The problem with the above-mentioned, end-user-centred approach is that it does not take into account the fact that the simulator also has to support trainers in their work. Issues related to simulator training capabilities have typically not been considered in discussions on simulator fidelity, and there is no existing classification of issues that are included in the simulator’s suitability for training. Important topics in this respect are, e.g., perceptiveness (visibility, audibility) of operator activities and discussions; ability to record operator activities and later view and analyse them; provision of alerts, warnings and advice to inform the trainer that, e.g., model parameters have exceeded particular values (ANSI/ANS-3.5-1998, 1998); ability to flexibly control different aspects of the simulation (e.g., stop, repeat, slow down or speed up the pace of a simulator run); and ability to flexibly communicate with operators and give instructions to them during a simulation run, if needed.
Importance of fidelity: Fidelity and transfer of training There are many reasons for the use of simulators in different industries. For example, simulators are needed, since training in an actual performance context is not possible in many cases, and a simulator can also be easily controlled for instruction. An important concept is transfer: transfer in training occurs when trainee knowledge and skills learned in one context can be used in a different context (e.g., Liu et al., 2008). Positive transfer occurs when learned knowledge has a positive impact on performance; negative transfer occurs when the impact of learned knowledge is negative (e.g., Kahana, 2012). Transfer can be measured, for example, by measuring the reduction in the amount of time that is needed to learn the task at hand.
It is often thought that there is a link between simulator fidelity and learning results so that the higher the fidelity of a simulator the more successful the transfer of training. Even though this sounds as a rationale argument, there is a lot of evidence showing that there is no direct effect of simulator fidelity to learning results, and some evidence even suggests that high levels of fidelity can sometimes result in negative transfer (Malik et al., 2009).
There is, however, much evidence that simulator training is necessary for learning procedural skills to such a degree that they can be performed fluently, since there must be a close match between the training environment and the baseline in order to be able to learn these kinds of skills (e.g., Baum et al., 1982). In addition, there is much experimental evidence that fidelity interacts with many factors, which makes it difficult to assess the direct effect of fidelity on training. For example, physical fidelity interacts with the type of task, target level of training, prior level of proficiency and user acceptance (Dahl et al., 2010; Malik et al., 2009). Overall, for many applications, there is a need to identify an optimal level of simulator fidelity without compromising positive transfer. But how to determine this optimal level is still a tricky task to perform.
Measuring of simulator fidelity In order to find the optimal level of fidelity, some kind of measurement of simulator fidelity is needed. Both quantitative and qualitative measures exist for the measurement of simulator fidelity. Basically, all the existing methods are based on the rating the simulator for similarity to the baseline system. A common feature in most of the fidelity evaluation methods is that they are subjective in nature and based on expert judgement: People who are familiar with the baseline system have to first identify the discrepancies that are critical to training, and second, they are asked for their opinion on the criticality of each discrepancy between the simulator and the baseline system.
A very simple model is based on two subsequent phases: The first task is to divide the target system into a set of tasks or conditions that are considered to be important to training (Schricker et al., 2001). Each of these tasks and conditions is then provided a rating of either zero or one depending on to what degree the simulator environment represents the baseline system. The overall fidelity measurement is provided by calculating an average of these values. Recently, some mathematical, model-based approaches to simulator fidelity and training effectiveness have been developed. (for a review, see Liu et al., 2008; Schricker et al., 2001).
FIDELITY OF NUCLEAR TRAINING SIMULATORS In the nuclear domain, the plant-specific full-scope training simulators should provide adequate support for the learning of operating skills. Therefore, the fidelity of nuclear training simulators has to be addressed before the execution of training activities. In most cases, the main problem in the evaluation of the NPP simulators is that the referent CR does not yet exist, and therefore the evaluation has to be based on written documents and drawings.
The critical question is to what degree the lack of simulator fidelity can be expected to have a negative impact on operator performance. For the evaluation of this question, the identification of possible deviations and the evaluation of their criticality is an important issue. For example, from the perspective of operator training, possible differences in HSIs of safety I&C systems are more important and critical than differences in process I&C systems.
ANSI/ANS-3.5-1998 provides functional requirements for NPP simulators that are used for operator training and for examination of operators. As a general requirement, the simulator should resemble the MCR to a degree that operators behave and act in a similar way in both of these environments. More specifically, according to ANSI/ANS-3.5-1998, the training simulator should fulfil the following specific requirements:
simulator response to all kinds of operator- or automation-initiated control actions should be realistic;
simulator should function in real time;
simulator should support the handling of all essential normal plant events in a continuous manner;
simulator should support the management of all important incidents and accidents, and it should support the recovery and mitigation of the consequences of these events;
simulator should include all CR equipment used by operators in the MCR to conduct all normal events and to respond to incidents and accidents;
all environmental features of the CR that support operator work in various plant states should be constructed.
According to ANSI/ANS-3.5-1998, all deviations that exist between the training simulator and the MCR should be assessed. All the deviations that do not impact the actions to be conducted by operators or deteriorate training may be considered acceptable. Regarding simulator performance fidelity, it is required that a validation test is conducted, and the results are evaluated against actual or predicted data from the baseline system (ANSI/ANS-3.5-1998).
There is a quite extensive discussion of fidelity in guides that provide requirements for training simulators that are used as validation test environments. According to NUREG/CR-6393 (O’Hara et al., 1996), “the degree to which the plant model and HSI deviate from the actual design determines the degree to which the representativeness is compromised and, thus, the degree to which threats to system representation validity emerge” (p. 5-2). NUREG/CR- 6393 lists several aspects of the simulator (i.e., human-system interface and process model) as significant factors for integrated system validation. These factors mainly refer to the resolution, accuracy and capacity of the simulation. According to NUREG-6393, the test facility should represent the control room in terms of human-system interface (HSI) completeness, physical fidelity, functional fidelity, data completeness, content and dynamics fidelity, and environmental fidelity. HSI completeness refers to the degree to which the simulator represents the unified referent CR (O’Hara et al., 1996). Data completeness fidelity refers to the degree to which all data from plant processes and systems are included in the simulation; data content fidelity refers to how accurately the simulated data replicates the data associated with the modelled system; and data dynamics fidelity refers to the degree to which the changes in plant data presented by simulator HSIs mimic those that are seen in the baseline system (O’Hara et al., 1996).
PROPOSAL FOR A METHOD FOR SIMULATOR FIDELITY EVALUATION Our starting point is that the simulator fidelity evaluation must be carried out in an iterative fashion, and the evaluation process should consist of at least three successive stages: The first phase can be carried out after the accomplishment of the sub-system level validation tests and simulator performance validation tests. In this phase, it is possible to evaluate the effect of those deviations that have been identified by that time. In the first phase, the assessment should include the following main tasks:
1) Selection of operational conditions/scenarios that are relevant for training/Integrated System Validation;
2) Familiarization with the specifications of the new MCR and simulator performance test data;
3) Identification of the deviations between the training simulator and the MCR that are relevant to the accomplishment of the selected simulator runs;
4) Evaluation of the criticality of the deviations according to a particular scale by an expert panel;
5) Assessment of the acceptability of the simulator for training and ISV.
The first three tasks create the prerequisite conditions for the fourth task, that is, for the evaluation of the impact and criticality of the identified deviations in terms of possible operator tasks. Therefore, it is important to put scores for deviations across selected training/ISV scenarios.
In the first evaluation round, simulator fidelity evaluation whose focus is on HF issues should, at least, assess the differences in HSI completeness fidelity, HSI physical fidelity (e.g., hardwired panels, controls/switches, instrumentation, alarms, displays, and procedures), and environmental fidelity (e.g., CR layout, furniture, and ambient environment). After simulator performance testing, it is also possible to deal with HF considerations related to functional fidelity and data completeness/content/dynamics fidelity.
The second evaluation round should be conducted after operator training and ISV, and it should focus on those deviations that have been identified in training and ISV tests. The third and the final evaluation cannot be conducted until the commissioning of the I&C and CR systems and until there is some experience of the use of the new HSIs of the MCR.
Simulator fidelity evaluation based on expert judgment Fidelity evaluation has to be based on the work of a panel of subject-matter experts. Expert-opinion elicitation and judgment can be defined as a heuristic and structured process of gathering information and answering questions of issues or problems (e.g., Ayyub, 2001; Meyer & Booker, 1991; Budnitz et al., 1997). In the nuclear domain it has been applied, e.g., in the characterization of sites for final disposal facilities of high-level radioactive waste (Kotra et al., 1996).
The expert-opinion elicitation process in training simulator fidelity evaluation includes the steps as shown in the figure 1. The tasks can be divided into three major groups: 1) Preparatory activities (including Establishment of expert elicitation process, Selection of study leader, integrator and facilitator, Definition of objectives and selection of main issues, Selection of experts and peer reviewers, Issue familiarisation and refinement and Aggregation and dissemination of background information), 2) Panel execution (including Pre-elicitation training and Elicitation of opinions and Rating of issue criticality), and 3) Concluding activities (including Analysis, accumulation and resolution, Administration of review process, and Documentation and communication of results).
An expert panel should consist of a couple of experts whose task is to evaluate the relative importance of the observed differences between the simulator and the MCR. Since only subject-matter experts can provide the required answers to questions that are discussed, they have to be carefully selected. Simulator trainers, safety engineers, experienced operators and CR designers can be considered as good candidates for the NPP fidelity evaluation panel team.
According to Ayyub (2001), an expert-opinion elicitation process involves a technical integrator and a facilitator who are responsible for the realization of the expert-opinion elicitation process, and they may be representatives of a consulting organization. The expert-opinion elicitation process is also required to include peer reviewers for quality assurance purposes (Ayyub, 2001; Budnitz et al., 1997).
The expert panel should focus on important deviations that may have safety implications. Before the conduct of the expert panel, it has to be communicated to the experts all the relevant background information, objectives of the elicitation process, and a list of the issues that are planned to be addressed in the expert panel meetings (Ayyub, 2001; Budnitz et al., 1997). These issues cover the deviations between the simulator and the MCR in the following topics: procedures, instrumentation, controls, dedicated safety-related controls and displays, alarms, large-screen displays, and workplace design (for a more detailed list, see, e.g., for O’Hara et al., 2002).
The elicitation and judgment process is conducted by a face-to-face panel of experts which is designed specifically for the discussion and resolution of the critical deviations between the simulator and the MCR (Ayyub, 2001; Budnitz et al., 1997). The clear communication of the main aims and goals of the expert panel to all experts is one of the first tasks at the panel meeting (Ayyub, 2001). Training of experts also has to be conducted during the panel meeting before the start of the elicitation process. In training sessions the aim is to train experts to provide answers in a desired way (Ayyub, 2001; Budnitz et al., 1997).
The elicitation process should proceed in a systematic fashion, and the discussions should be started with presentations of background material (Ayyub, 2001; Budnitz et al., 1997). After that, information of possible uncertainties, of expert-opinion elicitation process, of technical issues, and of accumulation of expert opinions should be presented (Ayyub, 2001; Budnitz et al., 1997). Each deviation between the MCR and the simulator has to be presented to the panel, and its potential influences have to be discussed. Participants are allowed to ask questions and clarifications to solve problematic issues, and they are provided clarifications of the scope and conditions for the deviations (Ayyub, 2001). The facilitator’s role is to encourage discussions on the discrepancy at hand (Ayyub, 2001; Budnitz et al., 1997).
I ESTABLISHMENT OF EXPERT ELICITATION PROCESS
II SELECTION OF STUDY LEADER, INTEGRATOR &
FACILITATOR III DEFINITION OF OBJECTIVES & SELECTION OF MAIN
ISSUES
IV SELECTION OF EXPERTS & PEER REVIEWERS
V ISSUE FAMILIARISATION & REFINEMENT
VI AGGREGATION & DISSEMINATION OF
BACKGROUND INFORMATION
VII PRE-ELICITATION TRAINING
DOCUMENTATION
VIII ELICITATION OF OPINIONS/RATING OF ISSUE
CRITICALITY
IX ANALYSIS, ACCUMULATION & RESOLUTION
X ADMINISTRATION OF REVIEW PROCESS
Figure 1. Expert-opinion elicitation process in simulator fidelity evaluation (see also Kotra et al., 1996).
Figure available in the original PDF paper.
For each deviation, some Likert-type scales are prepared dealing with the similarity of the simulator to the MCR, the importance/criticality of the possible deviations in terms of nuclear safety and their impact on training effectiveness (see, e.g., Malik et al., 2009). One open-ended question is included asking the experts to describe the differences that have the most significant effect on operators’ ability to perform their tasks and duties.
Finally, experts are asked to state their confidence in their responses by ranking the questions from 1 to 7, in which 1 means that “in comparing you own ability to answer this item with what you think the ability of other participants to be, you estimate that you have the relatively best chance of ending up with the correct solution than the others”; and 7 means that “you consider you have the smallest chance to end up in the correct solution” (Ayyub, 2001; Helmer, 1968).
Based on the experts’ ratings, a fidelity acceptance matrix can be developed, including dimensions for the difference between the MCR and the training simulator; and for the impact/criticality of the difference on operator tasks/duties and training performance (see figure 2; Ayyub, 2001; Wiggins, 1985). Results depicted in the matrix can, in turn, be used in the calculation of an acceptance index (Ayyub, 2001; Wiggins, 1985).
It is also useful to prepare a summary of the discussions and reasoning provided during the meeting (Ayyub, 2001). If divergent opinions have been raised at the meeting, bases and reasons for these differences should be carefully documented (Kotra et al., 1996). The summary results have to be presented to the experts, and they should have an opportunity to discuss the issues once again and revise their ratings (Ayyub, 2001; Budnitz et al., 1997). The experts are also asked to explain the basis and rationale for their revisions. The revised evaluations of the experts are analysed and consolidated into a final summary. Meyer and Booker’s (1991) book provides detailed guidance on the analysis of expert judgment data.
Figure 2. An example of a fidelity acceptance matrix that can be used in the determination of the acceptance of the training simulator for training/ISV (adapted from Ayyub, 2001; Wiggins, 1985). An acceptance index (below) can be derived from the matrix data.
Simulator fidelity evaluation at a Finnish nuclear power plant A Finnish nuclear power plant is just modernizing the I&C systems and main control rooms into new systems that are based on digital automation and digital human-system interfaces. The transition will be carried out during a period of over several years in a stepwise manner with four successive modifications of which the first one has already been finished. A new training simulator has also been built to address primarily the training needs raised by the modernization project, but the new simulator will also be used for verification and validation (V&V) purposes. Especially, it will function as a stage for the ISV tests of the 2 nd stage of the modernization project. There are, thus, good reasons for evaluating the fidelity of the training simulator before the start of the training programme and the ISV test activities.
It is proposed that the above-mentioned iterative approach is used in the HF fidelity evaluation of the simulator. The evaluation process should consist of three successive stages (figure 3): The first phase is conducted after the conduct of the last sub-system validation test and the conduct of the simulator performance validation tests. The second phase should be conducted after operator training and ISV tests to assess the impact of possible deviations identified through these activities, and the third final phase should be conducted until the commissioning of the CR changes and after the new CR has been used for a certain period of time.
In the first phase, it is possible to evaluate the effect of those deviations that have been identified after the sub- system validation tests. The main differences between the new training simulator and the CR that are known for the moment are the following:
Interactive large screen displays based on touch sensitive wall mounted displays have replaced the analog HSIs of the turbine process automation at the new training simulator.
Room layout of the new simulator is different from the plant CRs, e.g., because of the location of the observation gallery.
BACKGROUND INFORMATION
TRAINING/ISV REQUIREMENTS
REQUIREMENTS FROM STANDARDS/GUIDES
SSV Test Results
Training Needs
ANSI/ANS-3.5
Virtual Panel Test Results
Training Scenario Descriptions
NUREG/CR-6393
Specificationsfor New MCR
ISV Test Scenario Descriptions
Simulator PerformanceTest data
SIMULATOR FIDELITY ASSESSMENT (I PHASE) Expert-opinionElicitationProcess
APPROVAL FOR USE (IN TRAINING AND ISV)
OPERATOR TRAINING & ISV
SIMULATOR FIDELITY ASSESSMENT (II PHASE)
IMPLEMENTATION & COMMISSIONING
SIMULATOR FIDELITY ASSESSMENT (III PHASE)
Figure 3. Outline of a simulator fidelity evaluation process for the I&C and CR modernization.
Figure available in the original PDF paper.
Since the interactive large-screen panel system has already been implemented into the new simulator, it is possible to evaluate its effect on simulator fidelity at an early state. A usability test of the virtual panel system has been recently carried out, and these test results provide background knowledge for the fidelity evaluation, and together with knowledge that will be gathered in the SSV tests, they provide a good basis for the assessment of the impact of interactive large screen panels on operator training and ISV tests. Simulator performance fidelity has to be evaluated through validation tests before the start of operator training and ISV testing. In the first evaluation round, HF implications of functional fidelity and data completeness/content/ dynamics fidelity can thus also be evaluated based on what is known about plant dynamics and conditions.
The difference of the physical layout between the new simulator and the future MCR of Unit 1 and 2 may have an impact of operator training and ISV test performance. Because of the location of the operating gallery, the position of some of the safety HSIs are shifted some meters to the right. The criticality of these deviations has to be evaluated before the operator training and ISV tests can be started. Results of the last SSV test will provide background information for the assessment of the impact of these layout differences.
The results of the first iteration of the fidelity assessment have to be available before the start of the operator training programme and before the start of ISV tests. Similarly, the results of the second evaluation round should be available in good time before commissioning of the MCR. The final evaluation round which should focus on more detailed aspects of functional and data fidelity of the simulator model cannot be carried out until the operators have some experience of the use of the new MCR HSIs.
CONCLUSIONS
A methodology has been outlined for the evaluation of the simulator fidelity of NPP training simulators for operator training and validation purposes. It is suggested that the assessment of simulator fidelity is conducted in an iterative fashion, and it should consist of three subsequent stages in the lifecycle of the control room systems. This kind of iterative approach is needed, because all the information of the facilities to be compared is not available at the time the first evaluation has to be conducted, but it is accumulated over time by testing the new training simulator and by implementing the new control room. We propose that this approach may form the basis for the evaluation of the fidelity of the training simulator of a Finnish NPP. The iterative simulator fidelity assessment procedure is also one element in our more integrated approach to Human Factors Engineering approach that is tightly integrated into the engineering design process from the beginning.
REFERENCES
ANSI/ANS-3.5-1998 (1998), “Nuclear Power Plant Simulators for Use in Operator Training and Examination”. La Grange Park, IL: American Nuclear Society. Ayyub, B.M. (2001), “A Practical Guide on Conducting Expert-Opinion Elicitation of Probabilities and Consequences for
Corps Facilities. IWR Report 01-R-01”. Alexandria, VA: Institute for Water Resources. Baum, D.R., Riedel, S., Hays, R.T., Mirabella, A. (1982), “Training Effectiveness as a Function of Training Device Effectiveness. ARI Technical Report 593”. Alexandria, VA: US Army Research Institute. Budnitz, R.J., Apostolakis, G., Boore, D.M., Cluff, L.S., Coppersmith, K.J., Cornell, C.A., Morris, P.A. (1997),
“Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts: Main Report (NUREG/CR-6372)”. Washington, DC: U.S. Nuclear Regulatory Commission. Dahl, Y., Alsos, O.A., Svanäs, D. (2010). “Fidelity considerations for simulation-based usability assessments of mobile ICT for hospitals”, International Journal of Human-Computer Interaction Volume 26 pp. 445-476. Helmer, O. (1968), “Analysis of the future: The Delphi method”, in: Technological Forecasting for Industry and Government,
Bright, J. (Ed.). Englewood Cliffs, NJ: Prentice Hall. Kahana, M.J. (2012), “Foundations of Human Memory”. New York, NY: Oxford University Press. Kotra, J.P., Lee, M.P., Eisenberg, N.A., DeWispelare, A.R. (1996), “Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program (NUREG-1563)”. Washington, DC: U.S. Nuclear Regulatory Commission. Laarni, J., & Savioja, P. (2014), “Developing an integrated Human Factors Engineering process for the nuclear domain. VTT-R-
00536-14”. Espoo: VTT. Liu, D., Macchiarella, N.D., Vincenzi, D.A. (2008), “Simulation fidelity”. In: Human Factors in Simulation and Training, Vincenzi, D.A., Wise, J.A., Mouloua, M., Hancock, P.A. (Eds.). pp. 61-74. Boca Raton, FL: Taylor & Francis. Meyer, M., Booker, J. (1991), “Eliciting and Analyzing Expert Judgement. A Practical Guide”. London: Academic Press. Minsk, B.S., Branch, K.M., Bates, E.K., Mitchell, M.R., Gore, B.F., Faris, D.K. (2009), “Technical Basis for Physical Fidelity of
NRC Control Room Training Simulators for Advanced Reactors. PNNL-18865”. Richland, Washington: Pacific Northwest National Laboratory. O’Hara, J., Brown, W.S., Lewis, P.M., Persensky, J.J. (2002), “Human-System Interface Design Review Guidelines (NUREG- 0700, Rev. 2)”. Washington, DC: U.S. Nuclear Regulatory Commission. O’Hara, J., Higgins, J., Fleger, S.A., Pieringer, P.A. (2012), “Human Factors Engineering Program Review Model (NUREG-
0711, Rev. 3)”. Washington, DC: U.S. Nuclear Regulatory Commission. O’Hara, J., Stubler, W, Higgins, J., Brown, W. (1996), “Integrated System Validation: Methodology and Review Criteria (NUREG/CR-6393)”. Washington, DC: U.S. Nuclear Regulatory Commission. Schricker, B.C., Franceschini, R.W., Johnson, T.C. (2001), “Fidelity evaluation framework”. In: Proceedings of the 34 Annual
Simulation Symposium, 22-26 April, 2001. Wiggins, J. (1985), “ESA Safety Optimization Study. HEI-685/1026”. Houston, TX: Hernandez Engineering.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Physical Ergonomics and Human Factors: Part I
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2104-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional