Hiroshi Ujita (2024). Model, Method, and Data Issues in Human Reliability Analysis (HRA). In: Human Error, Reliability, Resilience, and Performance. AHFE Open Access, vol. 132, pp. 16–22. https://doi.org/10.54941/ahfe1004876
According to the results of the HRA comparison study by USNRC and Halden, the evaluation results are significantly different when comparing the method calculations alone, comparing the experiments alone, and further comparing the method and experiment trends [1]. It is considered that human correspondence characteristics have a large context dependency in plant behavior and operator cognitive mechanism variety, however they are not properly captured. It is said that experiments are required to understand cognitive processes and contexts, and models are required to develop evaluation methods that take cognitive mechanisms and context dependencies into consideration. These data and conclusions, which must be true, are still startling facts. Here, we will examine three comparative studies1.Cognitive Process Analysis Experiment (in House): Joint study of Japanese BWR utilities and vendors. When we estimate Human Error Probabilities including Time Reliability Correlation, we must remind its dependency of accident scenario, culture, crew type, etc. Some insights are useful for HRA method consideration [2].2.Three HRA Model Comparison (in House): Two representative HFEs from two types of reactors in Japan, Pressurized Water Reactor (PWR) and Boiling Water Reactor (BWR), were chosen [3]. HRA method to be used are THERP method, Electric Power Research Institute (EPRI) method, and IDHEAS method [4-6] For both PWR and BWR examples, different HEPs were obtained by different HRA methods also. The scenario analysis, qualitative analysis, and task analysis must be reflected into HEPs.3.USNRC & Halden Model & Experiment Comparison: The model is judged to be underestimated compared to the experiment in one accident scenario and another case, to be overestimated. The dispersion of the calculation results by the method and experimental results are close to three digits [1, 7]. The importance of qualitative scenario analysis was widely recognized, and it became clear that the level of analysis depends largely on the knowledge and experience of analysts. This is because the lack of clear guidance in the HRA methodology has led to discrepancies in the analyst's approach [7].References[1] U.S. Nuclear Regulatory Commission, The International HRA Empirical Study, Lessons Learned from Comparing HRA Methods Predictions to HAMMLAB Simulator Data, NUREG-2127, August 2014. [2] Ujita, H., Kawano, R., Yoshimura, S., An Approach for Evaluating Expert Performance in Emergency Situations, Reliability Engineering and Systems Safety, Vol.47, pp.163- 173, 1995.[3] Hiroshi Ujita, Yoshihiro Ide, ASRAM2020, Asian Symposium on Risk Assessment and Management 2020 Online Virtual Conference, November 30 - December 2, Ideal Way of Method and Data Issues in Human Reliability Analysis (HRA) (1) Extraction of HRA Method Issues by Comparison Analysis & (2) Extraction of Method and Data Issues from Research Survey.[4] A. D. Swain, H. E. Guttmann, “Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications Final Report”, NUREG/CR-1278, August 1983. [5] S. Lewis, et. al., “EPRI/NRC-RES Fire Human Reliability Analysis Guidelines Final Report”, NUREG-1921, July 2012. [6] USNRC, EPRI, “An Integrated Human Event Analysis System (IDHEAS) for Nuclear Power Plant Internal Events At-Power Application”, NUREG-2199, March 2017. [7] Taylor, Claire. Improving scenario analysis for HRA. Probabilistic Safety Assessment and Management, PSAM. 2012.
Recent trends include the development of new data to replace old THERP data and data collection in line with new assessment requirements for exam- ple as digitization, dynamic assessment, or multi-unit support, which have been tried by Microworld, HuREX (Park, 2018), and SACADA (Chang, 2014), etc. The problem is that when evaluating HRA results, there are large variations in experimental results, large variations in evaluation results by models, and further large variations between experimental results and evaluation results. Therefore before developing a data collection system, it is desirable to clarify what causes the large uncertainty in HRA evaluation results, due to task complexity, cognitive process problem, or environmental condition change etc.
According to the results of the HRA comparison study by US Nuclear Reg- ulatory Commission (USNRC) and Halden Reactor Project, the evaluation results are significantly different when comparing the method calculations alone, comparing the experiments alone, and further comparing the method and experiment trends (USNRC, 2014). It is considered that human corre- spondence characteristics have a large context dependency in plant behavior and operator cognitive mechanism variety, however they are not properly captured. It is said that experiments are required to understand cognitive pro- cesses and contexts, and models are required to develop evaluation methods that take cognitive mechanisms and context dependencies into consideration.
Model, Method, and Data Issues in Human Reliability Analysis (HRA) 17
These data and conclusions, which must be true, are still startling facts. Here, three comparative studies will be examined.
Cognitive Process Analysis Experiment (Joint Study) The experiments were conducted with Joint study of Japanese BWR utilities and vendors. When we estimate Human Error Probabilities including Time Reliability Correlation, it must be reminded its dependency of accident sce- nario, crew type, etc. Some insights are useful for HRA method consideration (Ujita, 1995).
It is observed that HRA results depend on scenario, crew type, and culture (between crews in USA and Japan) based on the Figure 1 (USNRC, 1984). Response times vary one order due to the operator crew differences. However, which response is better, quicker or slower, is not decided, because response action is considered to be correct if the time is within the allowable range.
Figure 1: Comparison on TRC curve of ‘SRV stuck open’ between NRC experiment vs. our experiment.
Cognitive process complexity is a good qualitative measure to evaluate performance in emergency situation.
The following is the result of examining the outlook by organizing the research and practice status of HRA in Japan and referring to the latest research trends (Takahashi & Ujita, 2020).
1. First generation HRA1 evaluates omissions, while second generation HRA2 evaluates commissions. Considering the impact on plant safety, it is important to evaluate commissions. 2. The method and data during normal operation are at a sufficient level with the THERP (Swain, 1983) and EPRI (Parry, 1992) methods of HRA1 and the US data, which no new problems have been pointed out in the results of ATHEANA (USNRC, 2007).
18 Ujita
3. IDHEAS (USNRC, 2017) and ATHEANA should be used, and research is being conducted to apply them to fire event, periodic inspections, and external events, when the problem of human contextual recognition under the conditions different from the normal state become important. 4. It can be judged that the effect of long-term stress can be dealt with within the scope of conventional methods. 5. As a long-term issue, the collection of case data for HRA2 should be planned with a view to international cooperation. 6. It is necessary to plan simulator experiments, as collecting data from sim- ulator training makes it difficult to acquire information on context and cognitive mechanisms. 7. We must examine the following problems and perform lots of tasks; Is it possible to carry out experiments during outages and external events? Can the situation be realistically set? Can experiments be conducted in multiple situations?, etc. 8. Knowledge and data based on the task analysis methods and experimen- tal results should be shared in cooperation with US national laboratories and researchers such as USNRC and HAMMLAB (USNRC, 2014), as it cannot be handled in a single country.
Three HRA Model Comparison (In House) In Probabilistic Risk Assessment (PRA), we use the probability of error in operator action, therefore we need to calculate the Human Error Probabil- ity (HEP). In order that, we use Human Reliability Analysis (HRA). A lot of HRA methods have been proposed. Over 30 years ago THERP (Technique for Human Error Rate Prediction) method (Swain, 1983) was proposed, and recently EPRI (Electric Power Research Institute) method (Lewis, 2012) was proposed, and IDHEAS (Integrated Human Event Analysis System) method (USNRC, 2017) is under development. One human failure event (HFE) must have one HEP value ideally, thus even by different HRA method, it is ideal that the same value of the HEP be calculated for one HFE. In the study (Ujita, 2020), in order to test this point, we select one HFE for each PWR (Pressur- ized Water Reactor) and BWR (Boiling Water Reactor) plant and calculate the HEPs by different HRA methods with the standard steps of the methods and compare those HEPs.
Two representative HFEs from two types of reactors in Japan, PWR (JINS, 2000) and BWR (JINS, 1999), were chosen. HRA methods to be used are THERP method, EPRI method, and IDHEAS method. For both PWR and BWR examples, different HEPs were obtained by different HRA methods also. Thus, scenario analysis, qualitative analysis, and task analysis must be reflected into HEPs.
When the execution HEP is calculated, in the IDHEAS method the error probability for the task (a series of action) is assessed based on the human characteristics, on the other hand, in the THERP method and the EPRI method the error probability for each step based on the procedure is assessed for the execution HEP. Therefore, because the viewpoint of assessment for calculating the execution HEP in the IDHEAS method is different form the
Model, Method, and Data Issues in Human Reliability Analysis (HRA) 19
THERP method and the EPRI method, it is reflected in the calculated HEP values, and the resultant calculated HEP values are also considered different.
The dominant failure mode in the cognition considered in the IDHEAS method was the critical data misperceived, and in the assessment of the exe- cution the human-system interface, recovery potential, and workload were considered. In these points the IDHEAS method were also different from the THERP method and the EPRI method, which is considered to be reflected in the calculated HEP, and the resultant HEP value were also considered different.
Thus, it can be judged that research on team and organizational reliability analyses is necessary. As an approach like HRA2, first, assess the environ- mental conditions of the team and the organization as situational awareness, and consider the situations (Team Forcing Context and Organizational Forc- ing Context) that lead to the team or organizational error. And then, we can predict the behaviours of individuals in teams and organizations by setting the working hypothesis to select the most appropriate policy in that situation. That is, it is not an absolute rationality but a decision in the bounded rational- ity that can be obtained in time by utilizing the information and knowledge available under resource constraints.
We just showed the small number of analysis examples, and we explained how different the HEPs were depending on the HRA methods, however, there is a study which reinforce our work. In the study more HFEs are ana- lyzed by more HRA methods, the HEPs are also estimated by experiments, and the experiments results are compared with the calculated HEPs. In the next, the experiment and simulation performed in international study will be introduced.
USNRC & Halden Model & Experiment Comparison (International Study) The exact same accident scenario was analysed by three Human Reliability Analysis (HRA) methods, whose results differed significantly by the method, and further same tendency appeared in the different type of scenario also (Ujita, 2020). According to the results of the HRA comparison study by NRC and Halden, the evaluation results are significantly different when comparing the method calculations alone, comparing the experiments alone, and further comparing the method and experiment trends (USNRC, 2014).
It is considered that human correspondence characteristics have a large context dependency, however they are not properly captured in the analyses. It describes experiment requirement to understand cognitive processes and contexts, and model requirement to develop evaluation methods that take cognitive mechanisms and context dependencies into consideration by task analysis (Tayler, 2012).
Domestic human error data collection is discussed in joint study (Ujita, 1995), especially the experiment is highly required for considering human characteristics of cognitive process and context dependency.
In the study by the US NRC that comprehensively surveyed and evalu- ated cognitive models for HRA, approximately 300 models were discussed,
20 Ujita
in which treatments of teamwork, communication, and leadership were included (USNRC, 2016). In addition, according to the time response curves of the safety relief valve stuck open, which were carried out to compare the team cognitive process (decision and execution time) between the exper- iments by the NRC and in Japan, team structure has a great influence on team response performance (Ujita, 1995). Thus, it can be judged that research on team and organizational reliability analysis is necessary. Com- parative study has been performed by USNRC and Halden, in which 14 model calculations by various research organizations were compared first, for six scenarios for LOFW, Loss of Feedwater. And then three scenarios of these were compared between simulation results and experimental results performed in HAMMLAB of Halden.
The model is judged to be underestimated compared to the experiment in one accident scenario and another case, to be overestimated as shown in Figure 2. The dispersion of the calculation results by the method is close to three digits, and the experimental results also vary by about three digits. The dispersion of the calculation results by the method and experimental results are close to two digits.
1B: Primary Bleed and Feed before SG dry out: – The model is judged to be underestimated compared to the experiment.
2B: 1Blate recovery: – The model is judged to be overestimated.
Large context dependency was observed. Therefore, experiment is required to understand cognitive process and context, while models and these methods must consider cognitive mechanism and context dependency.
Figure 2: Comparison of the calculated results for the six accident scenarios, and comparisons with experiments in three of these scenarios.
Model, Method, and Data Issues in Human Reliability Analysis (HRA) 21
The importance of qualitative analysis was recognized such as scenario conditions and their influencing factors, error mechanism and its cause, interaction between crew and procedure, and information that crew can obtain.
A reliable and structured process and its procedure for converting quali- tative analysis results to Human Error Probability are required.
Due to the dynamic nature of the scenario, the cognitive activity of the crew should be considered, even if the procedure is followed.
The method focusing on the error mechanism and the process of the context element enables more qualitative analysis than the Performance Shaping Factor-centric method.
The importance of qualitative scenario analysis was widely recognized, and it became clear that the level of analysis depends largely on the knowledge and experience of analysts. This is because the lack of clear guidance in the HRA methodology has led to discrepancies in the analyst’s approach (Tayler, 2012).
CONCLUSION
According to the results of the HRA comparison study by NRC and Halden, the evaluation results are significantly different when comparing among the method calculations alone, comparing among the experiments alone, and further comparing the method and experiment trends. It is considered that human correspondence characteristics have a large context dependency, how- ever they are not properly captured in the analyses. Experiments are required to understand cognitive processes and contexts, and models are required to develop evaluation methods that take cognitive mechanisms and context dependencies into consideration. These data and conclusions, which must be true, are still startling facts. Here, three comparative studies will be examined.
The importance of qualitative scenario analysis was widely recognized, and it became clear that the level of analysis depends largely on the knowledge and experience of analysts.
ACKNOWLEDGMENT
The authors would like to acknowledge Dr. Kawano and Dr. Ide for their valuable discussions.
REFERENCES
Chang, Y. J., et al., “The SACADA Database for Human Reliability and Human Performance,” Reliability Engineering & System Safety, 125, 117–133 (2014). Japan Institute of Nuclear Safety (JINS) in Nuclear Power Engineering Corporation
(NUPEC), “Heisei 11 nendo PWR Puranto no shutsuryoku untenji reberu 1 PSA ni kansuru houkokusho” [Report on PWR Plant At-Power Level 1 PSA in 1999 Fiscal Year], March 2000 (in Japanese). JINS in NUPEC, “Heisei 10 nendo ABWR gata puranto no shutsuryoku untenji reberu 1 PSA ni kansuru houkokusho” [Report on ABWR Type Plant At-Power Level 1 PSA in 1998 Fiscal Year], March 1999 (in Japanese).
22 Ujita
Lewis, S., et. al., “EPRI/NRC-RES Fire Human Reliability Analysis Guidelines Final Report”, NUREG-1921, July 2012. Park, et. al., “A Comparison of Human Error Probabilities Collected from HuREX
and SHEEP Frameworks”, INL/CON-21-64816-Revision-0, November 2018. Parry, G. W., et al. “An Approach to the Analysis of Operator Actions in Probabilistic Risk Assessment”, EPRI TR-100259, June 1992. Swain, A. D., Guttmann, H. E., “Handbook of Human Reliability Analysis with
Emphasis on Nuclear Power Plant Applications Final Report”, NUREG/CR-1278, August 1983. Takahashi, M., Ujita, H., For Better Understanding of PRA -Guidance for Better Usage and Application of PRA (5); Current State of Human Reliability Analysis, Atomic Energy Society of Japan, 2020 (in Japanese). Taylor, Claire, Improving scenario analysis for HRA. Probabilistic Safety Assessment
and Management, PSAM. 2012. Ujita, Hiroshi, Kawano, Ryutaro, Yoshimura, Seiiti, An Approach for Evaluating Expert Performance in Emergency Situations, Reliability Engineering and Systems Safety, Vol. 47, pp. 163–173, 1995. Ujita, Hiroshi, Ide, Yoshihiro, ASRAM2020, Asian Symposium on Risk Assessment
and Management 2020 Online Virtual Conference, November 30 - December 2, Ideal Way of Method and Data Issues in Human Reliability Analysis (HRA) (1) Extraction of HRA Method Issues by Comparison Analysis & (2) Extraction of Method and Data Issues from Research Survey. US Nuclear Regulatory Commission, The International HRA Empirical Study, Lessons Learned from Comparing HRA Methods Predictions to HAMMLAB Simulator Data, NUREG-2127, August 2014. US Nuclear Regulatory Commission, Electric Power Research Institute, “An Inte-
grated Human Event Analysis System (IDHEAS) for Nuclear Power Plant Internal Events At-Power Application”, NUREG-2199, March 2017. US Nuclear Regulatory Commission, “Cognitive Basis for Human Reliability Analysis”, NUREG-2114, 2016. US Nuclear Regulatory Commission, “ATHEANA User’s Guide”, NUREG-1880,
2007. US Nuclear Regulatory Commission, “Safety related operator action; Methodology for developing criteria”, NUREG/CR-3515, 1984.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Error, Reliability, Resilience, and Performance
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-08-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional