Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Evaluation model for human error response training

Publication year
Digital Object Identifier10.54941/ahfe1006288

Abstract

Currently, many companies provide human error response training. The objective of this training is to gain knowledge about human error, participate in safety activities on their own, and make use of this knowledge in actual workplaces. However, many companies do not properly measure the effectiveness of education. Therefore, this study examined the creation of a model that can evaluate the effectiveness of human error response education. We created a new questionnaire tool by referring to engagement surveys currently conducted in various companies and questionnaires that measure the personality of individual characteristics. We then conducted a questionnaire at an IT company that conducts human error response education and attempts to measure its effectiveness in terms of exercise scores. Multiple regression analysis was conducted based on the results of the questionnaire and the exercise scores, and a model was created to enable measurement of the effectiveness of the education. In this study, we were able to find clues for creating an evaluation index to measure the effectiveness of human error response education. However, the measurement of effectiveness was unclear in some cases, and issues remained regarding the accuracy of the measurement. Once this model is established, it is expected that companies that have not yet been able to measure the effectiveness of their human error response training will be able to do so by using a simple questionnaire. Based on the results of this study, we plan to further expand the data and create an evaluation index that will enable more accurate measurement of the effectiveness of education.

INTRODUCTION

Currently, many companies are providing human error response training. The purpose of this training is to have knowledge about human error, participate in safety activities on their own, and make use of this knowledge in actual workplaces.

However, many companies do not properly measure the effectiveness of the training. If the effectiveness of education is not properly measured, it is not possible to evaluate whether the education is effective, it is not possible to review the education that leads to it, and participants are not aware of their own level of learning, which leads to the problem of not being able to participate in safety activities on their own initiative. Thus, measuring the effectiveness of human error response education is considered necessary to revitalize corporate safety activities.

© 2025. Published by AHFE Open Access. All rights reserved. 139

140 Shibuya and Okada

Therefore, the purpose of this study is to create a model that can evaluate the effectiveness of human error response education. Figure 1 contains an image of the research outline. The model aims to evaluate the effectiveness of education using various indicators that are considered important in companies and are being measured, such as engagement and psychological safety. In this study, we attempted to create a model that evaluates educational effectiveness using indicators of personality and engagement.

Figure 1: Outline of research.
Original PDF Figure 1:

METHOD

In this study, a questionnaire on personality and engagement was first developed and administered at one company. Then, a new index was created based on the results of the questionnaire and the indicators used in that company for measuring educational effectiveness.

At the company where the survey was conducted, a basic human factors, management skills, and conceptual skills course was held five times over a three-month period in the form of classroom lectures and problem exercises.

Regarding the questionnaire items, we focused on engagement because we believed, through interviews with company education staff, that the characteristics of employees who are effective in education are related to their high level of company loyalty and their willingness to participate in safety activities. In addition, since each employee has different characteristics, the effectiveness of education is considered to vary from person to person. Therefore, we also focused on personality and created a questionnaire that incorporated these factors.

The format of the exercises conducted at the company was not simply to measure the retention of knowledge, but also to learn how to apply the knowledge in the actual workplace and thus was an indicator for measuring the effectiveness of education. In addition, based on the results of interviews with educators who indicated that those who scored higher on the exercise tend to be more motivated to participate in on-site safety activities, we decided to treat the score of the exercise as an indicator for measuring the effectiveness of the training in this study.

Creation of an Evaluation Model for Human Error Response Training 141

VERIFICATION The questionnaire was based on the questionnaires measuring personality and engagement, respectively, and a new questionnaire with a total of 117 items was created. The questionnaire for personality was based on the Big Five theory, which was developed by Nobuhiro Murakami et al. In addition, we incorporated Ariki’s seven factors, a new decomposition and expansion of the Big Five theory into seven factors. The questionnaire was answered on a five-point scale: yes, somewhat yes, neither yes nor no, somewhat no, and no. A score of 5 was given for “yes,” with scores evenly distributed from there, and 1 for “no.” The results of the questionnaire were compared to the results of a survey conducted by Murakami Nobuhiro et al. The results of the questionnaire express the seven characteristics of personality on a scale of 0–100.

There are several types of engagement, but the three most representative lineages of engagement at work are work engagement, personal engagement, and employee engagement. Therefore, we decided to measure these three types of engagement. In addition, burnout, the opposite concept, was also measured. In terms of measurement methods, we used the 9-item Utrecht Engagement Score for work engagement, the 13-item measurement tool for personal engagement, the Q12 for employee engagement, and the JBS for burnout. As with personality, all responses to each of these questionnaires were scored on a 5-point scale of yes, somewhat yes, neither, somewhat no, and no, with yes receiving 5 points, from which the scores were evenly distributed, and no receiving 1 point. The survey results are rated on a scale of 0–100 points for each of the three types of engagement and burnout.

On the other hand, the evaluation of safety training conducted at companies is based on descriptive responses to questions such as how to respond when there are actual issues in the field. The educators score the training on a 100-point scale based on their overall judgment of whether the statements are clearly expressed, logical, versatile, and considerate of diversity.

In this study, a questionnaire was administered to employees working at one company which provides safety training. The questionnaire and exercise scores were analyzed using data from 22 employees.

RESULT

First, the entire sample was subjected to principal component analysis using personality scores, and groupings were made using the principal component scores, resulting in three groups. Figure 2 shows a plot of the principal component scores.

For variable selection, correlations were calculated between each of the seven personality characteristics, three types of engagement, and burnout scores, and the scores of the exercises. Table 1 shows the correlation coefficients between the objective variable (exercise scores) and the individual characteristics for each group.

142 Shibuya and Okada

Figure 2: Principal component score relationships.
Original PDF Figure 2:
Table 1: Correlation coefficient with exercise scores.

Table available in the original PDF paper.

Groupx Groupy Groupz Sociability −0.721 0.509 −0.0456 Proactivity 0.093 0.415 0.033 Agreeableness 0.119 −0.044 0.136 Diligence −0.175 0.292 0.808 Optimism 0.877 −0.090 −0.100 Stability 0.365 −0.053 0.465 Intelligence −0.090 0.676 0.343 Sociability −0.018 −0.138 −0.013 Proactivity −0.015 0.072 −0.130 Agreeableness 0.549 0.138 −0.238 Diligence 0.875 −0.239 −0.194

Those with an absolute correlation coefficient of 0.45 or less with the exercise scores were determined to have no correlation, and multiple regression analysis was conducted with these excluded variables as explanatory variables and the exercise scores as objective variables.

As a result of the multiple regression analysis, the coefficient of determination was determined to be 0.9 or higher, and the analysis was determined to be valid for group 1 (number of samples: 7), which will be discussed in this section. From the principal component analysis, it was found that Group 1) is characterized by high optimism.

Creation of an Evaluation Model for Human Error Response Training 143

The coefficients and multiple regression equations obtained from the results of the multiple regression analysis with the exercise scores as the objective variable, and the graph with the measured values on the horizontal axis and the estimated values on the vertical axis are shown below.

Table 2: Results of multiple regression analysis.

Table available in the original PDF paper.

Coefficient Intercept 2.863 Sociability −1.098 Optimism 0.075 Stability 1.154 Employee engagement 0.015 Burnout 0.712

y = −1.10x1 + 0.08x2 + 1.15x3 + 0.02x4 + 0.71x5 + 2.86 (y: exercise scores x1: Sociability x2: Optimism x3: Stability x4: Employee engagement x5: Burnout)

Equation 1: Multiple regression equation.

Figure 3: Relationship between measured and estimated values.
Original PDF Figure 3:

The adjusted coefficient of determination adjusted for degrees of freedom was 0.995.

Then, the elements used as explanatory variables in the multiple regression analysis were analyzed and structured using ISM (Interpretive Structure Modeling). The figure shows the relationships among the elements as a result of ISM (Interpretive Structure Modeling).

144 Shibuya and Okada

Figure 4: Results of ISM.
Original PDF Figure 4:

The results indicate that the relationship between personal characteristics and educational effectiveness for those with high optimism is positive for optimism, stability, employee engagement, and burnout scores, while sociability is negatively related.In particular, employee engagement was found to be an important factor for educational effectiveness, followed by optimism and burnout, stability, and sociability, in that order of importance to educational effectiveness.

Employee engagement is a measure of employees’ willingness to contribute to the organization, which focuses on their attachment to their organization and employer and their contribution behavior and has been proven to be related to organizational performance. It is convincing that a high state of such characteristics has a positive impact on the results of exercises that ask how to respond to real problems when they occur.

On the other hand, sociability was negative, resulting in a negative impact on the exercise scores. Many of the employees surveyed in this study were in middle management positions, and it is possible that they have become more cautious and less accustomed to management due to the increased number of situations in which they have to make logical decisions in the face of various challenges. However, we believe that this can be said of the companies that conducted the survey and cannot be regarded as general. The results of the survey indicate that it is not necessarily correct to judge the negative relationship between sociability and exercise points based on these results alone, and therefore, it is necessary to improve the accuracy of the results.

In addition, the grouping conducted from the principal component scores produced some groups with results and some groups without results. The reason for this is that the first principal component has a high contribution ratio to many items, and the grouping was not done well, although this is not a matter of speculation due to the lack of data. More data is needed to improve these results.

Creation of an Evaluation Model for Human Error Response Training 145

CONCLUSION

In this study, we were able to find clues regarding the creation of an evaluation index to measure the effectiveness of human error response education, using a questionnaire that measures personal characteristics.

However, this study only analyzed a small number of groups, and it is still unclear what personalities, and individual characteristics influence the effectiveness of the education. The small number of data and the fact that data from employees of a single company was used made the measurement of effectiveness unclear in some cases, and the evaluation indexes were still insufficient. In the future, we will conduct surveys of various industries and occupations to expand the data, create and verify models, and improve evaluation indicators for more accurate measurement.

REFERENCES

Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of school psychology, 44(5), 427–445. Attri, R., Dev, N., & Sharma, V. (2013). Interpretive structural modelling (ISM)

approach: An overview. Research journal of management sciences, 2319(2), 1171. Barlett, C. P., & Anderson, C. A. (2012). Direct and indirect relations between the Big 5 personality traits and aggressive and violent behavior. Personality and individual differences, 52(8), 870–875. Chika Ariki (Keio University), Yusaku Okada (Keio University). An Evaluation on

the Working Adaptability by Analyzing User’s Personality. 2006 International Ergonomics Association 16th World Congress on Ergonomics. Gallup. Gallup’s Q12 Employee Engagement Survey. Gallup. Available: Gallup’s Q12 Employee Engagement Survey - Gallup. Hayes, N., & Joseph, S. (2003). Big 5 correlates of three measures of subjective

well-being. Personality and Individual differences, 34(4), 723–727. Kular, S., Gatenby, M., Rees, C., Soane, E., & Truss, K. (2008). Employee engagement: A literature review. May, D. R., Gilson, R. L., & Harter, L. M. (2004). The psychological conditions of

meaningfulness, safety and availability and the engagement of the human spirit at work. Journal of occupational and organizational psychology, 77(1), 11–37. Murahashi, M., Okada, Y. (2023). Method for Enhancing Evaluation of the Human Error Probability in Disaster Risk Assessment for Industrial Plants. In: Ronald Boring (eds) Human Error, Reliability, Resilience, and Performance. AHFE (2023) International Conference. AHFE Open Access, vol. 82. AHFE International, USA. http://doi.org/10.54941/ahfe1003556 Okada, Y., Aoyagi, T. (2023). A Failure Event Virtual Learning Method that Replaces

Field Experience and Its Effectiveness Measurement. In: Ravindra S. Goonetilleke and Shuping Xiong (eds) Physical Ergonomics and Human Factors. AHFE (2023) International Conference. AHFE Open Access, vol. 103. AHFE International, USA. http://doi.org/10.54941/ahfe1003033 Ruriko Yano (Keio University), Yusaku Okada (Keio University). 2nd International Conference on Applied Human Factors and Ergonomics. A Proposal of Human Error Tendency Estimation Method on Human Personality - For Nurse’s Work in Hospital.

146 Shibuya and Okada

Saks, A. M., & Gruman, J. A. (2014). What do we really know about employee engagement? Human resource development quarterly, 25(2), 155–182. Shibuya, Y., Okada, Y. (2024). Methods of Measuring the Effectiveness of On-

site Human Error Response Training Based on Employee Engagement Indicators. In: Tareq Ahram and Waldemar Karwowski (eds) Human Factors in Design, Engineering, and Computing. AHFE (2024) International Conference. AHFE Open Access, vol. 159. AHFE International, USA. http://doi.org/10.54941/ ahfe1005797

Cover of the proceedings volume Human Error, Reliability, Resilience, and Performance

Explore the complete digital volume

Human Error, Reliability, Resilience, and Performance

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-49-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →