Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Methods of Measuring the Effectiveness of On-site Human Error Response Training Based on Employee Engagement Indicators

Graduate School of Science and Technology, Keio University, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1005797

Abstract

One of the important issues in the field of safety management is effective human error prevention education for on-site staff. Currently, many sites not only provide basic human error response training such as confirmation and thorough implementation of basic actions, but also various education methods such as work improvement that takes human factors into consideration, active follow-up with team members, and communication methods that lead to accurate reporting, contact, and consultation. However, the effectiveness of such human error prevention education has only been measured qualitatively.Therefore, this study focused on the engagement of workers and examined a method to multifacetedly evaluate each trainee's attitude toward safety activities before and after the course, including the individual characteristics of the workers (personality such as personality analysis). A questionnaire was designed with work engagement, personal engagement, burnout, employee engagement, psychological safety, personality (Big Five theory), and attributes (job type, position, years of experience) as basic parameters, and a model was created using machine learning to evaluate the following four main factors based on the answers. The four main factors are: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being). These indicators were evaluated on a 10-point scale. The effectiveness of the method for measuring the effectiveness of on-site human error prevention training based on the employee engagement index obtained in this study was verified through a survey at several hospitals in Japan. However, there are cases where the effectiveness measurement is unclear, and we are currently continuing to improve the accuracy by expanding the data.

INTRODUCTION

Many companies provide safety education, such as safety management training. The two purposes of education mentioned here are to improve safety-related knowledge and to develop human resources who understand the essence of safety education so that they can actively participate in safety activities and apply it to their practical work. The former is measured quantitatively using assignments, reports, and tests in the course. The latter is particularly important, but it is mostly measured qualitatively based on the

© 2024. Published by AHFE Open Access. All rights reserved. 2289

2290 Shibuya and Okada

instructor’s subjectivity, and no quantitative measurement method has been established.

The purpose of this study was to assist in measuring the effectiveness of education and the subsequent positive attitude toward safety activities, and a model was created that can assess the following four main factors: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being). These indicators were evaluated on a 10-point scale based on the results of the questionnaire responses regarding the personal characteristics and engagement.

In human resource development, an important perspective is the spirit of company loyalty. This is because even when safety training is provided, the response to accidents and emergencies varies greatly depending on the degree of company spirit. This is due in large part to engagement, which is the connection between the company and the individual. Therefore, I wanted to use engagement to measure the part related to human resource development, which is the purpose of education, and created a questionnaire on engagement.

METHOD

Below are details on personality and engagement, and the sequence of steps from measuring personality and engagement on the questionnaire to calculating the main factor scores.

Extraction of Factors Personality and engagement were categorized in detail.

Personality was based on the Big Five theory. It is a common trait theory model that attempts to comprehensively understand personality through five trait factors.

The five traits in the Big Five Theory are mainly Extraversion, Emotional Stability, Intelligence, Agreeableness, and diligence.

In addition, the Big Five theory has been expanded to seven factors by incorporating the perspective of the relationship between personality and human behavior at work into the Big Five theory, which is called Ariki’s Seven Factors. Among the five factors, “extroversion”is classified into “sociability” and “proactivity”, and “emotional stability” is classified into “stability” and “optimism”. From the above, personality was classified into seven factors.

Also, several types of engagement exist. Mainly, there are three typical systems of engagement at work: work engagement, personal engagement, and employee engagement.

In addition, burnout exists as a counter-concept to engagement. I used these four in this study because each engagement and burnout has a slightly different meaning, and the use of multiple engagements allows us to measure engagement from different angles.

The eleven elements of personality and engagement were used to measure the four main factors. Table 1 lists the eleven factors and their descriptions.

Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2291

Table 1. Analysis categories.

Table available in the original PDF paper.

Personality Sociability A disposition in which mental energy is directed toward an object, such as another person or thing. Lively, energetic, talkative, adventurous, and aggressive personality Proactivity Agreeableness Indicates the degree of cooperation with one’s surroundings. Warm, friendly to all, pleasant, humane, generous, and cooperative personality. Diligence It is a dimension related to self-discipline, which indicates solidity in planning and execution of things. Responsible, conscientious, energetic and diligent in work and study. Optimism A tendency to show emotional irritability. A tendency to be less likely to cause mental confusion in response to stress. An ideal personality that is stable in mood, free of complaints, easy going, and not jealous. Stability Intelligence Curious, has a wide range of knowledge, analyzes and thinks things through. Thoughtful, creative, and intelligent. Engagement Work engagement

A positive and fulfilling state of mind in which one is energetic (willing to persevere), enthusiastic (ambitious and challenging in one’s work), and absorbed (focused on one’s work) in one’s work. Personal engagement

A state of feeling a sense of meaningfulness (a sense of being able to expect a return on one’s investment in performing one’s role), security (a sense of being able to use and express oneself), and availability (a sense of having the necessary resources to perform one’s role) when performing one’s role at work. Employee engagement

The voluntary attitude and behavior of employees who are willing to contribute to the organization. Burnout Burnout with symptoms of emotional exhaustion (feeling physically and mentally drained and not wanting to do anything), depersonalization (no longer caring about others), and decreased sense of personal accomplishment (unable to take pleasure in their work and underestimating the importance of their work).

Structuring the Factors Next, a questionnaire was administered regarding the relationship between the 11 elements and each of the four primary factors. In this case, the questionnaire was administered to one subject who had sufficient knowledge of personality and engagement. The questionnaire was a pairwise comparison of the elements, asking which was more related to the main factor, element

2292 Shibuya and Okada

i or element j. This questionnaire was taken for all elements and the results were put into the ISM(Interpretive Structure Modeling) and structured.

Figures 1∼4 show the ISM structural model derived from the questionnaire results for each of the four main factors: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being).

The factors at the top indicate that they were rated as having a greater impact on the main factor.

Figure 1: Structuring the factors that form “loyalty to work”.
Original PDF Figure 1:
Figure 2: Structuring the factors that form “desire for growth”.
Original PDF Figure 2:

Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2293

Figure 3: Structuring the factors that form “desire to contribute to the company”.
Original PDF Figure 3:
Figure 4: Structuring the factors that form “well-being”.
Original PDF Figure 4:

To score the main factors, an influence score was given. Elements of the top row of the figure 1∼4, which was more related to each main factor, was assigned a score of 1, and the elements of the second upper row was assigned 0.5, and this score was used as the influence score. After the third upper row, the main factor was not considered relevant, and an influence score was assigned to the second upper row.

2294 Shibuya and Okada

Calculation of Main Factors Score The relationship between the 11 factors and the main factor is quantified. The scores for each factor are obtained by taking a questionnaire.The questionnaire consists of 70 personality questions and 43 engagement questions. After each person completes the questionnaire, they receive a radar chart showing their personality and engagement scores, and comments corresponding to their scores on the personality factors.

These comments were prepared after organizing each of the personality traits and the behaviors that are considered likely to result from various psychological perspectives.

For the personality analysis, the factor structure of the results of Murakami’s main 5-factor personality test was used to score everyone’s personality building factors, with a minimum score of 0 and a maximum score of 100.

In addition, there are different ways to measure engagement. The Utrecht Work Engagement Scale was used for work engagement, the 13-item measurement tool for personal engagement, the Q12 for employee engagement, and the JBS scale for burnout. For this survey, all responses were scored with a minimum score of 0 and a maximum score of 100. For burnout, a score closer to 100 indicates that the employee is not in a state of burnout.

The figure 5∼7 shows a portion of the questionnaire, the radar chart that appears after the survey, and comments.

Figure 5: Questionnaire sheet.
Original PDF Figure 5:

The personality trait scores and engagement scores (xi) obtained from the questionnaire responses were multiplied by their corresponding impact scores (αi) and summed (y = P αixi). All the scores were then re-scaled to a 10-point scale, which was used as the main factor score.

Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2295

Figure 6: An example of inidividual score evaluated by the proposed method.
Original PDF Figure 6:
Figure 7: An example of advice document according to the individual score.
Original PDF Figure 7:

Verification In this study, 21 employees of a company that conducts human error education and human error response activities were surveyed about their personality and engagement, and their main factor scores were calculated.

The employees who completed the questionnaire were divided into two groups: those who had received individualized active learning and those who had not. The average of the main factor scores calculated for each group was averaged and a graph was created. In the figure5, the blue bars indicate employees with individualized active learning (8 employees) and the red bars indicate employees without it (13 employees). Figure 5 shows the results.

The vertical axis is the average of each main factor score, on a 10-point scale, with a score closer to 10 indicating higher loyalty to work, desire for growth, desire to contribute to the company, and sense of happiness (well- being).

2296 Shibuya and Okada

Figure 8: Average of main factor scores.
Original PDF Figure 8:

The figure shows that the group receiving individualized active learning had higher overall scores than the group not receiving this form of education. This suggests that this form of education may increase the spirit of company loyalty. In fact, interviews with the employees receiving the training revealed that they were positive about their work and the company, commenting that they were able to rely on the organization when they had problems in their fulfilling work, indicating that the measurement was effective.

In this study, we attempted to measure educational effectiveness through the scores of the four main factors. In the future, I intend to further analyze the relationship between educational methods and educational effects in more detail. First, I would like to characterize each educational method by evaluating its involvement in the following seven axes. The seven axes are (1) expressiveness, (2) logic, (3) diversity, (4) comprehensiveness, (5) ripple effect, (6) adaptability to the field, and (7) versatility. The method of characterization was determined by morphological analysis of the explanatory documents of the educational methods.

Finally, I would like to create a matrix of “advice for future training plans” based on the relationship between “each worker’s four main factors” and “each worker’s degree of involvement in the seven axes with respect to the training content already learned.”

This matrix allows each worker to understand their current achievement status and obtain the outline of the next step of training.

CONCLUSION

One of the most important issues in safety management is effective human error prevention education for on-site personnel. A sense of company loyalty and volunteerism are essential characteristics for personnel who play a central role in safety activities. In this study, we measured these characteristics

Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2297

of belongingness to the company, work engagement, personal engagement, burnout, employee engagement, psychological safety, personality (Big Five theory), and attributes (job title, position, and years of experience) to determine the following: (1) loyalty to the job, (2) desire to grow, (3) desire to contribute to the company, and (4) happiness (wellbeing). The effectiveness of the developed method was verified through surveys at several hospitals and companies in Japan. However, the practicality of the method cannot be achieved unless it is connected to the proposal of guidelines for improvement of educational methods through effectiveness measurement. In the future, we plan to establish a method for characterizing educational methods based on the seven characteristic indices described in the previous chapter and develop it into a method that can evaluate and improve competency education practiced in companies.

REFERENCES

Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of school psychology, 44(5), 427–445. Attri, R., Dev, N., & Sharma, V. (2013). Interpretive structural modelling (ISM)

approach: an overview. Research journal of management sciences, 2319(2), 1171. Barlett, C. P., & Anderson, C. A. (2012). Direct and indirect relations between the Big 5 personality traits and aggressive and violent behavior. Personality and individual differences, 52(8), 870–875. Chika Ariki (Keio University), Yusaku Okada (Keio University). An Evaluation on

the Working Adaptability by Analyzing User’s Personality. 2006 International Ergonomics Association 16th World Congress on Ergonomics. Gallup. Gallup’s Q12 Employee Engagement Survey. Gallup. Available: Gallup’s Q12 Employee Engagement Survey - Gallup. Hayes, N., & Joseph, S. (2003). Big 5 correlates of three measures of subjective

well-being. Personality and Individual differences, 34(4), 723–727. Kular, S., Gatenby, M., Rees, C., Soane, E., & Truss, K. (2008). Employee engagement: A literature review. May, D. R., Gilson, R. L., & Harter, L. M. (2004). The psychological conditions of

meaningfulness, safety and availability and the engagement of the human spirit at work. Journal of occupational and organizational psychology, 77(1), 11–37. Murahashi, M., Okada, Y. (2023). Method for Enhancing Evaluation of the Human Error Probability in Disaster Risk Assessment for Industrial Plants. In: Ronald Boring (eds) Human Error, Reliability, Resilience, and Performance. AHFE (2023) International Conference. AHFE Open Access, vol 82. AHFE International, USA. http://doi.org/10.54941/ahfe1003556 Okada, Y., Aoyagi, T. (2023). A Failure Event Virtual Learning Method that Replaces

Field Experience and Its Effectiveness Measurement. In: Ravindra S. Goonetilleke and Shuping Xiong (eds) Physical Ergonomics and Human Factors. AHFE (2023) International Conference. AHFE Open Access, vol 103. AHFE International, USA. http://doi.org/10.54941/ahfe1003033 Ruriko Yano (Keio University), Yusaku Okada (Keio University).2nd International Conference on Applied Human Factors and Ergonomics. A Proposal of Human Error Tendency Estimation Method on Human Personality - For Nurse’s Work in Hospital-. Saks, A. M., & Gruman, J. A. (2014). What do we really know about employee

engagement?. Human resource development quarterly, 25(2), 155–182.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →