Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

End-user engagement in developing Virtual Reality Training systems for Industry 5.0

1 Cranfield University, Cranfield, MK43 0AL, UK ↗
2 Tecnalia, Basque Research and Technology Alliance (BRTA), C/ Leonardo Da Vinci 11, 01510 Miñano, Araba, Spain ↗
3 Laboratory for Manufacturing Systems and Automation, University of Patras, 26504, Greece ↗
Publication year
Digital Object Identifier10.54941/ahfe1006452

Abstract

Virtual reality (VR) training has emerged as an effective means of training workers in safety procedures and in learning new technologies and roles. In the field of industrial human-robot collaboration (HRC), VR can be a promising means of training and upskilling industrial operators. Operators familiarize themselves with robots through a virtual simulation before in-person interaction with the robots. As with any technology, the development of VR training systems should include user-centred research to ensure user friendliness of the training interface, and effective use and acceptance by the end-user. The following study presents a methodology for engaging end-users in the initial stages of development of VR training systems. The end-user engagement took place as an online workshop with 5 industrial operators from the automotive industry. They first discussed their current training practises and expectations from future training for HRC. Following this, they were shown first-person videos of a user interacting with the VR training system and their opinions and feedback regarding the system and the process of VR training were captured. The operators’ feedback about the VR training system, and integration of this feedback are discussed in detail.

INTRODUCTION

While Industry 5.0 involves designing new smart production systems that leverage human-robot collaboration (HRC), it also involves centralizing the wellbeing of the worker and prioritizing workforce sustainability (Industry 5.0 - European Commission, 2024). This human-centred approach of Industry 5.0 means empowering operators to perform their new roles of working with collaborative robots though effective training and upskilling. Virtual reality (VR) has emerged as an effective means to train operators for smart production systems, as it can help them gain confidence in interacting with the robots in a virtual simulation, prior to in-person HRC. Involving the industrial operators in the design and development process of these VR

© 2025. Published by AHFE Open Access. All rights reserved. 81

training systems ensures that their needs and expectations from training will be met, and that the VR training solutions are user-friendly. The following paper presents a study that employed user-engagement workshop to capture operators’ feedback and preferences for the VR training systems under development.

BACKGROUND

Industrial HRC and Training Industrial HRC is being increasingly adopted in smart-manufacturing and production systems. HRC in industrial applications leverages the strengths of a robot and human operator accordingly, wherein a robot performs tasks that are physically challenging, non-ergonomic, repetitive or mundane and the human operator performs those aspects of the tasks that require creativity, problem-solving, and use of tacit knowledge (Johnson et al., 2019). However, the success of HRC requires industrial operators to feel psychologically and physically safe working alongside or with a robot. One of the main challenges to successful industrial HRC is ensuring operators’ trust in the robot (Hancock et al., 2011; Kopp, Baumgartner and Kinkel, 2021). This lack of trust stems from operators’ concerns about a robot’s reliability, their own ability to work with the robots, and having clearly defined safety areas and protocols during HRC (Sashidharan et al., 2024).

These concerns held by the operators can be addressed and mitigated through training. Charalambous et al., (2015) found that the provision of training to operators is an important enabler of successful HRC, as it allows operators to gain confidence and ownership of the new system, reduces negative attitudes towards the system and increases their acceptance. Training with the robots and other new machines can provide operators with the opportunity to familiarize themselves with the robot’s performance and understand the safety protocols of working with the robot. Ideally, this training should be over an adequate period of time wherein the operators get multiple chances to train with the robot. However, repeated in-person training with a robot often requires a partial or complete halt of activities in the shopfloor, causing a fall in productivity, a loss of time and significant costs due to machine idle time (Dianatfar, Latokartano and Lanz, 2020). There is also a possibility of misusing or damaging the robots due to inexperience of the newly recruited operators.

In light of these drawbacks, VR training can be an effective solution to help operators familiarize themselves with the robots and the new collaborative industrial tasks. In other industries, VR training has had successful outcomes in training operators for high-risk jobs such as construction (Hilfert, Teizer and König, 2016; Vahdatikhaki et al., 2019; Song et al., 2021), mining (Pedram et al., 2022; Pamidimukkala and Kermanshachi, 2023), and chemical substances production (Chan et al., 2023). VR training in construction was found to yield higher enjoyment, awareness of risk hazards and a higher commitment to safety than lecture based training (Rey-Becerra et al., 2023), and has also been found to positively affect

End-User Engagement in Developing Virtual Reality Training Systems for Industry 5.0 83

intrinsic motivation, absorption in the task, learning, and behavioural intention (Chan et al., 2023).

Research on VR training for industrial HRC is limited. Existing studies show that operators who underwent VR training prior to teleoperation of a robot showed significantly higher trust levels in the robot as well as higher levels of self-efficacy and situational awareness, compared to those who were trained through a lecture (Adami et al., 2022, 2023). VR training has also been found to effectively teach safety protocols for industrial HRC tasks (Shayesteh et al., 2023). While there are some studies that evaluate the final outcomes of VR training, there has been no investigation into the expectations and preferences industrial operators have from VR training, and how VR training can be developed to meet their needs. The current study presents a method to capture end-user’s feedback for VR training systems under development.

User-Centred Research for VR Training Development Any technology intended to be user-friendly and widely accepted requires end-user involvement throughout its development. This approach, known as user-centred design, involves an iterative process where prototypes are evaluated by end-users, and their feedback is used to refine the technology (Pais et al., 2022).

Employing a user-centred design approach to developing the VR training systems is crucial for effective development of the system. It is important to capture operators’ feedback on the VR training systems during initial stages of development in order to understand their needs from training and assess if the VR training systems are meeting these needs. The operators can assess if the initial prototypes of the VR training systems are effective in conveying training information and seem easy to use. Additionally, research shows that operator participation in the implementation of collaborative robotic applications is an enabler of successful industrial HRC (Charalambous, Fletcher and Webb, 2015). Therefore, involving operators in the design of their training for HRC, can also increase their openness to the HRC system as a whole.

There has been no research yet that employs a user-centred approach to designing and developing the VR training systems. The current study aims to meet this gap, by proposing a user-centred methodology for engaging end- users in the development of VR training systems for industrial HRC training.

METHOD

Use Case The end-user engagement study was conducted with operators in the automotive industry who worked on the task of manual sanding and polishing of automotive draw dies. The manual process involves operators inspecting the die for defects and using the appropriate sanding tools for polishing the defected area.

This manual task is being redesigned to include collaborative robots that actively assist human operators in the die polishing processes. In the new process, the operators inspect the die for wear or tear and damage and mark the area for repair with a smart pen. The operator determines the defect type and tools required for sanding. Once the damaged area is marked, the robot will take over and polish the marked area. Following this, the operator will inspect the robot’s work and confirm if the polishing has been performed adequately.

Participants Participants in the workshop were five male operators working in the automotive industry. Their roles were a process code engineer, a CAD software programmer, and three tool and die engineers. All of the operators’ roles are linked to the sanding and polishing of draw dies.

MATERIALS

Future HRC Solutions The future HRC redesign for the sanding and polishing task were presented to the operators through a power point presentation. The presentation showed pictures of simulated graphics of the proposed future industrial process with explanations.

Discussion Questions for Current Training and Expectations for Future Training

1. How does current training take place?

a. Is it on-site or off-site b. Is it done individually or in a group? c. How long does it take? d. What do you like about it? e. Would you change anything about it?

2. How would you like to be trained in the future? 3. How do you envision the training for working with the robots?

Videos of VR Training System Two first-person videos of a user engaging with two modules in the VR training system designed for the use-case:

1. A Tutorial module which is designed to introduce the users to the system’s basic functions. This module teaches users the essential tools and gestures required for effective VR interaction. 2. A Training module which is a VR simulation that trains operators in the HRC task. The operators go through a VR simulation of the full task and is provided with information on the process for assistance.

End-User Engagement in Developing Virtual Reality Training Systems for Industry 5.0 85

Discussion Questions for the VR Training System

1. Do you think it would be easy or challenging to use the training interface? What do you think will be easy and challenging? 2. Did you find anything confusing? 3. How do you feel about the training system overall? What did you like and dislike about it? 4. What do you think of the colour scheme used in this interface? Does it help or hinder your understanding of the screen? 5. How do you feel about the text size and readability on this screen? Would you suggest any changes?

The System Usability Scale (SUS) The SUS is a questionnaire designed to evaluate a user’s perception of how usable a system is. It has been designed to be suitable for a wide range of systems and applications. The questionnaire consists of 10 items are rated on a 5-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree”. The SUS yields a score that provides a global view of subjective usability of the system. (Brooke, 1995). For the current study, the SUS items were adapted to assess how usable the operators perceive the system to be, as opposed how usable it is, as the operators were not directly using the system and were watching someone else use it.

Procedure The workshop took place online, and the operators joined the MS Teams link that was sent by the researchers one week prior to the workshop date. The operators and the use case leader were all present in one room from where they joined the online workshop.

The session commenced with the researchers introducing themselves to the participants, following which, the researchers explained what the operators would be asked to do, their right to refrain from participation and withdraw, and the option to abstain from answering any questions that made them uncomfortable. The information briefing was translated by the use case leader for the participants and the operators’ verbal consent was obtained. The researchers then stated the purpose and agenda of the workshop.

The workshop then began with researchers asking the operators to give a brief introduction of the roles they performed, and the researchers giving an overview of the future HRC solutions for the use-case. Following this, the discussion took place in three steps. The operators were first asked about their current training for performing the manual task of sanding and polishing draw dies. They were then asked to discuss their expectations and visions for future training with the HRC solutions involving the new smart technologies. Finally, after the purpose and method of VR training was introduced, the operators were shown the two first-person videos of a user engaging with the VR training system. This was followed with a semi- structured discussion to capture the operators’ feedback about the training interface. The operators were then sent a link to answer the System Usability Scale. However, due technical difficulties, the operators could only answer the

survey after the workshop had concluded. The researchers then concluded the workshop and thanked the operators for their participation.

FINDINGS Details About Current Training

Currently, training takes place individually, where the operator trains with a “master” of the process, and the training takes place over one month. The training includes familiarization with the various materials and their functions, assessing the defect and determining the polishing requirements and learning the task of polishing the area accordingly.

Expectations for Future Training The operators said that they envision training with robots to take place offsite initially, and then onsite. They stated the importance of retaining the training in manual sanding and polishing, as they would require that skill to verify the robot’s sanding performance in the HRC solution. They also stated that the training should include comprehensive details about the robot’s performance, for e.g., each of the steps the robot will follow, and the specifications of the tools the robot will use.

Feedback on the VR Training System and Interface After viewing the first-person videos of a user interacting with the VR training environment, the operators had positive reactions stating that “everything looks nice”, and it would be “useful”. They specifically appreciated that the Tutorial module to learn gestures in the VR environment, stating that it would be very useful. They mentioned that the training in the VR environment would help operators feel more comfortable with the robot.

When asked if the operators would find anything challenging or confusing about the VR training interface, the operators mentioned that while the interface seemed easy to use, simply getting used to wearing the VR glasses may be challenging as it is “something new and strange”. They also mentioned that wearing the VR glasses along with the typical gear of their daily tasks would likely be uncomfortable. They suggested that the initially, training should be in a private room where the operators can use the VR system without the possibility of being visible to others. They said that an operator may feel strange and shy if someone had to be in the VR environment in front of other people.

Another suggestion they made was to include an option to add custom gestures in the VR system. They explained that if an operator has a suggestion for a new gesture to interact in the VR environment, the operator should be able to add this gesture after receiving approval from their line manager or supervisor.

With regard to the colours and text used in the training interfaces, the operators responded that there was no issue with the colours. However, the text used in some of the infographics was not legible. They would like the

End-User Engagement in Developing Virtual Reality Training Systems for Industry 5.0 87

option to customize the size of the text as different operators have different preferences for text sizes.

System Usability Scale Results The SUS yields a composite score that falls between 0 – 100, representing the overall usability of the system by considering issues such as ease of use, need for prior training, need for support while using the system, and opinions on system design (Brooke, 1995). The mean of SUS score for 5 operators was 83.5 (SD = 7.4). Considering the total score the SUS can yield is 100, the operators’ scores indicate a perception of high system usability. Previous research has indicated that SUS scores that average 71 and above are correlated with a “Good” experience, while scores averaging 85 and above are correlated with an “Excellent”experience with the system (Bangor, Kortum and Miller, 2009). Based on this grading, the operators’ scores indicate a usability level of near excellence.

FEEDBACK DISCUSSION AND INTEGRATION The feedback from the operators was relayed to the developers of the VR training system. The feedback was reviewed and ways to integrate the end- user feedback into the VR training interfaces were discussed, as explained below, and summarized in Table 1.

With regard to operators stating the importance of retaining training in the manual task, the developers concluded that they could address this need by ensuring that VR training interface instructs operators to provide confirmation of completing manual training. Only with this confirmation can the operators begin the HRC training module.

The technology developers stated that developing an option for operators add their own gesture to use in the VR interface would be challenging. However, as an initial step towards addressing this request, the VR training system can present information to the user about providing feedback about their VR experience. For e.g., an infographic giving the operators an email address to correspond with. There would need to be a line of communication in place to ensure that this feedback is received by the technology developers who can discuss issues like adding gestures, with the use-case leaders.

The operators mentioned that they should be provided with more information about the robot’s functions – for example the steps the robot will follow, the tools it will use, the specifications of those tools, etc. This can be done by adding infographics in the VR environment. During future in-person testing of the VR interfaces with operators, they will be asked to explain in greater detail about the robot information they would like to receive while training. Similarly, with regard to the concern about being uncomfortable with the VR headset and requiring a separate room to perform the VR training, these two points will be investigated during future in-person usability testing. Since the operators only saw a video of the VR training experience and did not experience it in person, it is crucial to revisit these concerns after they test the VR system in person. The VR developers also stated the user can move towards a text box in the VR environment, thereby

making text more legible. Since the operators could not view this through the video they saw in this workshop, they raised concerns about the text size. Therefore, this too shall be investigated in a future in-person usability testing session.

Although an average score of 83.5 on the SUS indicates that the operators perceived the VR system to be highly usable, it is important to consider that this score was given after watching a video of the system. It is crucial that the SUS is re-administered after operators use and interact with the VR system in-person, since a first-hand experience with the system is necessary for more reliable usability evaluations.

Table 1: Operators’ feedback and feedback integration.

Table available in the original PDF paper.

Operators’ Feedback Feedback Addressal and Integration Training on manual task should VR training interface can ask operators continue even after implementation of HRC task

The VR training system can present

to provide confirmation of completion of manual training. Only with this confirmation can the operators begin the HRC training module. Option for operators or line managers/supervisors to add their own gesture in the VR system.

This can be done by adding infographics

information to the user about providing feedback about their VR experience. For e.g., an infographic giving the operators an email address to correspond with where operators can detail a request for a new gesture. This feedback will be received by the technology developers. The operators mentioned that they in the VR environment. Future in-person testing of the VR interfaces will capture details of information required by operators should be provided with more information about the robot’s functions – for example the steps the robot will follow, the tools it will use, the specifications of those tools, etc.

There could be potential discomfort

To be investigated further during

with VR headset

in-person usability testing A separate, private room to perform the

To be investigated further during

VR training will be preferred

in-person usability testing Text size should be more legible and

To be investigated further during

customizable

in-person usability testing

CONCLUSION

Overall, the operators had positive reactions to the proposal of the VR training system. They agreed that it would be a useful way for operators to familiarize themselves with the actual task with the robot. While they had a few opinions regarding the interface with regard to the option of customizing gestures and text size, most of their feedback revolved around the implementation of the training (setting) and the content of the training (training on manual sanding, more information about the robot).

End-User Engagement in Developing Virtual Reality Training Systems for Industry 5.0 89

Implementing these recommendations would help improve the training solution in favour of operators’ comfort and acceptance.

The current study is a very initial stage of capturing user feedback where operators simply viewed the VR system being used and gave their impressions. The next step is to accordingly implement this feedback where possible and then reiterate the workshop with more developed prototypes. In these future iterations of the workshops, the operators will be asked to use and interact with the VR training system in-person and carry out various functions. These in-person tests of user interaction will help provide greater clarity to concerns raised from the current study, and aim to assess usability, functionality and user-experience of the VR training system.

ACKNOWLEDGMENT

The work has been conducted on the CONVERGING project (101058521 HORIZON-CL4-2021-TWIN-TRANSITION-01-01 – Innovation action) and the authors would like to thank all consortium members for their collaboration.

REFERENCES

Adami, P. et al. (2022) ‘Impact of VR-Based Training on Human–Robot Interaction for Remote Operating Construction Robots’, Journal of Computing in Civil Engineering, 36(3), p. 04022006. Available at: https://doi.org/10.1061/(ASCE) CP.1943–5487.0001016. Adami, P. et al. (2023) ‘Participants matter: Effectiveness of VR-based training on

the knowledge, trust in the robot, and self-efficacy of construction workers and university students’, Advanced Engineering Informatics, 55, p. 101837. Available at: https://doi.org/10.1016/j.aei.2022.101837. Bangor, A., Kortum, P. and Miller, J. (2009) ‘Determining what individual SUS scores mean: Adding an adjective rating scale’, J. Usability Studies, 4(3), pp. 114–123. Brooke, J. (1995) ‘SUS: A quick and dirty usability scale’, Usability Eval. Ind., 189. Chan, P. et al. (2023) ‘Study of motivation and engagement for chemical laboratory

safety training with VR serious game’, Safety Science, 167. Available at: https:// doi.org/10.1016/j.ssci.2023.106278. Charalambous, G., Fletcher, S. and Webb, P. (2015) ‘Identifying the key organisational human factors for introducing human-robot collaboration in industry: An exploratory study’, The International Journal of Advanced Manufacturing Technology, 81(9–12), pp. 2143–2155. Available at: https:// doi.org/10.1007/s00170-015-7335-4. Dianatfar, M., Latokartano, J. and Lanz, M. (2020) ‘Concept for Virtual Safety Training System for Human-Robot Collaboration’, Procedia Manufacturing, 51, pp. 54–60. Available at: https://doi.org/10.1016/j.promfg.2020.10.009. Hancock, P. A. et al. (2011) ‘A Meta-Analysis of Factors Affecting Trust in Human-

Robot Interaction’, Human Factors, 53(5), pp. 517–527. Available at: https:// doi.org/10.1177/0018720811417254. Hilfert, T., Teizer, J. and König, M. (2016) ‘First Person Virtual Reality for Evaluation and Learning of Construction Site Safety’, International Symposium on Automation and Robotics in Construction (ISARC) Proceedings, 2016 Proceedings of the 33rd ISARC, Auburn, USA, pp. 200–208. Available at: https:// doi.org/10.22260/ISARC2016/0025.

Industry 5.0 - European Commission (2024). Available at: https://research- and-innovation.ec.europa.eu/research-area/industrial-research-and-innovation/ industry-50_en (Accessed: 18 November 2024). Johnson, T. L. et al. (2019) ‘How and why we need to capture tacit knowledge in manufacturing: Case studies of visual inspection’, Applied Ergonomics, 74, pp. 1–9. Available at: https://doi.org/10.1016/j.apergo.2018.07.016. Kopp, T., Baumgartner, M. and Kinkel, S. (2021) ‘Success factors for introducing

industrial human-robot interaction in practice: An empirically driven framework’, The International Journal of Advanced Manufacturing Technology, 112(3), pp. 685–704. Available at: https://doi.org/10.1007/s00170-020-06398-0. Pamidimukkala, A. and Kermanshachi, S. (2023) ‘Virtual Reality Training for Geological and Mining Workers to Prevent Powered Haulage Accidents’, pp. 728–736. Available at: https://doi.org/10.1061/9780784484883.063. Pedram, S. et al. (2022) ‘Influence of age and industry experience on learning

experiences and outcomes in virtual reality mines rescue training’, Frontiers in Virtual Reality, 3. Available at: https://doi.org/10.3389/frvir.2022.941225. Rey-Becerra, E. et al. (2023) ‘Improvement of short-term outcomes with VR-based safety training for work at heights’, Applied Ergonomics, 112, p. 104077. Available at: https://doi.org/10.1016/j.apergo.2023.104077. Sashidharan, V. et al. (2024) ‘Collaboration and Co-creation in Industry 5.0: User-

engagement with industrial operators through co-creation workshops’, in Human Aspects of Advanced Manufacturing, Production Management and Process Control. AHFE (2024) International Conference, AHFE Open Acces. Available at: https://doi.org/10.54941/ahfe1005168. Shayesteh, S. et al. (2023) ‘Human-robot teaming in construction: Evaluative safety training through the integration of immersive technologies and wearable physiological sensing’, Safety Science, 159, p. 106019. Available at: https:// doi.org/10.1016/j.ssci.2022.106019. Song, H. et al. (2021) ‘Effectiveness of VR crane training with head-mounted

display: Double mediation of presence and perceived usefulness’, Automation in Construction, 122, p. 103506. Available at: https://doi.org/10.1016/ j.autcon.2020.103506. Vahdatikhaki, F. et al. (2019) ‘Beyond data visualization: A context-realistic construction equipment training simulators’, Automation in Construction, 106, p. 102853. Available at: https://doi.org/10.1016/j.autcon.2019.102853.

Cover of the proceedings volume Human Aspects of Advanced Manufacturing, Production Management and Process Control

Explore the complete digital volume

Human Aspects of Advanced Manufacturing, Production Management and Process Control

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-60-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →