Alessandra Ferraro, Marco Pirozzi, Eduardo Palermo, Daniela Freda, Luciano Di Donato (2024). Virtual Reality and Extended Reality for training of operators. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 159, pp. 1187–1193. https://doi.org/10.54941/ahfe1005686
Training on field for operators of confined or suspected polluted environments is very challenging for guaranteeing safety of attendees and adapting the scenarios to many different environments and risks. In this context Inail realized in collaboration with universities, through a call for bid “SIDE - Development of an exoskeleton for simulated dynamics and haptic interface”, a bi-articular robotic system for the upper limb that can be interfaced with virtual or augmented reality systems. This robotic system can physically reproduce "virtual" stresses of force/contact interaction of the upper limb in a controlled virtual environment, simulating human/environment interactions. In this way, the simulation provides real-life experience: the use of tactile sensation offers the perception of physical effort in a confined space under normal operating conditions and in emergency situations. This simulation helps workers to experiment, facilitating accurate learning of relevant knowledge, skills, and emotions. The solution allows operator to wear visors, the wearable robot (think of a backpack to which a haptic robotic limb is connected) and find yourself in a confined space in virtual reality. Patent application no. 102023000023538 has been filed for this system. The scenario reproduced in virtual reality is a hostile environment - for example, a confined and/or suspected pollution environment - in which the worker had to perform a task in a place very hard to reach and/or with a quantity of oxygen - available limited or with a bad air quality. Thanks to the pre-acquired skills in the motor task carried out in VR, it is possible to obtain a fundamental difference in terms of safety. In this paper we will describe the solution focusing on the characteristics of interaction between the reality and the VR scenario.
The use of AR, VR and MR techniques through visors or HoloLens are increasingly being used to educate, inform, and train workers with particular attention to maintenance techniques for machines, plants and work equipment in general. The advantages are obvious: first and foremost, the training activity does not involve any risks for the operator, since the same are simulated. In addition to this, with AR VR and MR it is possible to carry out the training and instruction activity easily in any type of environment, as it is possible to virtually replicate even large or particularly complex
environments. The personnel to be trained or instructed do not have to travel to a specific location, reducing logistics costs. Confined and/or suspected polluted environments are high risk environments for the workers who work in them as well as for those who support them (OSHA, 2015). The number of fatal accidents is very high; moreover, for each accident that occurs, the number of workers involved is often greater than one (Song Lu, 2022). For this reason, specific education and training is required to make the worker informed and trained on the specific risks and on the equipment and tools to be used for safety and, in addition, capable of handling the possible emergency. It is evident the importance of effective information, education, and training, but also the difficulty of carrying out especially the training phase, when the worker must practise the task.
The use of AR VR and MR for the training and instruction of workers who work in confined and/or pollution-suspected environments represents a significant opportunity since these environments are particularly complex and risky and where, in the event of an accident, the worker must also be able to rescue uncooperative operators (Di Donato et al., 2020). Simulating education and training reduces costs, time, and risks. Its effectiveness is all the greater the more the worker feels he/she is living the experience as in reality. In this sense, implementing virtual reality with a haptic system makes it possible to feel the weight of the objects that interface with the work (fire extinguisher, portable lamp, circular valve, etc.), making the action performed in the virtual environment much more real by reducing the distance with the action performed in the real world (Carignan et al., 2009).
THE SYSTEM SIDE The activity was developed in four steps:
1. Simplified scenarios for verification of functionality of the prototype. 2. Low level software for exoskeleton interface/virtual reality engine. 3. Implementation of a realistic scenario. implementation and adjustment of parameters defining haptic sensitivity as a function of exoskeleton performance. 4. Final development of the VR system: SIDE test and demonstration scenario.
The development of the elements was carried out in parallel and then the final integration allowed to synchronize the movements in the virtual scenario and haptic feedback in a realistic and vector way.
Side Virtual Reality The virtual environment is aimed at simulating human-environment interactions typical of confined environments. There are two scenarios within the application.
The first one consists of a series of rooms and objects with which it is possible to interact. The first scenario can be seen as a demonstration scene of all the actions that can be simulated using the exoskeleton.
Virtual Reality and Extended Reality for Training of Operators 1189
The second scenario is a virtual reconstruction of a real simulator used by INAIL for the training of workers in confined environments.
The virtual reality scenario involves the implementation of user interface software realised in 3D graphics capable of simulating the virtual scenario of a confined environment. Inside, the subject in first-person view will find the activities to be performed, the realisation of which will provoke the sensation of articular effort provided by the exoskeleton to the user.
Each interaction is characterised by an event (e.g. the simple picking up of an object as in Fig. 1) and an associated force corresponding to that which the user would have to exert to perform that interaction. The exoskeleton processes the modulus and direction of that force, which is then exerted on the user’s arm. The application can interface with SIDE (SImulated Dynamics Exoskeleton) and can be used through appropriate commercial headsets optimised for room-scale simulations (an HTC Vive Pro Eye was used for the prototype).
Figure 1: Interaction in VR with a torch.
To test the prototype, a simple scenario has been created in which the user can interact with different objects. Later, some interactions were chosen that could generate haptic feedback emulated by the exoskeleton.
Based on these interactions, the code needed to detect hand collisions - surfaces was written. Unity’s physical engine can calculate with a good degree of realism the components of the pulses resulting from collision events. The realized scenario can visualize in real time the vectors impulse generated by the collisions through the arrows of red color. The components of these pulses constitute the parameters to be sent to the low-level interface of the exoskeleton.
The control algorithm was therefore implemented on a single board computer which can communicate via wi-fi with the configuration laptop. Once configured with the selected scenario, it can control the motors autonomously. Again, via wi-fi, it manages communication with the prototype on which the VR Unity scenario runs, which the operator explores
via visor (HTC Vive pro). When the operator manipulates or meets objects in VR, a force input to be simulated is transmitted to the UDOO board via wi-fi. The board controls the motors in pairs, solving the inverse dynamics algorithm implemented inside it. The high-level controller is the last developed component. It allows the virtual reality simulation software to communicate with the Supervisory Controller, i.e. with the exoskeleton.
Exoscheleton SIDE is a bi-articular robotic system for the upper limb that can be interfaced with virtual or augmented reality systems, capable of physically reproducing “virtual”stresses of force/contact interaction of the upper limb in a controlled virtual environment, simulating human/human interactions in a typical confined and/or suspected pollution environment, but its application can be extended to many other areas. The situations that the device can reproduce are for example, lifting objects, manipulating tools or machinery, opening and closing valves, hitting fixed elements, etc.
It consists of a wearable robotic device, fixed to the subject’s body through a corset and connected to the upper limb in two points where it transfers forces to reproduce the mechanical load applied to the shoulder and elbow joints. The robotic arm is made up of 4 motors, positioned on the 4 joints, and 4 links.
The degrees of freedom provided by the 4 joints allow stresses (moments) to be applied to the shoulder joint on the 3 anatomical planes, leaving the subject the possibility of carrying out as many free rotations with the shoulder as well as lifting the glanhumeral joint without obstacles (limits). The last joint of the kinematic chain allows to apply a flexion-extension moment from the elbow and allows the same rotation.
SIDE is an active exoskeleton realized to increase the immersivity and effectiveness of the virtual experience of the subject, then it was necessary to increase the stability of the control algorithm, allowing as much as possible to perform a force or impedance control rather than a position control.
Research on interaction sensors, with the related supporting electronics, has focused on being able to obtain information regarding the contact force exchanged between the subject and the exoskeleton and to measure the relative movement of the subject with respect to the exoskeleton.
Motor performance tests were carried out to measure maximum torques and powers that can be delivered by the chosen motors, to ensure that they can compensate for the weight of the structure and provide force feedback, even impulsive, when the situation encountered in the VR scenario makes it predict. This would guarantee the effectiveness of the device in providing the greatest possible immersion and realistic haptic feedback to the user. The motion actuator RMD-X8 pro (Figure 2) is a highly integrated power output module.
Power is supplied via two 4500 mAh 22.2V 6-cell LIPO batteries. Control is carried out via two cards that are connected to the computer via Wi-Fi.
VIO (visual-inertial odometry) sensors were used to measure the subject’s movement. In these devices, information obtained from 3D depth cameras is fused with data from traditional inertial sensors, using special algorithms
Virtual Reality and Extended Reality for Training of Operators 1191
called SLAM (simultaneous location and mapping). The advantage, compared to traditional IMUs, lies in the possibility of tracking with a fixed video-based reference, limiting the inaccuracy due to integration drift, typical of IMU systems. Furthermore, the low cost of these devices would guarantee the possibility of installation inside the exoskeleton without excessive increase in the cost of the device.
Figure 2: SIDE robotic arm & virtual reality.
During its operation, the device applies a stress replicating that which the subject would feel if he really carried out the activity he carries out in a virtual environment.
The software architecture includes multiple layers to distribute the functions according to the required computational capabilities. The levels are as follows:
1. Low-Level Layer: directly controls the actuation system (4 motors via CANBUS interface), acquires the status of the sensors locally, performs low- complexity processing.
2. Mid-Level Layer: coordinates the state of the Low-Level Layer, completes instructions from the outside [via WIFI connectivity], locally acquires the state of other sensors (inertial system), performs moderately complex processing.
3. High-Level Layer (Behavioral Controller): Sends commands to the Supervisory Controller, which correspond to the “temporary” behavior that the exoskeleton must assume, depending on the interaction between exoskeleton and virtual environment simulated at every moment by the reality engine virtual.
TEST AND RESULTS The complete solution of the prototype SIDE was tested by several subjects. The evaluation of the test represents an indicator for the effectiveness of SIDE in simulating dynamics in virtual reality, and regarding ergonomics and wearability. A total of 11 questionnaires were collected in different development phases. Nine subjects provided an evaluation of the prototype in its first version. The main aspects considered were:
simulated dynamics’ faithful of reproduction: scores from 1 low faithful – 5 high faithful. The results recorded were 4 for 44,4% of subjects and 5 for 55,6%).
assessment of perceived overall weight: scores from 1 bearable weight – 5 not bearable weight. The results recorded were 3 for 44,4 % of subjects and 4 for 55,6%).
weight balance: scores from 1 very unbalancing – 5 not unbalancing. The results recorded were 2 for 22,2 of subjects, 3 for 44,4% and 4 for 33,3%).
The results highlighted the first model was perceived too heavy and uncomfortable. All this took an improvement of the mechanical design.
Then it was necessary to complete the prototype with a spinal structure and unloading belt to distribute the weight on the pelvis, lightening the weight on the shoulders, and to even better counteract the unbalancing effect given by the haptic arm.
In this new version, two subjects provided their evaluation by answering the same questionnaire. Overall, the reviews highlight a more than satisfactory behavior in terms of simulation of dynamics and haptic interface. Ergonomics and comfort are improved in the last version.
CONCLUSION
Virtual reality (VR) allows workers to practice in a space where they face risks typical of confined space but without real danger. The exoskeleton, reproducing “virtual” stresses of force/contact interaction of the upper limb, improve simulated dynamics’ faithful of reproduction. Tactile perception enhances immersion, increasing the feeling of a real experience for the worker.
The exoscheleton has been improved during the project, as explained above, to improve the ergonomics and comfort aspects.
Next activity steps require to develop tests to characterize the tools and then suggest criteria for such innovative training.
Moreover, it is very important to complete the prototype with the other arm and improve the effectiveness of the training model for example testing the behaviour of attendees during the physical training with or without a previous virtual training.
REFERENCES
Carignan C., Tang J., Roderick S. (2009). Development of an Exoskeleton Haptic Interface for Virtual Task Training. IEEE/RSJ International Conference on Intelligent Robots and Systems October 11–15, 2009 St. Louis, USA.
Virtual Reality and Extended Reality for Training of Operators 1193
Di Donato L., Longo F., Ferraro A., Pirozzi M. (2020). An advanced solutions for operators’ training working in confined and/or pollution suspected space. Procedia Manufacturing 42. Grajewski D., Górski F., Hamrol A., Zawadzki P. (2015). Immersive and Haptic
Educational Simulations of Assembly Workplace Conditions. Procedia Comput. Sci., vol. 75, no. Vare, pp. 359–368, 2015. Protecting Construction Workers in Confined Spaces: Small Entity Compliance Guide (2015) OSHA 3825–09 2015. Song Lu, et al. (2022). Development and validation of a confined space rescue
training prototype based on an immersive virtual reality serious game. Advanced Engineering Informatics 51 101520. Zhuo X. et al. (2022). Design and Development of Digital Teaching Resources for HSE Training on Safety Management of Confined Space Operations Based on VR Technology. https://dx.doi.org/10.4108/eai.24-11-2023.2343379
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional