Katharina Kuznecova, Susanne Werking, Gerrit Meixner (2024). Mobile Solution for Ergonomic Training in Industry: A HoloLens 2 Mixed Reality Approach. In: Human Factors in Virtual Environments and Game Design. AHFE Open Access, vol. 137, pp. 41–49. https://doi.org/10.54941/ahfe1004985
In industry, many employees suffer from limb and spine ailments. These physical disorders lead to long-term pain syndromes and musculoskeletal issues. As a proactive measure, Augmented Reality can aid in conducting interactive ergonomic training sessions for employees and provide real-time support for monitoring individual workers on the shop floor. Existing setups for ergonomic evaluations limit the user’s mobility because these are typically stationary and require external motion-capturing systems. Here, we address Musculoskeletal Disorders using Head-Mounted Displays as a standalone, hands-free, and mobile tool for real-time ergonomic evaluation. We selected the Microsoft HoloLens 2 as a Head-Mounted Display and Rapid Upper Limb Assessment as the ergonomic analysis method. We implemented parts of this system using the Unity development environment. Our Proof of Concept shows that a hands-free Mixed Reality application with the HoloLens 2 can support ergonomic evaluations without requiring stationary setups or additional camera or sensor systems. In the current setup, the live feed from the camera of the HoloLens 2 captures the observed person. A local computer analyzes the posture via a wireless network connection, and the results are sent back to the Unity application, where they are visualized with a 3D model. Our work offers a comprehensive perspective on the components used and provides an implementation for the suggested system. It also serves as a foundation for future work, including testing the system’s efficiency in practical use and expanding the user interface, specifically for training in a working environment.
Despite numerous efforts, Musculoskeletal Disorders (MSDs) remain the most common health issue among workers in both the United States and the European Union (Niu, 2010; Punnett & Wegman, 2004; for Safety et al., 2019; Russo et al., 2020). These disorders, which have socio- eco- nomic consequences (Bevan, 2015), include symptoms such as discomfort, numbness, tingling, stiffness, and tension in affected individuals. Influencing factors include age, gender, work duration, posture, and ergonomic aware- ness. MSDs can result from repetitive mechanical overload without sufficient recovery or a single excessive load leading to injury. These are known as Work-Related Musculoskeletal Disorders (WRMSDs) when associated with risk factors at work (Niu, 2010; Punnett & Wegman, 2004).
Measures against WRMSDs include consistent worker monitoring, com- prehensive ergonomic training, eliminating hazards, and improving the work environment (Etuknwa & Humpheries, 2018). The impact of those proac- tive measures can be influenced by various factors. Studies indicate that ergonomic training, when conducted in generic settings with generic postures, lectures, pamphlets, and videos, rather than at job sites, lacks the necessary work context and interaction for more effective training (Burke et al., 2006; Casey et al., 2021). Therefore, visually engaging workers and active practice of participants during these learning sessions can lead to improved long-term outcomes (Etuknwa & Humpheries, 2018).
Previous work indicates that Augmented Reality can support trainees in adopting improved ergonomic practices by enhancing their understanding of ergonomics and providing opportunities for hands-on learning (Vicente et al., 2023). Existing solutions that utilize Head-mounted displays (HMDs) require supplementary sensor systems to capture the human posture, and therefore necessitate the establishment of controlled settings with fixed components (Evangelista et al., 2023; Bruno et al., 2020; Omidi et al., 2023). To enable employees to assess postures while freely navigating their working environ- ment, such as the shop floor, and observing working processes, we aim to address the gap in existing systems by creating a solution without stationary components. This will afford users the opportunity to navigate and interact within their operational contexts.
In this study, we explore a combination of technologies and methodologies to develop a mobile solution for ergonomic pose assessment using Augmented Reality with HMDs.
Our contribution lies in advancing ergonomic practices through AR- based pose estimation, promoting healthier and more productive work environments.
RELATED WORK
Existing ergonomic tools rely on the following common elements, outlined below: a method for evaluating human body postures for ergonomics, a tech- nology for capturing these postures, and a system for feedback or display of the results.
Tools that support human posture analysis for ergonomic purposes in the workplace often utilize observational methods, such as the Rapid Upper Limb Assessment (RULA) (Evangelista et al., 2023; Manghisi et al., 2020; Vicente et al., 2023; Padilla et al., 2019; Lin et al., 2022; Paudel et al., 2022). A monitoring and training system based on RULA is the Ergosen- tinel tool (Manghisi et al., 2020). This assessment tool evaluates ergonomic postural risks at a workstation, providing a semi-automatic evaluation for each frame (Manghisi et al., 2020). It includes an advanced setting inter- face, allowing for the manual selection of factors such as wrist deviation, and features an interactive timeline plot with RULA grand-scores for offline analysis (Manghisi et al., 2020). To track the operator’s body, a low-cost D- RGB camera (Kinect v2) was used, and acoustic warnings were added to the
Mobile Solution for Ergonomic Training in Industry: A HoloLens 2 Mixed Reality Approach 43
visual feedback to alert the operator against repeatedly adopting unfavorable postures (Manghisi et al., 2020).
To facilitate automatic ergonomic assessments, the human posture must first be captured. This can be achieved through sensors or markers directly attached to the human body, like inertial measurement units (IMUs) (Vig- nais et al., 2013; Padilla et al., 2019; Kim et al., 2018), or through a motion capture system like Optitrack, where the user wears a motion caption suit with a number of spherical reflective markers (Bruno et al., 2020). Alter- natively, posture can be captured solely by visual means, using RGB and depth-sensing cameras (Manghisi et al., 2020; Vicente et al., 2023; Lee et al., 2018; Evangelista et al., 2023; Omidi et al., 2023). Systems like those mentioned often require specific setup procedures or are stationary. Examples include marker-based motion capture (Bruno et al., 2020) and on-body sensor systems (Bruno et al., 2020; Vignais et al., 2013; Kim et al., 2018).
To widely implement ergonomics on the shop floor, solutions employing low-cost sensors are being investigated (Manghisi et al., 2020). These meth- ods involve contactless ways of capturing human postures, utilizing depth sensors like the Kinect v2 (Omidi et al., 2023; Manghisi et al., 2020; Evange- lista et al., 2023; Lee et al., 2018), as well as images or videos (Vicente et al., 2023). Mixed Reality is already utilized to support evaluating ergonomics during the working process (Omidi et al., 2023; Evangelista et al., 2023).
To enhance ergonomic training in work environments, systems incorpo- rating Augmented Reality (AR) are being investigated (Vicente et al., 2023; Evangelista et al., 2023). In a study testing real-time visual feedback in ergonomic training, a prototype was developed. This prototype overlaid semi-transparent holographic spheres on the body joints, color-coded in traffic light colors according to the risk exposure (Vicente et al., 2023). A similar tool utilizes the Ergosentinel system and a Head-mounted Display, the Hololens 2 (Evangelista et al., 2023). Instead of displaying the feedback on a handheld device, the results are presented on the Hololens 2 display (Evange- lista et al., 2023). This approach focuses on enhancing present environments with visual feedback about ergonomic aspects, while using the contactless motion-capturing device Kinect v2 to facilitate a relatively mobile, hands-free solution (Evangelista et al., 2023).
However, the implementation of training in actual work settings necessi- tates the removal of static elements like the Kinect to facilitate unimpeded mobility across the shop floor.
To introduce the novelty of enabling users to move unrestricted and hands- free while observing the ergonomic postures of their co-workers in real-time, we omitted the use of external and static sensor systems like the Kinect v2 (Evangelista et al., 2023). Our objective is to exclusively utilize the internal sensors provided by the HMD device. To achieve this goal, we investigate whether using only the sensors integrated in an HMD device is sufficient to support this kind of training.
Consequently, we aim to develop a system that facilitates real-time human posture capture, affording users the flexibility to conduct ergonomic analyses within the company environment while remaining hands-free and mobile.
METHODOLOGY
The approach of this work is to identify the essential elements required for implementing this ergonomic application. To determine the specific require- ments for the necessary components, research was conducted on existing components and their characteristics in the respective fields.
System Requirements Related works have indicated that three aspects are required for the intended AR solution:
1. Ergonomic Analysis Method: This involves the need for a metric or sys- tem to evaluate the ergonomic poses of individuals. It should be easily automateable, ontact-free, utilizing sensors that can be either added to or are included in the HMD device, such as RGB or depth cameras. 2. Human Pose Estimation: It involves acquiring the human pose digitally and processing it to obtain usable information in accordance with the cho- sen ergonomic metric. The chosen solution should be resource- efficient and capable of real-time evaluation with three-dimensional information for ergonomic analysis 3. Head-Mounted Device: This entails the use of mobile hardware for over- laying three-dimensional feedback on the observed person’s ergonomics in an augmented reality environment. Additionally, for our use case, the HMD must have sensors to facilitate the necessary data acquisition, offer hands-free control options, and provide an accessible development environment to support 3D visualizations.
These requirements must be fulfilled to achieve a 3D mobile, hands-free tool for real-time detection of non-ergonomic body poses using AR.
Selection of Components For a mobile, hands-free system in the workplace, common ergonomic analy- sis methods, deep-learning models for human pose estimation, and potential AR devices are explored, chosen, and described in this section.
In comparison to observational, and therefore contact-free, posture-based methods like Rapid Entire Body Assessment (REBA) and Ovako Work- ing Posture Analysis System (OWAS), RULA is the most frequently applied method (Kee, 2022). Additionally, RULA is for static postural analysis and appears to be a valid evaluation method based on previous works (Vicente et al., 2023; Evangelista et al., 2023; Manghisi et al., 2020; Padilla et al., 2019) which can be used for automatic evaluation with only the angles of body parts.
To evaluate human postures, OpenPose (Cao et al., 2019), BlazePose (Bazarevsky et al., 2020), YOLO-Pose (Maji et al., 2022), and AlphaPose (Fang et al., 2022) are all skeleton-based models used for real-time appli- cations with images. BlazePose and AlphaPose additionally include pose tracking, enabling them to predict keypoints over time, even if the person is partly occluded (Bazarevsky et al., 2020; Fang et al., 2022).
Mobile Solution for Ergonomic Training in Industry: A HoloLens 2 Mixed Reality Approach 45
To facilitate development and execution of these algorithms on mobile devices, the models are developed and packaged into libraries, such as BlazePose GHUM Holistic pipeline for 3D human body landmarks and pose estimation (Chung et al., 2022; Grishchenko et al., 2022).
We have decided to use the MediaPipe API, based on BlazePose by Google, for our ergonomic analysis. It is specifically optimized for real-time detection of human body postures (Grishchenko et al., 2022). Its ability to recognize the upper and lower body, as well as extremities like hands and feet in 3D with 33 keypoints, makes it well-suited for ergonomic analyses in real-time.
Augmented Reality adds visual information to the user’s field of view by displaying content on a video stream of the real world or an optical see- through display (Doughty et al., 2022). Optical see-through wearable devices include the Meta 2, Magic Leap 2, Hololens 2, and smart glasses like Google Glass 2.
To maintain situational awareness while using the device, the HMD should be optical see-through and operatable without occupying hands.
Both Magic Leap 2 and Hololens 2 fulfill the criteria of being mobile stan- dalone devices equipped with depth and RGB cameras, the latter using hand gestures instead of a controller (Magic Leap, Inc., 2023; Scooley, 2023). Therefore, we chose to proceed with Microsoft Hololens 2, which stands out for its ergonomics in comparison to other Mixed Reality HMDs (Comes & Neamt, 2023) and its availability at the time of this work.
Envisioned System and Implementation The Hololens 2 serves as the platform for our digital system, with Mediapipe as the human pose estimation tool, and RULA as the ergonomics evaluation tool, all implemented using Unity as the development platform.
The first implementation approach involved using an RGB camera feed along with a Unity Plugin for MediaPipe (GitHub, 2023). While the imple- mentation on a Windows platform was successful, the Plugin lacked certain DLL files necessary for deployment on the Universal Windows Platform (UWP), which is a prerequisite for running it on a Hololens 2.
Therefore, we decided to stream the RGB camera feed of the Hololens 2 and process the analysis on an external device, before returning the result via Transmission Control Protocol (TCP) back to the HMD device.
Figure 1: An overview of application for Hololens 2 with existing visualisation (blue) and missing implementation (red).
RESULTS
The resulting system architecture for the Hololens 2 is designed to function without additional external sensors for pose estimation. It utilizes a network connection for a hands-free, mobile approach in real-time, aiming to display results to the user on the HMD screen (see Fig. 1).
The Hololens 2’s color camera live feed is accessed using the 5 MBit video API version, configured with a designated IP address and port.
A Python script, running on a computer facilitates access to this live feed, employing the OpenCV computer vision library for the analysis process.
For human pose estimation, MediaPipe 0.10.2 is used with the video input. Landmark detection is performed using an asynchronous streaming method, where joint evaluation is executed through the PoseLandmarkerResult class.
The extracted keypoints are then used for ergonomic analysis based on RULA, applying techniques from prior research (Vicente et al., 2023) to each frame. This implementation provides six posture values relating to arms, shoulders, neck, and trunk, along with their corresponding action levels, every time the landmark detection method is invoked.
These six values were successfully transmitted to a Unity application and displayed on a ’Human Amateur Skeleton’ within a Windows application. In this three-dimensional human model, the joint sections are color-coded to indicate the following action levels, highlighting potential areas of concern:
Green: Acceptable if the posture is not maintained or repeated for extended periods.
Yellow: Further investigation is necessary, and adjustments might be required.
Orange: Further investigation is necessary and adjustments are required soon.
Red: Immediate investigation and adjustments are required.
This modification demonstrates the successful implementation of the val- ues in the Unity application on a Windows platform. However, it was initially planned to extend this functionality to the Unity UWP application on the Hololens 2 device for visualization via the TCP. Due to technical issues and time constraints, this aspect was not completed.
DISCUSSION
Since the implementation is not yet complete due to technical issues in utiliz- ing TCP for transmitting the RULA evaluation, no study has been conducted so far. Currently, the PoC allows for an ergonomic evaluation in Python with Mediapipe, using a video stream.
A TCP connection from the Hololens 2 to a Python script is feasible. However, the current issue is suspected to be related to the asynchronous Mediapipe streaming solution. Additional potential issues might include the HMD’s performance in sending a video feed, establishing and maintaining a TCP connection, and simultaneously running the application with visual feedback. One of the primary challenges involves conducting an in-depth analysis and harmonizing the distinct threading paradigms of Unity and UWP.
Mobile Solution for Ergonomic Training in Industry: A HoloLens 2 Mixed Reality Approach 47
Our aim with this system was to demonstrate that sensors in HMDs, like the Hololens 2, can be used for real-time ergonomic support, paving the way for a mobile, hands-free application.
The results indicate future possibilities, suggesting that a lightweight AR HMD with a network connection could suffice for the envisioned sce- nario. Furthermore, with future advancements in Neural Networks, even a standalone solution might be feasible by leveraging Visual Studio with Windows ML.
CONCLUSION AND OUTLOOK Existing real-time Mixed Reality systems typically require additional sen- sors, such as the Kinect, for ergonomic pose estimation. We explored various approaches, utilizing pose estimation with the MR HMD device Hololens 2. The key finding is that an external sensor system for human pose capturing is not necessary for ergonomic evaluation with Hololens 2. The chosen system operates by streaming content to and from the HMD device via the inter- net. This demonstrates the feasibility of a hands-free mobile AR application, based on the RULA metric, can effectively support ergonomic training in the working environment.
REFERENCES
Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F. & Grundmann, M. (2020), ‘Blazepose: On-device real-time body pose tracking’. Bevan, S. (2015), ‘Economic impact of musculoskeletal disorders (MSDs) on work in Europe’, Best Practice Research Clinical Rheumatology 29(3), 356–373. Occupation and Musculoskeletal Disorders. URL: https://www.sciencedirect.com/science/article/pii/S1521694215000947 Bruno, F., Barbieri, L. & Muzzupappa, M. (2020), ‘A mixed reality system for the ergonomic assessment of industrial workstations’, International Jour- nal on Interactive Design and Manufacturing (IJIDeM) 14, 805–812. URL: https://api.semanticscholar.org/CorpusID:225517293 Burke, M. J., Sarpy, S. A., Smith-Crowe, K., Chan-Serafin, S., Salvador, R. O. &
Islam, G. (2006), ‘Relative effectiveness of worker safety and health training meth- ods’, American Journal of Public Health 96(2), 315–324. PMID: 16380566. URL: https://doi.org/10.2105/AJPH.2004.059840 Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E. & Sheikh, Y. (2019), ‘Openpose: Realtime multi-person 2d pose estimation using part affinity fields’. Casey, T., Turner, N., Hu, X. & Bancroft, K. (2021), ‘Making safety training stickier: A richer model of safety training engage- ment and transfer’, Journal of Safety Research 78, 303–313. URL: https://www.sciencedirect.com/science/article/pii/S0022437521000815 Chung, J.-L., Ong, L.-Y. & Leow, M.-C. (2022), ‘Comparative analysis of skeleton-based human pose estimation’, Future Internet 14(12). URL: https://www.mdpi.com/1999-5903/14/12/380 Comes, R. & Neamt, U, C. (2023), ‘Head-mounted display wearable devices ergonomics and usability’, Acta Technica Napocensis - Series: Applied Mathematics, Mechanics, and Engineering 65(3S). URL: https://atna-mam.utcluj.ro/index.php/Acta/article/view/1944
Doughty, M., Ghugre, N. R. & Wright, G. A. (2022), ‘Augmenting performance: A systematic review of optical see-through head-mounted displays in surgery’, Journal of Imaging 8(7). URL: https://www.mdpi.com/2313-433X/8/7/203 Etuknwa, A. & Humpheries, S. (2018), ‘A systematic review on the effective-
ness of ergonomic training intervention in reducing the risk of musculoskeletal disorder’, 03. Evangelista, A., Manghisi, V. M., Romano, S., De Giglio, V., Cipriani, L. & Uva, A. E. (2023), ‘Advanced visualization of ergonomic assessment data through industrial augmented reality’, Procedia Computer Science 217, 1470–1478. 4th International Conference on Industry 4.0 and Smart Manufacturing. URL: https://www.sciencedirect.com/science/article/pii/S1877050922024310 Fang, H.-S., Li, J., Tang, H., Xu, C., Zhu, H., Xiu, Y., Li, Y.-L. & Lu, C. (2022),
‘Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time’. for Safety, E. A., at Work, H., IKEI & Panteia (2019), ‘Work-related musculoskeletal disorders: Prevalence, costs and demographics in the EU’. GitHub (05.08.2023), ‘homuler/mediapipeunityplugin: Unity plugin to run mediapipe’. URL: https://github.com/homuler/MediaPipeUnityPlugin Grishchenko, I., Bazarevsky, V., Zanfir, A., Bazavan, E. G., Zanfir, M., Yee, R.,
Raveendran, K., Zhdanovich, M., Grundmann, M. & Sminchisescu, C. (2022), ‘Blazepose ghum holistic: Real-time 3d human landmarks and pose estimation’. Kee, D. (2022), ‘Systematic comparison of OWAS, RULA, and REBA based on a literature review’, International Journal of Environmental Research and Public Health 19(1). URL: https://www.mdpi.com/1660-4601/19/1/595 Kim, W., Lorenzini, M., Kapıcıo˘glu, K. & Ajoudani, A. (2018), ‘Ergotac: A tac-
tile feedback interface for improving human ergonomics in workplaces’, IEEE Robotics and Automation Letters 3(4), 4179–4186. Lee, L. B., Shin, J. G., Bae, H. & Saakes, D. (2018), Interactive and situated guidelines to help users design a personal desk that fits their bodies, pp. 637–650. Lin, P.-C., Chen, Y.-J., Chen, W.-S. & Lee, Y.-J. (2022), ‘Automatic real-time occu-
pational posture evaluation and select corresponding ergonomic assessments’, Scientific Reports 12. Magic Leap, Inc. (28.07.2023), ‘Magic leap 2: Device’. URL: https://www.magicleap.com/magic-leap-2 Maji, D., Nagori, S., Mathew, M. & Poddar, D. (2022), ‘Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss’. Manghisi, V. M., Uva, A. E., Fiorentino, M., Gattullo, M., Boccaccio, A. & Evan-
gelista, A. (2020), ‘Automatic ergonomic postural risk monitoring on the factory shopfloor –the ergosentinel tool’, Procedia Manufacturing 42, 97–103. Interna- tional Conference on Industry 4.0 and Smart Manufacturing (ISM 2019). URL: https://www.sciencedirect.com/science/article/pii/S2351978920306569 Niu, S. (2010), ‘Ergonomics and occupational safety and health: An ILO perspective’, Applied Ergonomics 41(6), 744–753. Special Section: Selection of papers from IEA 2009. URL: https://www.sciencedirect.com/science/article/pii/S0003687010000499 Omidi, M., Van de Perre, G., Hota, R., Cao, H.-L., Saldien, J., Vanderborght, B.
& El Makrini, I. (2023), ‘Improving postural ergonomics during human–robot collaboration using particle swarm optimization: A study in virtual environment’, Applied Sciences 13, 5385. Padilla, B., Glushkova, A., Menychtas, D. & Manitsaris, S. (2019), Designing a webbased automatic ergonomic assessment using motion data, pp. 528–534.
Mobile Solution for Ergonomic Training in Industry: A HoloLens 2 Mixed Reality Approach 49
Paudel, P., Kwon, Y.-J., Kim, D.-H. & Choi, K.-H. (2022), ‘Industrial ergonomics risk analysis based on 3d-human pose estimation’, Electronics 11, 3403. Punnett, L. & Wegman, D. H. (2004), ‘Work-related musculoskeletal disorders: the epidemiologic evidence and the debate’, Journal of Elec- tromyography and Kinesiology 14(1), 13–23. State of the art research perspectives on muscoskeletal disorder causation and control. URL: https://www.sciencedirect.com/science/article/pii/S1050641103001251 Russo, F., Di Tecco, C., Fontana, L., Adamo, G., Papale, A., Denaro, V. & Iavicoli, S.
(2020), ‘Prevalence of work related musculoskeletal disorders in italian work- ers: is there an underestimation of the related occupational risk factors?’, BMC Musculoskeletal Disorders 21. Scooley (29.03.2023), ‘Hololens 2 hardware’. URL: https://learn.microsoft.com/enus/hololens/hololens2-hardware Vicente, D., Schwarz, M. & Meixner, G. (2023), Improving ergonomic training
using augmented reality feedback, in V. G. Duffy, ed., ‘Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management’, Springer Nature Switzerland, Cham, pp. 256–275. Vignais, N., Miezal, M., Bleser, G., Mura, K., Gorecky, D. & Marin, F. (2013), ‘Innovative system for real-time ergonomic feedback in indus- trial manufacturing’, Applied Ergonomics 44(4), 566–574. URL: https://www.sciencedirect.com/science/article/pii/S0003687012001858
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Virtual Environments and Game Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-13-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional