Eduardo Pontes, Azin Moradbeikie, Rolando Azevedo, Cristiano Jesus, S Lopes (2023). 'Ergonomic posture assessment and tracking for industrial cyber-physical-human systems: A case study in the heavy metalworking industry. In: Human Aspects of Advanced Manufacturing. AHFE Open Access, vol. 80, pp. 149–158. https://doi.org/10.54941/ahfe1003517
In the context of Industry 5.0, collaborative automation and decision support through ubiquitous digital processing provide a privileged ap-proach to examining the human factor in the industry. At the same time, new technologies that have been under development since the in-ception of Industry 4.0, such as IoT devices, artificial vision systems, image processing algorithms, artificial intelligence, and others, have brought important opportunities to the industry, which has historically based its process transformation and management on a preventive ap-proach that helps improving decision-making. Musculoskeletal disor-ders in industrial working scenarios are often associated with the accu-mulation of stress over time, which can impact the muscles, tendons, ligaments, joints, and other parts of the body. To prevent injuries and complications in the short, medium, or long term, Cyber-Physical-Human Systems (CPHS) can be adopted to correct risky actions of op-erators in real-time. This work presents preliminary results regarding the study, understanding, and identification of operator postures in the heavy metalworking industry. The study was based on the comparison of the commonly used methods for ergonomic posture and movement assessment. The adopted approach takes advantage of computer vi-sion for operator pose identification and tracking to effectively detect the most frequently repeating body postures. The most repeated pos-tures are then categorized according to their ergonomic compatibility. To evaluate the proposed approach, a dataset has been acquired based on the observation of real operator actions. Based on the results, the implemented system enables us to actively evaluate the appropriate-ness of workers' postures in real-time.
In the context of Industry 5.0, taking as reference the principles enunciated by the European Commission in 2021 (European Commission, 2021), ergono- mics and safety in industry play a central role in the development of resilient and human-centered production systems (Fraga-Lamas et al., 2022), aiming at economic growth and sustainability. In Industry 5.0, the application of technologies such as edge computing (Fernández-Caramés and Fraga-Lamas, 2018; Tanganelli et al., 2020), IIoT (Zhang et al., 2018) or Digital Twin (Mylonas et al., 2021; Wang et al., 2022) should be supported by computer vision systems that allow integrating these concepts with reality. The combi- nation of these technologies allows us to go beyond industrial exploitation and boost the principles of Industry 5.0: human centricity, sustainability, and resilience (Fraga-Lamas et al., 2022).
After the identification of posture and ergonomic improvement needs in a heavy metalworking factory, the proposal emerged to create a cognitive support (Grobelny and Michalski, 2020) that meets the need to increase ope- rational efficiency and productive resources of the factory in a harmonious way, thus taking better advantage of human abilities and workers’ well-being. Laboratory simulations were performed to replicate the postures and move- ments identified in the field. To do so, a vision system was adopted, which identifies and evaluates workers’ postures during operation. This system is composed of a camera, an image processing system that enables the recon- struction of the exoskeleton, the determination of the angles in the various joints of the operator and the subsequent qualitative evaluation of the posture adopted (Claeys et al., 2022).
The aim of this system is to intervene in real-time to correct and adapt the postures and movements in the activity, safeguarding the well-being of the operators and avoiding musculoskeletal injuries in the short, medium and long term. The system was built to reliably evaluate workers’ postures and intervene in real-time for corrections and adjustments (Jayaram et al., 2006). The postures and movements identified in the factory plant were simulated, obtaining promising results.
Within this developed work paradigm, the question arose whether, by using a vision system, we could apply the most recognized ergonomic methods currently available (Joshi and Deshpande, 2019). A study and investigation into the most used methods currently in industry for ergo- nomic posture and movement evaluations was done. Rapid Entire Body Assessment (REBA), Rapid Upper Limbs Assessment (RULA) and Ovako Working posture Assessment System (OWAS) were selected for implemen- tation (Joshi and Deshpande, 2019). Using an image processing system, it was possible to apply these methods in a combined evaluation between all the angles collected to try to infer a more objective and feasible result. Assuming that some principles of Industry 5.0, such as the centrality of the human being and the use of technologies for the harmonization of the industrial environment and the human in it, the implementation of this system and these practices meet the enunciated by Industry 5.0 (Battini et al., 2022).
Ergonomic Posture Assessment and Tracking for Industrial Cyber-Physical-Human Systems 151
After all of this work and in the context of Industry 5.0 two research questions arose: RQ1: Does implementing a vision system for posture capture and evalua- tion bring a contribution in line with the principles of Industry 5.0?
RQ2: From the results obtained in the data collection, is it possible and fea- sible to identify and act in real-time to correct the worker’s posture regarding these ergonomic assessments?
THEORETICAL BACKGROUND Over the past few years, it has become crucial, particularly for companies and managers, to mitigate occupational exposure to ergonomic risk factors (Rinaldi et al., 2022). Incorporating human factors within operational deci- sion processes has become an increasing concern in the last decade (Battini et al., 2022), following the upcoming paradigms of Industry 5.0, machine cognition, on top of vision and sensory technologies, must be improved to make the best decisions in an ever-changing work environment (Mourtzis et al., 2022).
Ergonomics in the metalworking industry has previously been addressed in studies where a multi-level ergonomic intervention program for muscu- loskeletal disorders was implemented (Choobineh et al., 2021) in which the incorporation of training workshops, participatory ergonomics and worksta- tion redesign led to workers reporting less musculoskeletal pain and fatigue during the intervention program. Consequently, a broad range of risk factors, including physical (e.g., manual material handling, repetitive movements, excessive energy expenditure, and awkward postures) and organizational (e.g., inadequate recovery time) factors exist in the metalworking industry, particularly for frontline workers (Choobineh et al., 2011).
Three main methods are used to simulate human motion on a computer. The first method is by combining biological principles to build human motion models as well as dynamic models through mathematical algorithms; the second method is by capturing real human motion data using motion capture devices, and the third method is by using real data with mathematical algori- thms. These three methods have been widely applied in assembly simulation to build digital human models (Yin and Li, 2023). Digital human mode- ling is a two-dimensional or three-dimensional computer-generated model of human structure and refers to the digitization of the human structure and the expression of various features of the human body (geometry, physics, physi- ology, action, behavior, perception, emotion, psychology, sociality, etc.). It is the digital replication of human features (Choobineh et al., 2011).
Video footage became an indispensable resource for ergonomic practice. With the upbringing of video technology, it has greatly facilitated this appro- ach and has provided engineers, and often workers themselves, with a chance to monitor and revise operations to identify hazards and opportunities for ergonomic improvements (Greene et al., 2017). Due to the progress in com- munication and information technology, movement recording systems are becoming more widely used to capture the position, posture, and moving track of assembly workers. The incorporation of actual human behavior into
the model building process can achieve actual human body motion posture and lead to more accurate human factor analysis. Data-driven digital human models can save the labor time of digital human posture adjustment and gre- atly improve the efficiency of simulating manual assembly operations (Yin and Li, 2023).
The benefit of employing non-contact human motion data collection devi- ces is that the operator does not need to wear any equipment whatsoever. It merely depends on image information on infrared sensors, RGB-D came- ras, or ordinary cameras to capture and identify human actions. Kinect has been used to capture human movement in real time and has been applied to ergonomic software or independently developed programs for assembly simulation with digital human models (Alipoor et al., 2021; Puthenveetil et al., 2015) more widely.
Currently, existing research methods and algorithms encounter a limita- tion in relation to partial limb occlusion. In the assembly environment itself, especially in the indoor space of large equipment, there are still cases where the human limbs are almost completely occluded by the body position or by some objects present in their workspace. There is still no improved solution to the problem of motion acquisition of occluded parts of the human body in these kinds of situations (Yin and Li, 2023). The actual factory site envi- ronment is complex, therefore it is challenging to assemble multiple motion capture cameras.
METHODS AND MATERIALS We simulated a workstation in a heavy metalworking factory where, after performing a laser cutting process in a metallic sheet, an operator would remove the pieces from the main sheet. This task consisted of picking up the pieces and placing them on a pallet for later transportation, sometimes requiring the use of a hammer to separate parts that had not been completely cut in the laser process. We observed that, on average, the weight of a piece was around 5 kgs and to be removed, each piece needed to be hammered 4 times.
Bearing this in mind, with a hammer and 4 kettlebells, of 4 kgs and 6 kgs, to represent the pieces, we reproduced this workstation, recreating a work cycle. This cycle consisted of hammering 4 times, carrying a kettlebell to the pallet, and repeating until all the weights were on the pallet. When all pieces were transported, both hands were placed on the workstation to symbolize the end of the cycle. Then, the pieces were placed back on the workstation and the cycle was restarted. This process was repeated throughout the spawn of 1 hour.
To create the dataset, we assembled a vision system consisting of 3 cameras off the shelf, forming an angle of 120
between them, connected to a server to process the collected recordings. The cameras used were: on the right, Cleyver ODWCAM90; in front, Logitech C120 Webcam; on the left, EyeToy USB Camera for Playstation 2. The computer where the video streams were stored and processed was a ThinkCentre M70q Tiny Desktop.
Ergonomic Posture Assessment and Tracking for Industrial Cyber-Physical-Human Systems 153
Figure 1: Experimental apparatus used for the dataset acquisition.Figure 2: Example of acquired images that represent the simulation of a sequence of manual operations.
For data collecting, one person acting as an operator starts performing the work process (hammering 4 times, carrying a kettlebell to the pallet, and repeating until all the weights are on the pallet) for one hour. To compute the worker’ postures, the angles of each part of the body is computed and reported in real time.
Then, the collected data was subjected to ergonomic classification using a program and according to REBA, RULA and OWAS methods. Mediapipe was used for skeleton detection, which is an open-source project that provide a suite of libraries and tools that apply artificial intelligence (AI) and machine learning (ML) techniques for detecting different parts of body. In the next
step, the angles of each part of the body are computed by using identified key body locations in real time. Finally, based on the computed angles, we compute and report body scores and posture assessments for each method.
The dataset created for this experiment is made available as open access via our GitHub repository1.
ERGONOMICS EVALUATION Using the collected dataset, the posture of the worker has been identified and tracked to detect his scores based on different ergonomic methods to evaluate and report the most repeated and harmful posture. For this purpose, we provide a classification of worker posture based on REBA, RULA and OWAS methods. In the following, the achieved results for each are presented in the next subsections.
In the REBA, the computed posture is classified into ten different sco- res (1 as negligible risk, 2–3 as low risk, 4–7 as medium risk, and 8–10 as high risk). The percentage of the worker posture score during the evaluation period is shown in Fig. 3.
Figure 3: Score according to the REBA method and continuous staying for each score.
In the RULA method, the computed posture is classified into seven diffe- rent scores (1−2 as negligible risk, 3–4 as low risk, 5–6 as medium risk, and 6+ as high risk). The percent of worker posture score is shown in Fig. 4. In the OWAS method, the computed posture is classified into four different scores (1 no corrective action needed, 2 corrective actions needed soon, 3 corrective actions must be performed as soon as possible, and 4 corrective actions are needed immediately). The percent of worker posture score is shown in Fig. 5. Based on REBA, 17 percent of the time, the worker is at medium risk (score is 4−7). One important point about the effect of each posture on worker health is the time duration that workers stay in each posture. To evaluate this point, the percent of continuous staying in each score is computed and shown in Fig. 3. When the worker is in score 7, he stays more than 15 seconds for almost 40 percent of the cases. Based on RULA, 54 percent of the time, the worker is at medium risk (score is 5−6). The percent of continuous staying in 1https://github.com/citin-pt/ergosafe_datasets
Ergonomic Posture Assessment and Tracking for Industrial Cyber-Physical-Human Systems 155
Figure 4: Score according to the RULA method and continuous staying for each score.Figure 5: Score according to the OWAS method and continuous staying for each score.Table 1. Comparison between the results obtained with REBA, RULA and OWAS methods by risk level.
each score is computed and shown in Fig. 4. When the worker is in score 6, he stays more than 15 seconds for around 2 percent of the cases and. Based on OWAS, 32 percent of the time, the worker needs corrective action soon (score 2) and 8 percent of the time he needs corrective action as soon as possi- ble. The percent of continuous staying in each score is computed and shown in Fig. 5. When the worker is in score 3, he stays more than 15 seconds for about 15 percent of the cases.
By using the achieved results, it is possible to track and identify the risky postures of the worker and which method is best suited for different tasks.
Analyzing all the methods and comparing them in risk levels, we can verify that, for the simulated task, RULA would present us with a more precise ergonomic evaluation.
STATEMENT OF LIMITATIONS In the implemented system, the vision system cannot identify objects and their associated weight. So, the average weight of any object or tool being handled should be considered for the simulation and should be reported by the worker in the company. Also, in the real environment, the worker moved along the cutting plate during hammering and loading the cut pieces. Whereas, in the simulation, the workstation was fixed at a single point and the considered motions were lower than the real one.
CONCLUSION AND FUTURE WORK In this paper, the implementation of a CPHS to track the worker postures for identification of the risky positions by using image processing method is presented. Supported by the literature and based on the selected ergono- mics methods, satisfactory results from the performed simulation have been achieved. As a result, we can answer the research questions as follows:
RQ1: Does implementing a vision system for posture capture and evalua- tion bring a contribution in line with the principles of Industry 5.0?
The implementation of this system and of these practices complies with some of the principles of Industry 5.0, such as the centrality of the human being and the use of technologies to harmonize the industrial environment and the human being in it. Either to raise awareness of the best practices in what comes to posture and workload handling and to prevent and protect the worker from excessive exposure to these hazards, this vision system can be a tool to help in this regard.
RQ2: From the results obtained in the data collection, is it possible and fea- sible to identify and act in real-time to correct the worker’s posture regarding these ergonomic assessments?
The obtained results allowed us to identify periods of time where a hazar- dous posture was kept for longer than 10 seconds. In these specific moments, intervention in real time can and must be performed to correct the worker’s posture and safeguard their wellbeing. Future work will focus on strength- ening the accuracy by implementing a 3D vision by using three cameras to provide more complete and accurate visibility of worker and environment situations.
ACKNOWLEDGEMENT
This work has been supported by operation NORTE-06-3559-FSE-000226, funded by Norte Portugal Regional Operational Program (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, through the European Social Fund (ESF).
REFERENCES
Alipoor, P., Daneshmandi, H., Fararuei, M., Zamanian, Z., 2021. Ergonomic Design of Manual Assembly Workstation Using Digital Human Modeling. Ann. Glob. Health 87, 55. https://doi.org/10.5334/aogh.3256
Ergonomic Posture Assessment and Tracking for Industrial Cyber-Physical-Human Systems 157
Battini, D., Berti, N., Finco, S., Zennaro, I., Das, A., 2022. Towards industry 5.0: A multi-objective job rotation model for an inclusive workforce. Int. J. Prod. Econ. 250, 108619. https://doi.org/10.1016/j.ijpe.2022.108619 Choobineh, A., Motamedzade, M., Kazemi, M., Moghimbeigi, A., Heidari Pahla-
vian, A., 2011. The impact of ergonomics intervention on psychosocial factors and musculoskeletal symptoms among office workers. Int. J. Ind. Ergon. 41, 671–676. https://doi.org/10.1016/j.ergon.2011.08.007 Choobineh, A., Shakerian, M., Faraji, M., Modaresifar, H., Kiani, J., Hatami, M., Akasheh, S., Rezagholian, A., Kamali, G., 2021. A multilayered ergono- mic intervention program on reducing musculoskeletal disorders in an industrial complex: A dynamic participatory approach. Int. J. Ind. Ergon. 86, 103221. https://doi.org/10.1016/j.ergon.2021.103221 Claeys, A., Hoedt, S., Domken, C., Aghezzaf, E.-H., Claeys, D., Cottyn, J., 2022.
Methodology to integrate ergonomics information in contextualized digital work instructions. Procedia CIRP 106, 168–173. https://doi.org/10.1016/j.procir.2022. 02.173 European Commission, D. G. for R. and I., 2021. Industry 5.0: towards a sustainable, human centric and resilient European industry. Publications Office, LU. Fernández-Caramés, T., Fraga-Lamas, P., 2018. Design of a Fog Computing,
Blockchain and IoT-Based Continuous Glucose Monitoring System for Crowd- sourcing mHealth, in: 5th International Electronic Conference on Sensors and Applications. Presented at the International Electronic Conference on Sensors and Applications, MDPI, p. 37. https://doi.org/10.3390/ecsa-5-05757 Fraga-Lamas, P., Barros, D., Lopes, S. I., Fernández-Caramés, T. M., 2022. Mist and Edge Computing Cyber-Physical Human-Centered Systems for Industry 5.0: A Cost-Effective IoT Thermal Imaging Safety System. Sensors 22, 8500. https: //doi.org/10.3390/s22218500 Greene, R. L., Azari, D. P., Hu, Y. H., Radwin, R. G., 2017. Visualizing stressful
aspects of repetitive motion tasks and opportunities for ergonomic improvements using computer vision. Appl. Ergon. 65, 461–472. https://doi.org/10.1016/j.aper go.2017.02.020 Grobelny, J., Michalski, R., 2020. Preventing Work-Related Musculoskeletal Disorders in Manufacturing by Digital Human Modeling. Int. J. Environ. Res. Public. Health 17, 8676. https://doi.org/10.3390/ijerph17228676 Jayaram, U., Jayaram, S., Shaikh, I., Kim, Y., Palmer, C., 2006. Introducing quan-
titative analysis methods into virtual environments for real-time and continuous ergonomic evaluations. Comput. Ind. 57, 283–296. https://doi.org/10.1016/j.co mpind.2005.12.005 Joshi, M., Deshpande, V., 2019. A systematic review of comparative studies on ergonomic assessment techniques. Int. J. Ind. Ergon. 74, 102865. https://doi.org/10. 1016/j.ergon.2019.102865 Mourtzis, D., Angelopoulos, J., Panopoulos, N., 2022. Operator 5.0: A Survey on
Enabling Technologies and a Framework for Digital Manufacturing Based on Extended Reality. J. Mach. Eng. 22, 43–69. https://doi.org/10.36897/jme/147160 Mylonas, G., Kalogeras, A., Kalogeras, G., Anagnostopoulos, C., Alexakos, C., Munoz, L., 2021. Digital Twins From Smart Manufacturing to Smart Cities: A Survey. IEEE Access 9, 143222–143249. https://doi.org/10.1109/ACCESS.2021. 3120843 Puthenveetil, S. C., Daphalapurkar, C. P., Zhu, W., Leu, M. C., Liu, X. F., Gilpin-
Mcminn, J. K., Snodgrass, S. D., 2015. Computer-automated ergonomic analysis based on motion capture and assembly simulation. Virtual Real. 19, 119–128. https://doi.org/10.1007/s10055-015-0261-9
Rinaldi, M., Fera, M., Bottani, E., Grosse, E. H., 2022. Workforce scheduling incorporating worker skills and ergonomic constraints. Comput. Ind. Eng. 168, 108107. https://doi.org/10.1016/j.cie.2022.108107 Tanganelli, G., Cassano, L., Miele, A., Vallati, C., 2020. A methodology for the
design and deployment of distributed cyber–physical systems for smart environ- ments. Future Gener. Comput. Syst. 109, 420–430. https://doi.org/10.1016/j.futu re.2020.02.047 Wang, B., Zhou, H., Yang, G., Li, X., Yang, H., 2022. Human Digital Twin (HDT) Driven Human-Cyber-Physical Systems: Key Technologies and Applica- tions. Chin. J. Mech. Eng. 35, 11. https://doi.org/10.1186/s10033-022-00680-w Yin, M.-Y., Li, J.-G., 2023. A systematic review on digital human models in assembly
process planning. Int. J. Adv. Manuf. Technol. https://doi.org/10.1007/s00170- 023-10804-8 Zhang, Y., Guo, Z., Lv, J., Liu, Y., 2018. A Framework for Smart Production- Logistics Systems Based on CPS and Industrial IoT. IEEE Trans. Ind. Inform. 14, 4019–4032. https://doi.org/10.1109/TII.2018.2845683
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Aspects of Advanced Manufacturing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-56-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional