Saverio Serino, Carlotta Fontana, Rosaria Califano, Nicola Cappetti, Alessandro Naddeo (2022). Virtual human centered design: an affordable and accurate tool for motion capture in mixed reality. In: Human Factors in Virtual Environments and Game Design. AHFE Open Access, vol. 50, pp. 211–218. https://doi.org/10.54941/ahfe1002080
The introduction of Digital Human Modeling and Virtual Production in the industrial field has made possible to bring the user to the center of the project in order to guarantee the safety of workers and well-being in the performance of any activity. Traditional methods of motion capture are unable to represent user interaction with the environment. The user runs a simulation without the realistic objects, so his behavior and his movements are inaccurate due to the lack of real interaction. Mixed reality, through a combination of real objects and virtual environment, allows to increase the human-object interaction, improving the accuracy of the simulation. A real-time motion capture system produces considerable advantages: the possibility of modifying the action performed by the simulator in real time, the possibility of modifying the user's posture and obtaining feedback on it, and finally, after having suffered a post - data processing, without first processing the recorded animation. These developments have introduced Motion Capture (MoCap) technology into industrial applications, which is used for assessing and occupational safety risks, maintenance procedures and assembly steps. However, real-time motion capture techniques are very expensive due to the required equipment. The aim of this work, therefore, is to create an inexpensive MoCap tool while maintaining high accuracy in the acquisition. In this work, the potential of the Unreal Engine software was initially analyzed, in terms of ergonomic simulations. Subsequently, a case study was carried out inside the passenger compartment of the vehicle, simulating an infotainment reachability test and acquiring the law of motion. This procedure was performed through two cheap MoCap techniques: through an optical system, using ArUco markers and through a markerless optical system, using the Microsoft Kinect® as a depth sensor. The comparison of the results showed an average difference, in terms of calculated angles, between the two methodologies, of about 2,5 degrees. Thanks to this small error, the developed methods allows to have a simulation in mixed reality with user’s presence and offers an accurate analysis of performed movements.
(WMSD) due to high production rates, improper environment design and improper work execution (Bommisetty et al. 2018).” Therefore, to avoid production slowdowns and absenteeism during work shifts, thanks to the evolution of virtual prototyping, engineers are allowed to design and analyze ergonomics in the design phase. “Respect for ergonomic principles translates into satisfaction of the well-being of the worker in terms of health and safety and greater productivity (Satheeshkumar, 2014).”
In recent decades, the use of three-dimensional human digital models (DHMs) within software has become increasingly widespread. “The use of human models, able to replicate the kinematics of the human body, scaled on the basis of anthropometric measurements, allows to create interactions between the user and the surrounding environment, evaluating their ergono- mics and other factors (Maurya et al. 2019).” The main fields of application of Digital Human Modeling in the industrial sector are: “evaluation of assem- bly and maintenance, evaluation of car interiors, design and optimization of work environments; simulation of human-robot collaboration, study of human behavior; formation (Zhu et al. 2019).”
With the discovery of the potential of virtual reality, “this technology has become increasingly consolidated in fields such as engineering, medicine, architecture and cinema (Wolfartsberger, 2019)”. As part of the design pro- cess, the interactive virtual prototype must be able to represent the human senses and simulate interactions with objects but it should also provide real-time feedback, which means having high-performance hardware and algorithms. For this reason, a virtual reality simulation doesn’t have to be very complex. To overcome these problems mixed reality (MR) is used which combines both the virtual world and the real world. With this technology, you have an immersion in both the real and virtual world, having both physical and virtual interaction available. In this work, mixed user interaction was performed with a real prototype. This typology consists in the use of tools that interact with the virtual and real object. “The advantages are obtained from the use of virtual prototyping, reducing the costs of making physical prototypes and reducing the time-to-market, and the advantages in using the physical object by increasing the sensation and perception of the object (Bordegoni et al. 2009).”
The purpose of the work was to create a tool within Unreal Engine in mixed reality that can capture the user’s movement and provide the law of motion in real time. A case study has been defined: a reachability test inside the passenger compartment of the car reproduced through a seating buck. The tests were carried out with two different motion capture systems: a marker-based system (ArUco Markers) and a markerless system, with the use of a depth sensor (Microsoft Kinect®).
METHODS
Mannequin Implementation For the implementation of an ergonomic analysis tool in Virtual Reality, the first step was the choice and the import, into the graphic engine, of a
Virtual Human Centered Design: An Affordable and Accurate Tool 213
Three male and female mannequins, belonging to the 5th, 50th and 95th percentile, were made. These percentiles are used to cover approximately 95% of the population. The anthropometric values used for the development of the mannequins, configured within the software, are reported in Table 1.
A rigging phase has been implemented, in order to let “the definition of a real digital human mannequin and to make it complete under the move- ments’ point of view (Arshad et al. 2019).” To simulate human behavior as accurately as possible, inverse kinematics is, normally, used; “Unreal Engine offers different solutions, through the implementation of several algorithms of inverse kinematics, such as the Fabrik (Aristidou et al. 2011).”In this work, the realization of a rigging for a forward kinematics is presented, as the goal is to create an interface in Unreal Engine that allows to animate the mannequin by inserting angles acquired by motion capture techniques, such as in DHM software. The Plugin “Control Rig” was used to create the rigging within the Unreal Engine software, with the aim to generate a hierarchy between bones. “Ranges of motion related to the various limbs have been inserted in order to accurately represent human behavior (Naddeo et al. 2015).”
ArUco Marker ArUco markers are used to detect body movements. The ArUco module is based on the “ArUco library”, a popular library for the detection of quadratic confidence indicators developed by Rafael Muñoz and Sergio Garrido. “An ArUco marker is a synthetic square marker composed of a large black border and an internal binary matrix that determines its identifier (id). The black border facilitates quick detection in the image, and binary coding enables identification and application of error detection and correction techniques. The size of the marker varies depending on the size of the internal matrix (Garrido-Jurado et al. 2014).” Given an image in which the ArUco markers are visible, the detection process must return a list of coordinates.
Figure 1: 3D CAD model of the built set-up.Figure 2: Position 0 and Position 1, related to Test 1 (on the left), and Position 1 of Test 2 (on the right).
Set-up and Tools Used Three digital cameras (one Nikon D3100 and two Nikon D3300) were used to frame the front, the side and the top views, with a frame size of 1920x1080 pixels and a capture rate of 24fps. An experiment setup has been set in order to represent, as better as possible, the standard configuration of an auto- motive cockpit. The setup (Figure 1) was built respecting certain dimensions such as the distance of the seat from the steering wheel, the distance of the infotainment from the steering wheel and from the seat, and the distance of the pedal unit from the steering wheel.
Basic mechanical assembly elements produced by Bosch Rexroth were used to create the physical set-up such as the 45x45L Profiles with 10 mm throat of various lengths and connecting elements such as the Angular 45/45 (with centering lugs for quick, precise and torsional - free assembly), hammer nuts for correct positioning in the profile groove and collar screws. At the base of the seats there are aluminum plates to allow users to place their feet on the ground.
Testing Protocol The testing protocol used is the following.
TEST 1: Position 0: both hands on the wheel →Position 1: the right arm pushes the handle of the INFOTAINMENT SYSTEM →Position 3: return to position 0 (Figure 3).
TEST 2: Position 0: both hands on the wheel →Position 1: the right arm rotates the infotainment volume knob (Figure 3) →Position 2: return to position 0.
Virtual Human Centered Design: An Affordable and Accurate Tool 215
Figure 3: Location of the ArUco markers.
Experiments Execution – ArUco Markers Participants were equipped with 22 markers for performing the experiments with ArUco system (see Figure 2) Each human joint is equipped with a couple of markers in order to identify the 3D coordinates of the points useful for creating a motion vector.
“The post-processing algorithm made it possible to create vectors only if both selected markers were present in an analyzed frame. The Code in Matlab performs the vector calculation, and the procedure is repeated for each pair of markers (Fiorillo et al. 2021).” The outputs are the angle of rotation and the first non-zero frame from which the analysis starts. This value allows you to choose the pair of markers to use, as we will certainly choose the one with the lowest starting frame to be sure not to skip any movement.
Experiments Execution – Markerless Motion Capture Kinect v1 was used for motion detection without the use of markers. Ipi Motion Capture software was used to transfer the detected motion to Unreal Engine. Before starting the simulation, the background of the scene was acquired and pre-processed in order to allow identifying the moving obje- cts. The system was able to stream the user movements in Unreal Engine, in real time.
Vr Tools and Hardware An Oculus Rift S viewer was used for virtual reality application. A Dell Preci- sion Tower 3620 with 32GB of Ram and 8 GB of Radeon (TM) Pro WX 5100 Graphics System has been used as hardware for VR application running. The tool implemented in Unreal Engine has the purpose of providing the move- ments made by the user to the mannequin, in terms of rotation angles. To simplify the post-processing of the data, the exit angles have been calcula- ted in such a way that the rotations performed by the joints (connecting the bones) are referred to the previous position.
RESULTS
The results obtained by the two motion capture systems have been compared. Analyzed data are referred to the right arm extension. The comparison was
Figure 4: Angle of extension upper arm right.Figure 5: Angle of extension lower arm right.Figure 6: Angle of abduction upper arm right.
made for both movements, i.e., when the user presses the button and when the user turns the knob. We can see a difference of 2 and 4 degrees among the measurements and a deviation of 1.5 degree respect the exact location for the Kinect and 1.75 degree for the ArUco Marker (see Figure 4).
The lower arm extension of the right arm was also analyzed. In this case we have a maximum difference between the two readings of 4 degrees and a deviation of 1.75 degree respect the exact location for the Kinect and 1.5 degree for the ArUco Marker (see Figure 5).
Furthermore, abduction of the right arm was compared. In this case a difference of maximum 2 and 1 degrees has been detected and a deviation of 0.75 degree respect the exact location for the Kinect and 1.5 degree for the ArUco Marker (see Figure 6).
The discrepancy between the two systems can be caused by multiple factors related to the ArUco system, some of them are: errors in the positioning of the ArUco markers, that can generate a measurement error; error in the acquisition of the Marker due for example to strong lighting; on the other
Virtual Human Centered Design: An Affordable and Accurate Tool 217
hand, the markerless system is affected by noises during the detection and by the randomness of movements (towards the one for which is, by default, programmed.
CONCLUSION
This work focused on creating a tool that offers the ability to apply marker- free motion capture in real time and accurately evaluate movements perfor- med in a mixed reality environment. This system is applicable in any other required field: ergonomists, training, maintenance, videogames, virtual expe- riences. The software that allows full integration into virtual reality is Unreal Engine.
A mannequin was developed to be easily imported into Unreal Engine and was evaluated against digital human models used in other software. A character rigging was created in order to create animations to represent cer- tain movements, knowing the law of movement. This tool can be modified if you are not interested in reconstructing an action, but want to simu- late a movement. Finally, an inexpensive and precise tool was created that allow the acquisition of the law of motion. Despite the more sophistica- ted and expensive systems, the tested systems are easy to apply and have been shown to provide accurate results. These results can be further impro- ved if a combination of sensors is used simultaneously, in order to cover the simulation area at different angles. The comparison made between the Mar- kerless MoCap and the ArUco Markers generated not only good accuracy, but also good precision, with an average difference between the two measu- rements of about 2.5°. This discrepancy is mainly due to the uncertainty of the movement, which is difficult to be repeated precisely. by the position of the ArUco markers, the type of sensor used and hardware used to process the acquisition.
The limits due to the application described are: the limitation of the tests carried out, in terms of type of movements and number of tests; the use of Kinect that is out of production; the use of a single sensor, so with problems in detecting motion when a part of the body covers itself; inability to acquire hand movement; the expensive analysis through ArUco markers, which con- sists of numerous steps in different software that limit their use for real-time applications.
REFERENCES
Aristidou, A. and Lasenby, J. (2011) “FABRIK: A fast, iterative solver for the Inverse Kinematics problem,” Graphical Models, 73(5). doi:10.1016/ j.gmod.2011.05.003. Arshad, M.R., Yoon K. H., Manaf A. A. A., and Ghazali M. A. M., (2019) “Physical
rigging procedures based on character type and design in 3D animation,” Inter- national Journal of Recent Technology and Engineering, 8(3), pp. 4138–4147. doi:10.35940/ijrte.C5484.098319. Bommisetty, U. and Rajamane, M. (2018) “International Journal of Science and Management Studies (IJSMS) Impact of Recent Advances in Ergonomic Design and Digital Human Models in Industrial Environment.” Available at: www.ijsmsjournal.org.
Bordegoni, M., Cugini U., Caruso G., and Polistina S., (2009) “Mixed prototyping for product assessment: A reference framework,” International Journal on Interactive Design and Manufacturing, 3(3), pp. 177–187. doi:10.1007/s12008- 009-0073-9. Fiorillo, I., Nasti, M. and Naddeo, A. (2021) “Design for comfort and social
interaction in future vehicles: A study on the leg space between facing-seats configuration,” International Journal of Industrial Ergonomics, 83, p. 103131. doi:10.1016/j.ergon.2021.103131. Garrido-Jurado, S. et al. (2014) “Automatic generation and detection of highly reliable fiducial markers under occlusion,”Pattern Recognition, 47(6), pp. 2280–2292. doi:https://doi.org/10.1016/j.patcog.2014.01.005. Maurya, C.M., Karmakar, S. and Das, A.K. (2019) “Digital human modeling (DHM)
for improving work environment for specially-abled and elderly,” SN Applied Sciences. Springer Nature. doi:10.1007/s42452-019-1399-y. Naddeo, A., Cappetti N., Vallone M., Califano R., (2015) “Comfort Driven Redesign Methods: An Application to Mattresses Production Systems,” The Open Mecha- nical Engineering Journal, 9, pp. 492–507. doi:10.2174/1874155X01509010492. Naddeo, A., Cappetti, N. and D’Oria, C. (2015) “Proposal of a new quantita-
tive method for postural comfort evaluation,” International Journal of Industrial Ergonomics, 48, pp. 25–35. doi:10.1016/j.ergon.2015.03.008. Satheeshkumar, M. (2014) Digital Human Modeling Approach In Ergonomic Design And Evaluation - A Review. Available at: http://www.ijser.org. Wolfartsberger, J. (2019) “Analyzing the potential of Virtual Reality for engineering design review,” Automation in Construction, 104, pp. 27–37. doi:10.1016/j.autcon.2019.03.018. Zhu, W., Fan, X. and Zhang, Y. (2019) “Applications and research trends of digi-
tal human models in the manufacturing industry,” Virtual Reality & Intelligent Hardware, 1(6), pp. 558–579. doi:https://doi.org/10.1016/j.vrih.2019.09.005.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Virtual Environments and Game Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-26-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional