Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Machine Learning improves use of Haptic Glove for engineers in Virtual Reality

Publication year
Digital Object Identifier10.54941/ahfe100979

Abstract

Haptic gloves with force feedback represent new and immersive devices for Virtual Reality (VR). They enable interaction with virtual objects and have a positive impact on virtual engineering processes. The position of the hand and its specific finger positions, such as grip types, are tracked in virtual space dur-ing assembly processes. Implementing rule-based recognition of these grip types is complex and error-prone due to hard- and software limitations. Ma-chine Learning (ML) can support engineers during the use and implementation of these applications by classifying user input as specific grip types. Two ML algorithms, one Neural Network (NN) and one Support Vector Machine (SVM), that detect nine grip types at runtime by only using the joint angles of the gloves exoskeleton as features, were developed and compared with a rule-based algorithm. Our research shows, that the ML algorithm reach a very high accuracy with only reading one feature compared to the rule-based algorithm.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

INTRODUCTION

During product development processes it is necessary to validate early on if and how it is possible to assemble the product according to plan. Therefore, simulation software is available, which provides functions that enable collision free path planning of components. In manual assembly processes it is furthermore necessary to add the user as a factor to the simulation, which can be done by using digital human models (DHM). These are digital representations of humans to assess ergonomic criteria like visibility, physical load or the path workers walk during their assembly task. Another possibility to assess the mountability with digital models is to use Virtual Reality (VR). In VR it is possible to conduct the task manually and assess whether or not a collision free path exists. This also enables companies to conduct assembly simulations with workers who have experience from their day to day tasks from the factories. Furthermore, it is possible to quickly assess different variants of assembly processes. One main impediment of using the results of the simulation of one individual worker is, that the values are often only valid for the person who performed the task, as people have different anthropometry and experiences. To eliminate this impediment, it is possible to transfer the interactions the user performed in VR to a DHM. On the one hand this increases the realm of the simulation, on the other hand it is possible to simulate manikin families within DHM software, where DHM are represented with relevant anthropometric differences (e.g. small, middle, tall body sizes). To be able to use the simulation it is necessary to define the simulation constraints. The constraints are grip type, grip position as well as how open or closed the hands are and the path of the object. Whereas it is easy to read the hand information in VR, it is difficult to detect the right grip type a user performed in VR.

Haptic feedback can enrich the VR experience by enabling tactile feedback to the user and therefore imitate surfaces of assembly objects to the engineer. For an ongoing research project, we are evaluating haptic gloves with force feedback as an input method for digital human models. Therefore, we investigate different machine learning (ML) algorithms as well as a rule-based algorithm to identify nine different grip types while grasping in VR. The grip types as seen in Figure 1 are used for the DHM in the software IPS IMMA (Hanson et al. 2011): the Chuck Grip (CG), Cylindrical Power Grip (CPG), Lateral Pinch (LP), Pistol Grip (PG), Diagonal Power Grip (DPG), Parallel Extension (PE), Prismatic Pinch (PP), Spherical Grip (SP) and Tip Pinch (TP). We conducted a user study to gain training data for the algorithms and evaluated the results of the algorithms with 12 participants.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 1. Image representation of all evaluated grip types that have been classified with the algorithms GRIP TYPE RECOGNITION IN VR
Original PDF Figure 1.

Multiple rule-based algorithms to detect grip types have emerged in the past, which make use of more than one feature in order to reach a high accuracy of grip type detection. (Zou et al. 2019) use the position of the fingertips and the direction of the palm, whereas (Aleotti and Caselli May 15-19, 2006) reach an accuracy rate of 94% for experienced users while taking three features into account: the position of the finger tips, the rate of how the finger touches an object and an additional movement parameter for the virtual fingers. Some input devices might not detect additional parameters which are needed for those algorithms, like the rate of touch or the hands velocity, and need to be implemented by the engineer according to the use case and in regards to the hardware limitation. Using ML for classification might help to predict grip types with devices that only support the joint angles of the fingers and help speed up implementation processes across multiple levels of user experience. Figure 2 shows an image with a user wearing the haptic glove SenseGlove DK1 that enables tactile feedback in VR. Whenever the user touches surfaces in virtual space, the exoskeleton of the glove blocks its articulation and therefore stops the fingertips from moving. This creates the impression of touch for the user in VR.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 2: SenseGlove DK1 haptic gloves worn by user with attached HTC Vive Tracker (left) and hand representation in VR (right) ML has already shown potential in engineering processes, for example as diagnostic tool for engine performance analysis (Giorgi et al. 2018), for complex pattern recognition from sensor data (Wuest et al. 2016) and for predictive maintenance to prevent system failures (Susto et al. 2015). ML excels in systems with a lot of data and is especially useful when the available data and its relations to each other are very complex and not easily interpreted.
Original PDF Figure 2:

Furthermore, many VR applications make use of ML for related fields like robotic, autonomous cars and advanced visualization (Reiners et al. 2021). Our goal was, to investigate if we can utilize ML for grip type classification with a limited number of features, which might not be feasible for a rule-based implementation.

USER STUDY Of 20 people who participated in the user study, 8 contributed to generating the data for the classification algorithms and 12 people verified the efficiency of the algorithms within a study in VR. Both groups had to perform all nine grip types three times, which were visualized with images in VR as seen in Figure 3. For each of the nine grip types we provided an image representation and one golden object to grasp. The test participants were all between 20 and 25 years old. Two of the 20 participants already had some experience in using VR and the SenseGlove.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 3: set-up for the user study in VR with grip type images and golden objects to grasp For each frame at a rate of 60 frames per second ten joint angles were recorded during the execution procedure. The values consist of two angles per finger which describe the angle of the exoskeleton. One is the extension/flexion in y-axis and the other consists of the z-axis which is the abduction/adduction. Table 1 shows the representation of one frame, with joint angle values for one finger and two tracked axis as shown in Figure 4 (SenseGlove). The full training dataset contained 13.425 fields of data, which averages 1.492 samples per gesture.
Original PDF Figure 3:
Figure 4: Axis of the hands as SenseGlove provides the data Table 1: Example of output for the joint angles of the thumb when performing a Chuck Grip. Finger Axis Joint Angle Normalized feature

Figure available in the original PDF paper.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 5: Execution of a Tip Pinch in VR with haptic gloves during a user study The classification rate for each algorithm was compared by their accuracy. Additionally, the F1-score was used as a metric to measure the performance of the ML algorithms. This metric factors in precision, which is the fraction of true positives compared to false positives and recall, which gives information about how many true positives are classified correctly rather than missed (Avola et al. 10/6/2019 - 10/9/2019).
Original PDF Figure 5:

RESULTS

Table 2 to 3 show the confusion matrices for both ML algorithms. Both algorithms were able to classify most grip types very well, with multiple grip types reaching 100% accuracy. There are three grip types that stand out with bad recognition rates across all algorithms. Pistol Grip, Parallel Extension and Prismatic Pinch have higher chance to be misclassified.

Table 2: Confusion Matrix for NN in percent with performed gesture (at the top) and recognized gesture (on the left) for Chuck Grip (CG), Cylindrical Power Grip (CPG),
Original PDF Table 2:

Lateral Pinch (LP), Pistol Grip (PG), Diagonal Power Grip (DPG), Parallel Extension (PE), Prismatic Pinch (PP), Spherical Grip (SP) and Tip Pinch (TP).

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

TP 0 0 0 0 0 0 0 0 100 Table 3: Confusion Matrix for SVM in percent with performed gesture at the top and recognized gesture on the left for Chuck Grip (CG), Cylindrical Power Grip (CPG),

Lateral Pinch (LP), Pistol Grip (PG), Diagonal Power Grip (DPG), Parallel Extension (PE), Prismatic Pinch (PP), Spherical Grip (SP) and Tip Pinch (TP).

Table 4 shows the F1 Score for both ML algorithms across all grip types. The mean value is at 0.82 for the NN and 0.85 for the SVM, whereas the standard deviation shows a low variability with 0.15 for both algorithms.

Table 4: F1-Score for all users with Chuck Grip (CG), Cylindrical Power Grip (CPG), Lateral Pinch (LP), Pistol Grip (PG), Diagonal Power Grip (DPG), Parallel
Original PDF Table 4:

Extension (PE), Prismatic Pinch (PP), Spherical Grip (SP) and Tip Pinch (TP), Mean Value (MV) and Standard Deviation (SD).

SVM 0.89 0.86 0.91 0.52 0.96 0.95 0.69 0.91 0.96 0.85 0.15 Table 5 shows a comparison of the classification rates for all three algorithms. The rule-based algorithm reaches a low mean value of 45.27 % across all grip types, with good results above 79% for the PG and the CPG and detection rates of below 50% for CG, LP, PE, PP and TP. The NN reaches a high mean value of 81.45% with a standard deviation of 22.87% whereas the SVM reaches 85.53% mean value with 16.78% standard deviation.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Table 5: Comparison of all algorithms in percent with probabilities that a performed grip type is classified correctly with Chuck Grip (CG), Cylindrical Power Grip (CPG), Lateral Pinch (LP),
Original PDF Table 5:

Pistol Grip (PG), Diagonal Power Grip (DPG), Parallel Extension (PE), Prismatic Pinch (PP), Spherical Grip (SP) and Tip Pinch (TP), Mean Value (MV) and Standard Deviation (SD).

DISCUSSION

Overall the results are satisfying to the research team, as they show, that both ML algorithm reach high probabilities to correctly classify grip types with the provided features. Considering all the results one notices the poor values of the Pistol Grips across all algorithms. This leads us to believe that the user group responsible for creating the training data had difficulties to execute the grip type. The accuracy of this grip could be improved by recording the training data once more with more specific instructions given to the participants and by providing a pistol grip as interaction object. The second noticeable bad recognition rates can be seen for Parallel Extension and Prismatic Pinch which is most apparent for the NN. Both grip types are very similar to the Spherical Grip and mainly differ in the pronation of the thumb, which was not considered for these algorithms. In addition to this, the training of the Spherical Grip required more training data, because it covers a wider range of motion than the other grip types. This might have led to overfitting for this specific grip type, mainly affecting the NN. To compensate this, the weights and biases of the affected classes could be adjusted so the Spherical Grip doesn’t get favored anymore and the data of the Spherical Grip could be thinned out to better fit in with the rest of the data. Using the rule-based algorithm the Tip Pinch has not once been correctly recognized, which leads us to think that the range of valid angles for this grip type has been set up too narrow. This error has not been noticed during early validation phases of the implementation and need to be investigated further.

We furthermore noticed, that the more experienced users had far better results and reached rates of 100% for many grip types. This group was unfortunately too small in order for us to properly separate and analyze the data for our user study. One possibility would be to repeat the study with a group of more experienced users in order to compare the results.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

CONCLUSION

Two ML models and one rule-based algorithm have been implemented and compared, that detect nine grip types at runtime by only using two joint angles of the exoskeleton as features. Our research shows, that the ML algorithms reach a very high classification probability of more than 80% for most grip types.

These probabilities show very good results with limited input data and short assess and implementation time. Once the algorithms are enhanced, we could apply the same approach to different haptic gloves in order to compare its efficiency for different hardware and features. Furthermore, the test group of experienced VR users has been too small so the study should be repeated with a bigger group of experienced VR users in order to properly assess differences in classification rates.

ACKNOWLEDGMENTS

This work was developed within the project called ED-VIMA (01QE1937C), which was funded by EUROSTARS.

REFERENCES

Aleotti, J./Caselli, S. (2006). Grasp recognition in virtual reality for robot pregrasp planning by demonstration. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006, 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006, Orlando, FL, USA, May 15-19, 2006. IEEE, 2801–2806. Avola, Danilo/Cinque, Luigi/Foresti, Gian Luca/Lamacchia, Francesco/Marini,

Marco Raoul/Perini, Luca/Qorraj, Kristjana/Telesca, Gabriele (2019 - 2019). A Shape Comparison Reinforcement Method Based on Feature Extractors and F1- Score. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy, 10/6/2019 - 10/9/2019. IEEE, 2155–2159. Giorgi, Maria Grazia De/Campilongo, Stefano/Ficarella, Antonio (2018). A diagnostics tool for aero-engines health monitoring using machine learning technique. Energy Procedia 148, 860–867. https://doi.org/10.1016/j.egypro.2018.08.109. Hanson, L./Högberg, L./Bohlin, R./Carlson, J. S. (2011). IMMA – Intelligently

Moving Manikins – Project Status 2011. In: First International Symposium on Digital Human Modeling. June 14-16, 2011 Lyon, France : conference programme. [Lyon, LBMC]. Reiners, Dirk/Davahli, Mohammad Reza/Karwowski, Waldemar/Cruz-Neira, Carolina (2021). The Combination of Artificial Intelligence and Extended Reality: A Systematic Review. Frontiers in Virtual Reality 2, 1143. https://doi.org/10.3389/frvir.2021.721933.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

SenseGlove. Sense Glove coordinate system, https://github.com/Adjuvo/SenseGlove-API/wiki/HandPose-Reference. Susto, Gian Antonio/Schirru, Andrea/Pampuri, Simone/McLoone, Sean/Beghi, Alessandro (2015). Machine Learning for Predictive Maintenance: A Multiple Classifier Approach. IEEE Transactions on Industrial Informatics 11 (3), 812– 820. https://doi.org/10.1109/TII.2014.2349359. Wuest, Thorsten/Weimer, Daniel/Irgens, Christopher/Thoben, Klaus-Dieter (2016).

Machine learning in manufacturing: advantages, challenges, and applications. Production & Manufacturing Research 4 (1), 23–45. https://doi.org/10.1080/21693277.2016.1192517. Zou, Yuanyuan/Liu, Honghai/Zhang, Jilong (2019). Real-Time Grasp Type Recognition Using Leap Motion Controller. In: Haibin Yu/Jinguo Liu/Lianqing Liu et al. (Eds.). Intelligent Robotics and Applications. Cham, Springer International Publishing, 472–480.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8988-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →