Bethany Lowndes, Ana Laura Pineda Gutierrez, Santosh Pitla, Salem Rumuri, Joseph Siu, Aaron Yoder (2024). Feasibility of Integrating Electromyography and Computer Vision for Occupational Safety during Tractor Ingress and Egress. In: Safety Management and Human Factors. AHFE Open Access, vol. 151, pp. 47–55. https://doi.org/10.54941/ahfe1005301
Entering and exiting a tractor requires strength, muscle coordination, and behaviors that can prevent injuries Research indicates between 15-20% of tractor-related injuries occur during ingress and egress (entry and exit, respectively) (Douphrate et al., 2009). Most agricultural producers know it is important to maintain three points of contact, to face toward the cab when climbing in and out, to have clean steps, and to wear anti-slip shoes to ensure safe ingress/egress. However, these steps are not always taken by producers in the field. Efforts are needed to improve safety of ingress and egress behaviors including learning more about human performance and capabilities. This paper will describe the application of a custom computer vision system to assess behaviors for tractor ingress and egress along with the physical performance. The objective of this study was to demonstrate the feasibility of integrating EMG and computer vision data to study biomechanics and behavior to assess tractor ingress and egress safety. Two participants had EMG sensors placed bilaterally. They completed a grip strength test using a hand dynamometer and then climbed into and out of the tractor. The extensor carpi radialis longus (ECRL) muscle activation on both sides demonstrated the highest activation as a percentage of the maximum activation. When climbing down, the participants’ shoulders retracted, causing the ECRL to have more activation than the ingress movements. However, during ingress the participants received more visual information by looking at their hands and rails, which could contribute to why the strategy they use is completely different. During egress the participant was observed to focus on their legs and the visual cues they received were in the form of looking down at their feet. The value of these different visual stimuli is vastly different and require further research to see how they contribute to overall movement strategies. This could lead participants to either leaning toward the cab or away from the cab and requiring different muscle activation in their upper body. It is anticipated that during egress the leg muscles could be more active than the arm muscles with even more activation during ingress due to working against gravity to climb into the cab. The computer vision algorithm was able categorize safety risk levels during the trials. The use of EMG combined with computer vision has the potential to describe movement patterns and behaviors that could impact ingress and egress safety. Further refinement and synchronization of these systems are needed to use this method for developing and testing targeted fall prevention interventions and to create user-centered ingress and egress design solutions.
Ingress and Egress Safety Research indicates between 15-20% of tractor-related injuries occur during ingress and egress (entry and exit, respectively; Douphrate et al., 2009). Falls during ingress and egress can result in very serious injuries due to the height of some farm machinery and other aspects of the equipment. Climbing requires a combination of strength, muscle coordination, and behaviors that facilitate safe task completion. This can become more challenging as producers age and both strength and mobility decrease.
Biomechanics Strength is required to safely complete farming tasks—including machinery ingress and egress—without overexertion, slipping, or falling. However, the specific strength requirements (upper or lower limb) for entering and exiting (ingress and egress, respectively) agricultural machinery or recovering from a slip have not been quantified.
Grip strength is one component of the tractor ingress/egress task. One way of quantifying grip strength is the handgrip test (Bohannon et al., 2015). Grip strength has been quantified as a form to reliably indicate the physical function and health status of a population (Huerta et al., 2021). A loss of hand grip strength can result in the decrease of contact forces with surfaces which increases the likelihood of fall-related injuries (Watts et al., 2008). Electromyography (EMG) has previously been used as a non-invasive tech- nique to quantify muscle activity during certain movements. Understanding to what extent and when the muscles are active during the required agricul- tural task is useful for the purpose of understanding the mechanism of an injury (Benos et al., 2020). Grip strength is a more feasible measurement of strength for field testing compared to lower limb strength testing. Therefore, we will focus on grips strength and muscle activation for this pilot feasibility study.
Ingress and Egress Behaviors Most agricultural producers know it is important to maintain three points of contact, to face toward the cab when climbing in and out, to have clean steps, and to wear anti-slip shoes to ensure safe ingress/egress. These are important safety measures to prevent a fall and injury from fall. However, these steps are not always taken by producers in the field. Previous research studies have conducted on-board monitoring to determine different behavior patterns (Jung et al., 2020). Efforts are needed to improve safety of ingress and egress behaviors including learning more about why and how the task is performed. This paper will describe the application of a custom computer vision system to assess behaviors for tractor ingress and egress.
Objectives The long-term goal of this research explores ways to improve the safety through the study of the workload required for agricultural producers to enter and exit tractors and design to support safe execution. The work will focus on supporting aging prodcuers. The specific objective of this study is to demonstrate the feasibility of integrating EMG and computer vision data to study upper limb biomechanics and behavior during tractor ingress and egress.
METHODS
For this pilot study, research members tested the equipment for monitoring ingress and egress behaviors through muscle activity and categorization of safety risk that could potentially be used for in field testing and training.
Participants There were 2 participants from the research team, 1 male and 1 female, for the data collection. Neither participant had a background using tractors in their daily routine. Participants were both 24 years of age. The male was 185cm with a shoe size of 26.7cm, and a hand size (from palm to tip of middle finger) of 20.75cm. The female was 157.5cm with a shoe size of 22cm, and hand size of 18cm.
Feasibility of Integrating Electromyography and Computer Vision 49
Equipment Electromyography (EMG) The researchers used an Electromyography (EMG) system by Delsys (Delsys Trigno LiteTM System, Boston, MA, USA) to track the electrical activity produced by muscles. The Trigno Mobile Sys- tem delivers EMG and movement data with a minimal setup on a tablet. This allowed the researcher to watch the activation in real-time. 3 EMG sensors were used which includes a dual-head sensor that was able to read smaller muscles such as the abductor pollicis brevis.
Agricultural Machinery Operator Monitoring System (Ag-OMS) This camera capture system and machine learning model has been trained to clas- sify the ingress/egress technique employed by individuals as they climb in or out of large machinery into safe (low-risk) and unsafe (medium-risk & high- risk) behaviors (Irumva et al., 2023). Our AG-OMS ML model was deployed with the help of an NVIDIA Jetson Nano 945-13541-0000-000 series con- nected to an Omnicharge P2F model as a power source. Jetson Nano is a powerful small computer capable of powering artificial intelligence applica- tions and devices. Attached to the Jetson Nano was an MSIP-REM-DZL- V-U0040 Logitech Webcam programmed to start recording once motion is detected.
Tractor for Ingress/Egress The tractor used for the study was a John Deere 7R Tractor (Serial No. 110101), North American Edition. This tractor had 2 handrails on the left side and 4 on the right-hand side that are considerably shorter than that of the left-hand side. Three of the rails on the right-hand side are found on the door once opened. The placement of the handrails can be seen below in Figure 1 (Left).
Computer Vision Our motion-triggered recording system was installed inside the cabin. A machine learning (ML) algorithm was designed to carry out feature identification and categorization to differentiate multiple oper- ators and evaluate the operator behavior during ingress/egress (i.e., three points of contact used for climbing, whether the operator is facing toward or away from cab door, and the levels of distraction). The ML models have been developed using a skeleton-detecting algorithm called OpenPose 1.7.0 to detect real-time human postures in a livestreaming video feed from a cam- era installed in the tractor cab. The model then has been trained on three separate categories of tractor operators’ safety behaviors, and this trained classifier will label operators’ safety behaviors in real-time based on the three safety classes.
Figure 1: (Left) highlights the handrails found on the tractor. The left handrails for the operator are mounted next to the steps. Three of the right handrails are mounted on the edge of the door and the fourth is to the right of the steps (right) the placement of the EMG sensors.
Ingress/Egress Task After the application of the EMG sensors, the par- ticipants were asked to complete a baseline grip strength test using a hand dynamometer. The participants squeezed the hand dynamometer for 5 sec- onds 3 times on each hand with a 30 second rest period between each set. After baseline testing was complete, participants were briefed on the stan- dard practice of climbing into and out of a tractor. The participants then climbed into the tractor, sat down for 1–3 seconds, and then climbed out of the tractor the same way they entered.
Data Processing For each task, the EMG data was downloaded from the Delsys Trio LiteTM EMG system as shpf and excel files. Participants 1 and 2 both generated data across 3 trials each for the ingress/egress tasks. There were 6 trials per participant for the grip strength task, 3 being from the left hand and 3 being from the right hand. Each of the 18 raw EMG files was filtered through the root mean square (RMS) filter when downloaded.
The videos of each participant were downloaded as an mp4 video sepa- rated per whole second. Videos were manually reviewed by a researcher to identify timestamps for ingress and egress behaviors within the file. The time (seconds) between task execution and completion was calculated and used to extract specific times relating to the ingress and egress movements. The occurrences of ingress and egress were separated for each EMG dataset (by trial) as identified in the video footage.
For each ingress and egress trial (6 per participant), the maximum and average muscle activation was calculated for every sensor/muscle. The maxi- mum activation across all 6 trials was calculated for each sensor/muscle using the highest number recorded during any trial. Grip strength data was sepa- rated between left and right hand, and the maximum and average for each sensor was calculated during seconds 2-3. The maximum activation across all 3 grip strength trials was calculated for each sensor/muscle. Trial Activations were normalized by comparing every average trial activation to the maxi- mum activation for each muscle. The maximum activation that was identified across all tasks (grip strength or ingress/egress tasks) was used for each trial comparisons.
RESULTS
Table 1. These data are separated by muscle, trial, and task. Each trial average muscle activation is a percentage of each muscle maximum for the participant.
Table available in the original PDF paper.
Abductor Pollicis Flexor Carpi
Average (% of Max) Extensor Carpi Radialis Longus (R)
Feasibility of Integrating Electromyography and Computer Vision 51
EMG The average EMG activation as a percentage of the maximum muscle activation is displayed in Table 1 for the ingress and egress by muscle for both participants. For Participant 2, there appears to be a slightly higher activation of the Right FCU during egress compared to ingress. No other potential patterns emerged in this limited dataset. Figures 2 and 3 show the RMS EMG data during ingress and egress for participant 2.
Figure 2: Ingress muscle activation for participant 2 during trial 1 from the RMS EMG data. All 4 muscles are included in this graph as while climbing into the tractor cabin.Figure 3: Egress muscle activation for participant 2 during trial 1 from the RMS EMG data. All 4 muscles are included in this graph as while climbing out of the tractor cabin.
Computer Vision The Ag-OMS computer vision algorithm categorized safety during the trials. Figures 4 demonstrates the OpenPose visualization with the low (left) or medium (right) safety risk label. Figure 5 is a short
excerpt from one egress task for Participant 2. This task was not part of the original 3 trials. Therefore, EMG was not collected and cannot be compared directly. However, it demonstrates the ability to categorize the safety behav- ior over time. In Figure 5, the participant has a medium risk at the beginning due to moving and hand and a leg at the same time. Most of the trial is in the low-risk category. The one instance where the software identified a high-risk incident was not actually high-risk (facing away from the cab) as determined by reviewing the video.
Figure 4: (Left) Participant 2 during ingress with a low risk behavior (facing cab and three points of contact). (Right) participant 2 during ingress with a medium risk behavior (facing cab and only two points of contact).Figure 5: Short Excerpt of the safety categorization for the egress task by participant 2 medium risk (Figure 4, left) and low risk (Figure 4, right).
The purpose of this study was to determine if using computer vision and EMG data would be a viable option for studying ingress and egress behav- iors. The results of this study found that the use of EMG tools to capture muscle activations combined with computer vision to identify safety risk that could potentially be used for in task testing and training, to shed light on the movement strategies that occur during tractor mounting and dismounting. As previously mentioned, each participant entered and exited the cabin of the tractor with 4 EMG sensors placed on different muscles. Specifically, muscles like the abductor pollicis brevis show significant activation while climbing to enter the cabin, as well as during egress movements when the participant was exiting the vehicle. Focusing on participant 2 trial 1, Figures 4 and 5 show each muscle activation during the ingress and egress period. These findings shed light on muscles that could be stressed during these activities if repeated often, if an individual is otherwise fatigued, or if individuals have lower grip
Feasibility of Integrating Electromyography and Computer Vision 53
strength. Additionally, for older individuals with reduced lower limb strength and mobility, they could rely more on their grip strength to safely complete this task. Monitoring agricultural producers during busy seasons, such as harvesting, could help researchers understand identify opportunities to sup- port producer safety in the future and prevent potential injuries with minimal invasion.
During ingress, the participant’s left hand is seen sliding up the railing while the right hand is used to climb into the cabin as well as being used to open the cabin door. This movement is consistent with other bimanual movement strategies where one hand primarily performs the task, and the other is geared towards stabilization and support (Saul et al., 2019). In gen- eral, while climbing there is one hand that is the anchor (holding its place and sliding up or down) and one hand is moving (letting go of a surface). The hand acting as the anchor tends to hold on with greater force while the moving hand tends to move lightly and increase grip force as needed (Stephen et al., 2023). While climbing into the cabin, the APB muscle had the largest mean and maximum activation values across all trials. The APB muscle’s pri- mary function is to oppose and extend the thumb, which is one of the main movements that contribute to grip. Many of the ingress movements require extensive grip force, which is supported by the APB values being the most present during the task. As observed in the video, participant 2 grabbed onto the rail/doorhandle around 5 times with their right hand, leading to the spikes of EMG activation seen within the APB muscle during ingress. While climb- ing in, the ECRL on the right hand was less active compared to the left hand ECRL. This is likely because more tension was placed on the left-hand side to compensate for the gravitational pull and stabilize this side to act as an anchor for the entire upward movement. This anchoring allows the left arm to slide upward while applying a constant grip force, thus freeing the right hand to move freely and grab onto surfaces to continue moving into the cabin. This could be influenced by the design of the cab with the right handrail on the door. Studying these behaviors across different cab designs can provide an understanding of how the design and features influence behavior and safety.
The ECRL muscle activation on both left and right sides demonstrated the highest muscle activation as a percentage of the maximum muscle acti- vation recorded for those muscles. When climbing down, the participant’s shoulders retract, causing the ECRL to have more activation than the ingress movements. However, during ingress the participant receives more visual information by looking at their hands and rails, which could contribute to why the strategy they use is completely different. During egress the partici- pant focuses on their legs and the visual cues they receive are in the form of looking down at their feet. The value of these different visual stimuli is vastly different and require further research to see how they contribute to overall movement strategies. This could lead participants to either leaning toward the cab or away from the cab and requiring different muscle activation in their upper body. It is anticipated that during egress the leg muscles could be more active than the arm muscles with even more activation during ingress due to working against gravity to climb into the cab.
There were some barriers to synchronization and correct participant iden- tification in the Ag-OMS output. The system was able to capture behaviors but only output data for one participant. The stationary researcher was occasionally captured instead of the participant completing the trial. There- fore, full data are not available for the safety behaviors of each participant throughout the trials. Better synchronization of the data streams would allow for a more direct comparison to determine if there is higher muscle activa- tion during medium-risk behavior (Figure 4, right) (not having three points of contact) vs low-risk behavior (Figure 4, left) (three points of contact). Addi- tionally, as identified in Figure 5, there are opportunities for data refinement. The ability of a participant to suddenly assume the high-risk safety behavior (facing away from the cab) for a few frames, is highly unlikely. There are opportunities to train the model to adjust or post-processing in time frames not feasible for a participant to turn around completely.
Limitations While the results of this study could be used as a foundation to help track and prevent some farm-related injuries, the study was not without its limi- tations. One limitation includes the low number of participants. With more participants, we could get a range of data that help further understand how much strength is needed to climb into a tractor, as well as understanding how participants with a variety of heights and weights can change their movement into a tractor. Another limitation includes that the exclusion of low extremity EMG in our dataset. Although our focus was on the upper body strength, the lower body also helps propel the body during ingress and can be quantified to understand overall strength. Further studies are needed to better understand the bodies overall required strength to ingress and egress into a tractor cabin.
The research presented in this paper is not yet generalizable to even a younger population for the study of ingress and egress behaviors. The pro- cessing of the data is limited and only intended to demonstrate the future research application. The Ag-OMS and EMG synchronization as well as making of events (ingress, egress, etc.) will need to be updated to improve accuracy and efficiency in the data processing. Additionally, the Ag-OMS algorithm will need some refining or some manual processing of the safety level during the tasks.
One final limitation includes that the research was conducted on one type of tractor type. There are many farm vehicles with different stair- type/entrances, which could change the amount of strength required to enter/exit them. Further investigation is needed to better understand the general requirement to be able to enter different ones.
CONCLUSION
The goal of this study was to demonstrate the feasibility of integrating EMG data and computer vision data to study biomechanics and behavior during tractor ingress and egress. We found that the use of EMG tools to cap- ture muscle activations combined with computer vision has the potential
Feasibility of Integrating Electromyography and Computer Vision 55
to capture movement patterns and behaviors that could impact safety. Fur- ther refinement of the data synchronization could allow for testing to shed light on the movement strategies that occur during tractor mounting and dismounting. In the future, this method could be used to test targeted fall pre- vention interventions and user-centered design solutions to support vehicle ingress/egress safety behaviors.
ACKNOWLEDGMENT
The authors would like to acknowledge support from the Center-States Cen- ter for Agriculture Safety and Health (CS-CASH) funded through the NIOSH Agriculture, Forestry, and Fishing Grant (U54 OH010162). We would like to thank the Nebraska Test Laboratory for allowing use of their space and equipment for indoor testing.
REFERENCES
Benos, L., Tsaopoulos, D. and Bochtis, D., 2020. A review on ergonomics in agriculture. Part I: Manual operations. Applied Sciences, 10(6), p. 1905. Bohannon, R. W., 2015. Muscle strength: clinical and prognostic value of hand-grip
dynamometry. Current Opinion in Clinical Nutrition & Metabolic Care, 18(5), pp. 465–470. Douphrate, D. I., Rosecrance, J. C., Reynolds, S. J., Stallones, L. and Gilkey, D. P.,
2009. Tractor-related injuries: an analysis of workers’ compensation data. Journal of agromedicine, 14(2), pp. 198–205. Huerta Ojeda, A., Fontecilla Diaz, B., Yeomans Cabrera, M. M. and Jerez-Mayorga, D., 2021. Grip power test: A new valid and reliable method for assessing muscle power in healthy adolescents. Plos one, 16(10), p. e0258720. Irumva, T., Mwunguzi, H., Pitla, S. K., Lowndes, B., Yoder, A. M. and Siu,
K. C., 2023. Agricultural Machinery Operator Monitoring System (Ag-OMS): A Machine Learning Approach for Real-Time Operator Safety Assessment. Journal of Agricultural Safety and Health, 29(2), pp. 85–97. Jung, T. H., Cates, B., Choi, I. K., Lee, S. H. and Choi, J. M., 2020. Multi-camerabased person recognition system for autonomous tractors. Designs, 4(4), p. 54. Saul, D., Steinmetz, G., Lehmann, W. and Schilling, A. F., 2019. Determinants for suc-
cess in climbing: A systematic review. Journal of Exercise Science & Fitness, 17(3), pp. 91–100. Stefan, R. R., Camic, C. L., Miles, G. F., Kovacs, A. J., Jagim, A. R. and Hill, C. M., 2022. Relative Contributions of Handgrip and Individual Finger Strength on Climbing Performance in a Bouldering Competition. International journal of sports physiology and performance, 17(5), pp. 768–773. Watts, P. B., Jensen, R. L., Agena, S. M., Majchrzak, J. A., Wubbels, C. S. and
Schellinger, R. A., 2004. Changes in EMG and finger force with repeated hangs from the hands in rock climbers. Medicine & Science in Sports & Exercise, 36(5), pp. S167–S168.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Safety Management and Human Factors
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-27-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional