Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Physiological and Eye Tracking determinants as markers of Skill Acquisition in Manual Inspection

Cranfield University, College Rd, Wharley End, Bedford MK43 0AL, United Kingdom ↗
Publication year
Digital Object Identifier10.54941/ahfe1005169

Abstract

Skill acquisition in the manufacturing industry is a crucial aspect of optimising performance, efficiency, and safety in complex work environments. Human factors play a significant role in skill acquisition, encompassing factors such as cognitive processes, perception, decision-making, and physical interactions within the work environment. Eye tracking data has emerged as a valuable tool for studying skill acquisition (Toker et al., 2014) in the manufacturing industry, offering insights into workers' visual attention, cognitive strategies, and task performance. Currently, in manufacturing settings, skill acquisition involves the mastery of various tasks, from operating machinery and assembly line processes to quality control and troubleshooting. Effective skill acquisition is essential for ensuring consistent product quality, minimising errors, and maximizing productivity. Human factors, including attentional processes, perception, and decision-making, influence how workers acquire and apply new skills in these dynamic and often high-pressure environments (Mark et al., 2020).The aim of the current research is to understand the duration required to acquire skills through procedural learning and the transition to routine development occurs when leaned behaviour becomes habitual and routine. Furthermore, the research also aims to understand how mental and physical fatigue impact their performance with manual quality control tasks. The study aims to showcase preliminary results regarding human factors and performance variations. Participants completed an inspection task that involved an industrial component (monitor) for their serial number, visual and tactile quality under a control condition: control (no manipulation) measuring physical demand and stress levels during each monitor inspection. Physiological measures were captured using a Empatica E4 wristband (capturing electrodermal activity (EDA), heart rate, skin temperature) and eye tracking was performed with Tobii Glasses 3, as well as subjective measures of performance via NASA TLX. The results from the physiological data show that the initial 10 minutes of the task showed a positive significant correlation between EDA and NASA Performance score (Spearman rho = .675, p =.016), the second set of 10 minutes positively correlated EDA and NASA TLX temporal demands (Spearman rho = .757, p =.004), while final 10 minutes showed a positive correlation between EDA and NASA TLX physical demand (Spearman rho = .639, p =.025). Such results indicate that skill acquisition over time goes through several stages – individual's anxiety of their performance, then concerns for timely performance, and finally experiencing physical impact – as well as that EDA is good indicator of changing workload demands.The current study will present the findings of the stress and physical discomfort levels, physiological data and preliminary eye tracking data whereby results found that for the initial inspection of the first two components participants move between the instructions for the inspection and the actual inspection, however, analysis of the third component and further components revealed that attention is mainly focused on the component inspection with little fixations towards the instructions. There was also an increase in the development of skill acquisition whereby by participants displayed a decrease in error count (from 1 to 0 over the inspection of 10 monitors), instruction analysis and increase in time of task completions as the average time reduced from 6 minutes 07 seconds for the first component to 1 minute 51 seconds for the last component. (The results from the eye tracking data reveal participants’ gaze patterns during the experiment highlighting the time spent on task comprehension and completion suggesting the skill was acquired during the experimental task. In summary, the current study demonstrates the development of skills through a variety of measures showing temporal progression, mental demand and the physicality of manual handling tasks is a complex process influenced by various human factors, including attention, fatigue, and tactile knowledge. By leveraging physiological data, eye tracking and stress levels the research provides scope into how human skill acquisition is progressive through various factors that can reduce overtime to upkeep and maintain the skills of workers perform in the manufacturing industry.

INTRODUCTION

Skill acquisition within the manufacturing industry constitutes a pivotal aspect of operational efficacy and workforce performance. It encompasses the process through which individuals acquire proficiency (Nakamura et al., 2021) in executing tasks, operating machinery, and navigating complex production environments. However, this acquisition of expertise is not in isolation; it is profoundly intertwined with various human factors, operator well-being, stress management, and physical discomfort, all of which signifi- cantly influence the learning process and subsequent performance outcomes. Moreover, advancements in technology have ushered in novel methodologies

© 2024. Published by AHFE Open Access. All rights reserved. 233

for assessing skill acquisition, including eye tracking (Mark et al., 2020) and physiological measures such as heart rate and electrodermal activity (EDA). Understanding the intricate interplay between these multifaceted dimensions (Reiman et al., 2021) is essential for optimising training protocols, enhancing worker safety, and maximising productivity within the manufacturing sector.

In recent years, technological advancements have introduced novel methodologies for assessing skill acquisition within manufacturing envi- ronments. Eye tracking technology enables researchers to monitor visual attention (Hodges et al., 2021) and gaze patterns (Toker et al., 2014), pro- viding insights into cognitive processes and task performance. Physiological measures such as heart rate and electrodermal activity (EDA) offer objec- tive indicators of cognitive load, stress levels, and emotional arousal, thereby facilitating a deeper understanding of skill acquisition processes.

MENTAL STRESS Operator well-being is another critical aspect that directly impacts skill acqui- sition within manufacturing settings. The physical and mental health of workers significantly influences their ability to learn (Sgarbossa et al., 2020), adapt, and perform tasks effectively. Mental demand refers to the cognitive workload placed on an individual, often resulting in feelings of overwhelm and exhaustion. When coupled with high physical demands, such as long work hours or strenuous activity, stress levels can escalate, impacting both mental and physical well-being.

Performance may suffer as attention becomes divided, and errors increase due to cognitive overload. Effort increases as individuals strive to main- tain optimal performance despite mounting stressors. Frustration levels rise when tasks exceed skill levels or when external factors impede progress (Sari et al., 2021). Excessive or chronic stress can impede skill acquisition, impair decision-making, and compromise worker safety (Omair et al., 2019). Therefore, effective stress management strategies are essential for optimising learning outcomes and promoting operator well-being. A study conducted on identifying factors of mental stress as mental demand, physical demand, tem- poral demand, performance, effort and frustration level (Hart & Staveland, 1988). These factors have helped mitigate the negative impact of stress on skill acquisition and performance.

PHYSICAL DEMAND Physical discomfort, stemming from factors such as repetitive motions, awkward postures, and prolonged standing, poses another significant challenge within manufacturing environments. These ergonomic stressors can contribute to physical fatigue (Yung et al., 2020), musculoskeletal (Márquez Gómez, 2020) disorders, and decreased productivity. Implement- ing ergonomic interventions such as adjustable workstations, mechanised assistance, and ergonomic training programs can alleviate physical discom- fort and enhance skill acquisition.

Physiological and Eye Tracking Determinants as Markers 235

THE PRESENT PAPER The current study addresses a cooker hood manufacturer, Silverline, within the AI PRISM project known as a use case. The use case consists of the cooker hood inspection whereby operators are required to ensure the appro- priate quality of cooker hood. The current paper focus on the human factors analysis to provide the benchmark information for the skill development. Specifically, the mental and physical demand involved in the manufacturing industry which shapes and dictates worker wellbeing as well as the acqui- sition of skills through their development process during task completion. Furthermore, the research also explores performance in terms of response time and accuracy during an inspection task alongside physiological mea- sures such as EDA and eye tracking data to explore how operators’ skill development is determined. To achieve these aims the initial work conducted in this study will perform observations conducted as part of a controlled lab experiment and quantitative data collection based on the Silverline use case.

METHODOLOGY

Participants The study involved twelve participants: all participants completed an inspec- tion task on a computer monitor. Participants were aware that they were being observed, hence an overt observation took place. three females (Mean Age = 30.5) and nine males (Mean Age = 27.3).

Ethics This research was approved by the Cranfield University Research Ethics Committee, and conducted in accordance with the Cranfield Research Integrity Policy, the British Psychological Society’s Code of Human Research Ethics, and the General Data Protection Regulation 2018.

Materials

PC Monitors: Three monitors were used as part of equipment to simulate a conveyor belt motion. Participants were to assess the quality of the monitors within a simulated testing area. Monitors were connected to a power source as well as a laptop and mouse for participants to navigate through the quality control aspect.

Eye tracking glasses: Tobii Eye Tracking Glasses 3. The eye-tracking data was analysed using Tobii 3 eye-tracking analysis software, utilising the semantic gaze mapping focusing on the two Areas of Interest (AOI) and inves- tigating the average duration of fixations in these areas as well as number of visits.

Empatica E4: A wristwatch type device (Empatica E4) measured heart rate and electrodermal skin activity (EDA). Heart rate is known to increase with decreased physical comfort/increased activity and decrease with increased engagement/trust (Edwards & Kelly, 2017), while EDA increase with increased discomfort and mental workload (Kosch et al., 2019).

Self Report Data NASA TLX: The Mental Workload was measured using the NASA Task Load Index (NASA-TLX; Hart & Staveland, 1988). It measures participants’ expe- rience of a task across six different dimensions such as physical, temporal and mental demand, effort, frustration and performance. For each, the participant rates their experience across a 21-point scale.

Additional self-report measures were collected, however not analysed in the current study- Nomophobia Questionnaire (NMPQ), Information Privacy Scale (IPS), Computer Aversion, Attitudes, and Familiarity Index (CAAFI), Ten item Personality Inventory (TIPI), Technology self-efficacy (TSE), Technology readiness (TR) as well as stress and physical discomfort scale.

Stress and Physical Discomfort Scale: Derived from the general labelled magnitude scale. The labelled magnitude scale (LMS) is a hybrid scaling tech- nique using a verbally labelled line with quasi-logarithmic spacing between each label. The scale consists of a vertical line, which is marked with verbal anchors describing different intensities (e.g., “weak,” “strong”) (Jones et al., 2017).

Procedure Upon being briefed on the data collection aims, participants were guided through the informed consent and signed that they voluntarily agree to take part in the study. The NMPQ, IPS, CAAFI, TIPI, TSE and TR was completed prior to the inspection task. Participants were then assisted in putting on the eye tracking glasses, Empatica E4 wristband. The participants were then shown once how to complete the task with reference to an instructions sheet placed above the simulated testing area. The instructions were as follows:

1. Collects monitor A from conveyor belt area across to testing area. 2. Connect the HDMI cable to the monitors HDMI port. 3. Connect the power cable into the socket on the monitor, then connect the plug into the spare socket on the wall. 4. Switch on the socket at the wall. 5. Press the power button on the monitor to switch it on. 6. Once the monitor is on, using the mouse provided, head to Settings -> Display Settings -> Advance Display Settings -> Properties -> Events 7. Check the DISPLAY\ serial number against the check sheet. 8. Once serial number has been identified, press the power button to switch the monitor off. 9. Switch the socket off at the wall. 10. Unplug from the wall. 11. Disconnect the power cable and HDMI cable. 12. Perform quality Control: touch around the edges of the monitor to see if there are any identifiable scratches. If you detect any, please not it down in the list. 13. Place the monitor at the end of the conveyor belt line and begin the same process again with the next monitor(s).

Physiological and Eye Tracking Determinants as Markers 237

The condition consisted of having a simulated conveyor belt area and a testing area for participants to navigate through the experiment. Participants lifted and collected one monitor at time from the conveyor belt to the testing area, they were then required to plug the monitor into the laptop so that it was functioning correctly as well as then connecting the HDMI cable to the laptop where they could then check the serial number and code of the specific monitor provided. Participants had to navigate through the serial number check by using the mouse provided and noting down the serial number on a predefined checklist. They then completed an inspection check around the monitor by looking out for any scratches using their hands by feeling for any tactile information and noting this onto the predefined checklist before disconnecting all cables. After each inspection check they were required to indicate their physical discomfort and stress level using scales provided. They then lifted the monitor and placed it back onto the conveyor belt. This process was then repeated for a total of 10 minutes, participants were then given instructions to complete the NASA TLX before returning to their session to complete an additional 10 minutes and this process was repeated overall 3 times.

RESULTS

Response Time and Accuracy There was also an increase in the development of skill acquisition whereby by participants displayed a decrease in error count (from 1 to 0 over the inspection of 30 minutes), instruction analysis and increase in time of task completions as the average time reduced from 3 minutes 20 seconds (SD = 0.08) for the first component to 1 minute 48 seconds (SD = 0.03) for the last component. The Spearman’s rho correlation indicated a strong nega- tive effect (rho = −0.794, p = 0.00, n = 10). Taking together accuracy and response time, results indicate that as time goes by participants acquire the new skill and internalise the task instructions/acquire the manual skill needed for the task.

Mental Workload One of the main interests of the study were the changes in the mental work- load through skill acquisition. The NASA TLX scores were repeated in the intervals of 10 minutes capturing the initial response, mid-process and late response to the task on six factors of Mental Demand, Temporal Demand, Physical Demand, Effort, Performance, and Frustration.

A 6 by 3 repeated measures ANOVA (6 factors of NASA TLX vs 3 time windows) showed the main effect of Factor to be significant (F(5, 50) = 13.07, p < 0.001, η2 = 0.566). The main effect of time and the interac- tion Factor by Time were not significant (p > 0.189). Post hoc analysis of the main effect of Factor with Bonferroni correction indicated that Performance factors was higher evaluated than other factors (p < 0.023, Table X), however there were no other significant differences.

Table 1. NASA TLX factors means, standard errors, and p-value comparing performance factor with other factors.

Table available in the original PDF paper.

Factor Mean Std. Error Performance vc other factors p value Mental Demand 3.152 0.574 0.002 Physical Demand 4.636 0.646 0.024 Temporal Demand 4 0.615 0.003 Performance 8.545 0.369 Effort 3.939 0.572 0.003 Frustration 3.636 0.7 0.023

Physiological Data The second step of the analysis was concerning physiological markers of the skill acquisition. As the behavioural data indicated decreasing response time for completing the task (participants were quicker over time) the data was segmented into three sections: early task performance 0:00–10:00 minutes, mid performance 10:01–20:00 minutes and late performance 20:01–30:00 minutes. These three-time windows were used to compare the data with non-parametric Friedman repeated measures test to see whether there were differences in physiological data (EDA, HR, Skin Temperature) depending on the task duration. The only approaching significance result was observed with the heart rate (χ2 = 5.17, p = 0.08). further investigation with Wilcoxon Signed Rank test showed that participants heart rate was significantly lower during the first 10-minute window compared to the second 10-minute win- dow (Z = 2.20, p = 0.028) and at a trend level lower than in the third time window (Z = 1.65, p = 0.099), however there was no significance between first- and third-time windows (p = 0.470).

To test whether and how the physiological data and self-report task load scores relate to each other, non-parametric Spearman’s rho correlation was performed. The results from the physiological data show that the initial 10 minutes of the task showed a positive significant correlation between EDA and NASA TLX Effort score (Spearman rho = 0.675, p = 0.016), the sec- ond set of 10 minutes positively correlated EDA and NASA TLX Temporal Demands (Spearman rho = 0.757, p = 0.004), while final 10 minutes showed a positive correlation between EDA and NASA TLX Physical demand (Spear- man rho = 0.639, p = 0.025) and Mental demand (Spearman’s rho = 0.580, p = 0.048). Interestingly, neither heart rate, nor skin temperature ratings correlated with NASA TLX scores.

Eye Tracking Finally, to understand how participants are learning the new task, eye track- ing gaze mapping and fixation duration analysis was performed with Tobii eye tracking glasses 3. Two areas of interest (AOI) were defined – the written instructions displayed in front of assembly area, and the monitors them- selves. The analysis is undergoing; however, the preliminary results suggest that unsurprisingly participants had larger durations of fixations and number

Physiological and Eye Tracking Determinants as Markers 239

of visits to the monitor AOI compared to instruction AOI (Table 2). Interest- ingly, number of visits to instruction AOI was related to number of errors participants performed in the assembly. Further analysis will consider the changes in fixation duration during three-time windows as well as changes in fixation count and its relationship to the errors in the assembly.

Table 2. Eye tracking metrics as a FUNCTION of AOI and error count performed in the quality inspection.

Table available in the original PDF paper.

Participant Total duration of fixations Number of Visits Monitor Instructions Monitor Instructions Errors

1 236949 8595 989 45 2 2 258417 3216 1053 1 3 3 28882 1072 17 2 1 4 6792 0 9 0 1 5 26569 0 35 0 0 6 1803 0 9 0 2 8 60730 0 110 0 1 9 9397 0 20 0 1 10 7323 0 20 0 0 11 87768 11691 201 63 0 12 1513 0 7 0 0

CONCLUSION

The exploration of skill acquisition as a dynamic process, characterised by distinct stages and influenced by various factors such as anxiety, timely performance concerns, physical impact, Electrodermal Activity (EDA), indi- vidual timings, and eye tracking, has provided valuable insights into the complexities of learning and performance.

The result indicates the process individuals undergo in acquiring new skills, from initial apprehension about performance displayed in EDA data to the challenges of meeting deadlines under new provided instructions and eventu- ally experiencing physical strain, a comprehensive framework has emerged, offering an understanding of the underlying mechanisms driving skill devel- opment. For example, the results indicate that the time taken to complete the task significantly reduced over the course of the experiment indicating that through constant repetition of specific movements and tasks meant that the skill involving inspection was acquired. As response time decreases and accu- racy improves, it reflects the refinement and consolidation of acquired skills through practice and learning. This reduction in response time and increase in accuracy underscore the progressive mastery and efficiency gained through experience and skill development.

Integrating physiological measures like EDA into the analysis has proven to be an effective method for monitoring changing workload demands through- out the skill acquisition process. By quantifying physiological responses to task demands, EDA relates to performance, physical and mental demand,

allowing for different workload factors through the time, however, more extensive data is needed to draw conclusion and establish whether monitor- ing of EDA can reliably inform about individuals experienced task loads. For example, sourcing the original workload factor that leads to increased EDA such as mental stress or performance demands. This supports the following research that stress and performance are factors influencing the mental work- load placed on workers the impact the development of skills (Omair et al., 2019). For example, the study found that average stress among workers on the production system was individualised through their individual experience through intensity and level of stress.

Furthermore, incorporating individual timings and eye tracking data enhances the granularity of analysis, providing insights into the visual atten- tion patterns associated with skill acquisition. The eye tracking data revealed. The eye tracking data revealed that participants had significantly longer durations of fixations and made more visits to the monitor AOI than the instruction AOI. Intriguingly, a correlation emerged between the frequency of visits to the instruction AOI, and the number of errors participants made dur- ing the assembly task, suggesting a potential link between attentional focus and task performance. These findings underscore the importance of under- standing visual attention allocation in complex tasks and its implications for error prevention. Thus, supporting previous evidence by reaffirming the sig- nificance of visual attention allocation in complex tasks (Toker et al., 2014), particularly in relation to error prevention. The longer durations of fixations and increased visits to the monitor AOI compared to the instruction AOI align with prior research indicating the importance of certain visual cues or stimuli over others in task performance.

Firstly, the generalisability of findings may be limited by factors such as sample size, participant characteristics, and task specificity. Additionally, the reliance on objective measures like EDA and eye tracking may overlook the subjective experiences and individual differences that influence skill acquisi- tion processes (Valtakari et al., 2021). To build on this work and address these limitations, future research should adopt a multi-method approach that inte- grates quantitative and qualitative measures to capture the full complexity of skill acquisition. Longitudinal studies tracking individuals over extended periods could elucidate the trajectory of skill development and identify criti- cal factors that facilitate or hinder progress. Furthermore, exploring the role of contextual factors such as learning environments and social support net- works could provide valuable in terms of insights into the time in which operators effectively (Sgarbossa et al., 2020) complete their tasks, EDA and the eye tracking data. Ultimately, by embracing interdisciplinary approaches and leveraging emerging technologies, we can continue to advance our under- standing of skill acquisition and enhance our ability to support individuals in their journey towards skill mastery.

REFERENCES

Deasy Lastya Sari, Sinaga, R. I. P., Helen Storyna, Fergyanto E. Gunawan, Muhammad Asrol, & A. A. N. Perwira Redi. (2021). The Relationship between Job Stress and Employee Performance in Manufacturing Industry in Indonesia. JARES (Jour- nal of Academic Research and Sciences), 6(2), 26–38. https://doi.org/10.35457/ja res.v6i2.1640

Physiological and Eye Tracking Determinants as Markers 241

Edwards, A., & Kelly, D. (2017). Engaged or Frustrated?: Disambiguating Emotional State in Search. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR’17, 125–134. https://doi.org/10.1145/3077136.3080818 Hart, S. G., & Staveland, L. E. (1988). Development of NASA- TLX (Task Load Index): Results of Empirical and Theoretical Research. In Advances in Psychology (Vol. 52, pp. 139–183). Elsevier. https://doi.org/10.1016/S0166-4115(08)62386-9. Hodges, N. J., Wyder-Hodge, P. A., Hetherington, S., Baker, J., Besler, Z., &

Spering, M. (2021). Topical Review: Perceptual-cognitive Skills, Methods, and Skill-based Comparisons in Interceptive Sports. Optometry and Vision Science, 98(7), 681–695. https://doi.org/10.1097/OPX.0000000000001727 Kosch, T., Karolus, J., Ha, H., & Schmidt, A. (2019). Your skin resists: Exploring electrodermal activity as workload indicator during manual assembly. Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS’19, 1–5. https://doi.org/10.1145/3319499.3328230 Mark, J., Curtin, A., Kraft, A., Sands, T., Casebeer, W. D., Ziegler, M., &

Ayaz, H. (2020). Eye Tracking-Based Workload and Performance Assessment for Skill Acquisition. In H. Ayaz (Ed.), Advances in Neuroergonomics and Cog- nitive Engineering (Vol. 953, pp. 129–141). Springer International Publishing. https://doi.org/10.1007/978-3-030-20473-0_14 Márquez Gómez, M. (2020). Prediction of work-related musculoskeletal discomfort in the meat processing industry using statistical models. International Jour- nal of Industrial Ergonomics, 75, 102876. https://doi.org/10.1016/j.ergon.2019. 102876 Nakamura, J., Nagayoshi, S., & Komiya, N. (2021). Cognitive Biases as Clues to Skill

Acquisition in Manufacturing Industry. 2021 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), 1275–1278. https: //doi.org/10.1109/IEEM50564.2021.9672925 Omair, M., Ullah, M., Ganguly, B., Noor, S., Maqsood, S., & Sarkar, B. (2019).

The Quantitative Analysis of Workers’ Stress Due to Working Environment in the Production System of the Automobile Part Manufacturing Industry. Mathematics, 7(7), 627. https://doi.org/10.3390/math7070627 Reiman, A., Kaivo-oja, J., Parviainen, E., Takala, E.-P., & Lauraeus, T. (2021).

Human factors and ergonomics in manufacturing in the industry 4.0 context – A scoping review. Technology in Society, 65, 101572. https://doi.org/10.1016/j.te chsoc.2021.101572 Sgarbossa, F., Grosse, E. H., Neumann, W. P., Battini, D., & Glock, C. H. (2020).

Human factors in production and logistics systems of the future. Annual Reviews in Control, 49, 295–305. https://doi.org/10.1016/j.arcontrol.2020.04.007 Toker, D., Steichen, B., Gingerich, M., Conati, C., & Carenini, G. (2014). Towards facilitating user skill acquisition: Identifying untrained visualization users through eye tracking. Proceedings of the 19th International Conference on Intelligent User Interfaces, 105–114. https://doi.org/10.1145/2557500.2557524 Valtakari, N. V., Hooge, I. T. C., Viktorsson, C., Nyström, P., Falck-Ytter, T., &

Hessels, R. S. (2021). Eye tracking in human interaction: Possibilities and limi- tations. Behavior Research Methods, 53(4), 1592–1608. https://doi.org/10.3758/ s13428-020-01517-x Yung, M., Kolus, A., Wells, R., & Neumann, W. P. (2020). Examining the fatiguequality relationship in manufacturing. Applied Ergonomics, 82, 102919. https: //doi.org/10.1016/j.apergo.2019.102919

Cover of the proceedings volume Human Aspects of Advanced Manufacturing, Production Management and Process Control

Explore the complete digital volume

Human Aspects of Advanced Manufacturing, Production Management and Process Control

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-22-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →