Yonghao Huang, Omar Alqatami, Wei Zhang (2024). Combinatorial Effects of Unmanned Vehicles on Operator’s Mental Workload and Performance for Searching. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 138, pp. 83–93. https://doi.org/10.54941/ahfe1005011
With the advancement of artificial intelligence technology, unmanned vehicle (UV) systems, including unmanned aerial vehicle (UAV) and unmanned ground vehicle (UGV), have been appearing in many scenarios with wide-ranging applications. It is a crucial factor for enhancing human-computer collaboration efficiency to analyze the impact of unmanned vehicle system combinations on operators. The research in this paper presents the effects of different combinations of UAVs and UGVs (1UAV + 1UGV, 1UAV + 2UGVs, 2UAVs + 1UGV, 2UAVs + 2UGVs) on searching performance and operator’s mental workload for accomplishing search tasks. Completion times for tasks and subjective data with operators (N=16) were collected by using Psychopy and questionnaires, respectively. The results in this research indicate that binary growth of controllable unmanned vehicles doesn’t improve the UV utilization rate, but costing longer completion time and increasing operator’s mental workload. Since an excessive number of unmanned vehicles could have a negative impact on task performance, the insights in the paper are helpful for the design of unmanned vehicle systems and the research of human-computer collaboration.
Unmanned vehicle systems have many increasing applications in logistics, emergency rescue, and military, etc. such that it has garnered attention towards the possibility of collaboration between unmanned vehicle system (UAVs and UGVs) and humans (Long et al., 2018). The complementary advantages among two types of unmanned vehicles can effectively enhance the capability of unmanned systems in accomplishing complex tasks (Asadi et al., 2020). However, constrained by technological limitations and ethical issues in specific scenarios, unmanned vehicle systems will remain semi- autonomous for a long time, requiring human command and monitoring (Ghamry et al., 2016). An abundance of redundant information usually is received during interaction between unmanned vehicle system and oper- ator together with some suboptimal cooperations, which leads to certain decreased performance and efficiency (Dadashi et al., 2013). Therefore, it is important and crucial to investigate behavioral patterns of human
interactions with unmanned vehicle systems for determining appropriate composition of human-machine team, ensuring overall performance, and prevent safety incidents.
Mental workload is defined as the information processing capacity or resources consumed by an operator to meet system demands (Eggemeier et al., 2020), which is commonly assessed by using a subjective, performance, and physiological measurement (Cain, 2007). The collaborative performance is strongly related to the level of operator’s mental workload (Parasuraman et al., 2008). It is well known that mental underloading or overloading may lead to some serious accidents (Hobbs & Lyall, 2016). The factors that affect mental workload in the interaction between humans and unmanned vehicle systems usually are categorized into four groups: environment, task, equip- ment, and operator (Hooey et al., 2018). Task complexity and the number of unmanned vehicles controlled have a significant impact on the opera- tor’s mental workload, thereby making an influence on the control process and task performance (Baber et al., 2011; Bommer & Fendley, 2015; Li et al., 2022). Current researches on human-unmanned system collaboration mainly focus on the allocation of tasks within human-robot teams, modes of human-robot interaction, design of interaction interfaces, and assessment of collaboration efficiency (Baber et al., 2011; Calhoun et al., 2018; Cummings et al., 2014; Schmitt & Schulte, 2015). It is not entirely clear about effects of task complexity and vehicle combinations on an operator in systems. Some studies indicate that an operator can effectively control 4 to 5 unmanned vehi- cles while a performance decrease occurs for more than 5 vehicles (Cummings et al., 2007). In addition, the level of autonomy of unmanned vehicle systems and control architecture also affect task performance and operator’s mental workload, subsequently influencing the maximum number of controllable unmanned vehicles (Cummings et al., 2014).
In this study, search tasks have been developed from the viewpoint of two task complexity levels involving UAVs and UGVs combinations, and their influence on both operator’s mental workload and searching performance has been investigated through simulation. on the. The research results carry significant implications for enhancing the operational efficiency, safety, and reduction of resource wastage in unmanned vehicle systems. They also imply some constructive suggestions for future autonomous interaction system design and the promotion of unmanned vehicle systems.
METHOD
Participants Sixteen students in total from Tsinghua University (Age: M = 25.44, SD = 2.24), including 8 males and 8 females, participated in this experiment. All participants took part in the experiment for the first time, had normal or corrected-to-normal vision, and no color blindness or color vision defi- ciencies. During the experiment, participants were informed that they could withdraw at any time if they felt uncomfortable or unwilling to continue. The entire experiment lasted approximately 45 minutes, and upon completion, each participant received a remuneration of 60 RMB.
Combinatorial Effects of Unmanned Vehicles 85
Experimental Design A two-factor repeated measures design (2×4) was employed to investigate the effects of task complexity level and unmanned vehicle combinations on oper- ator’s task performance and mental workload. Task complexity was divided into two levels: 4-targets search task (low-complexity) or 8-targets search task (high-complexity), which indicates the number of targets searched in each experimental scenario by each participant. Unmanned vehicle combina- tions were categorized into four levels: 1UAV + 1UGV, 1UAV + 2UGVs, 2UAVs + 1UGV, and 2UAVs + 2UGVs, corresponding the number of Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) that participants could operate in each scenario.
The search task in each scenario of research is simulated and requires par- ticipants to control UAVs and UGVs via a mouse and complete the search, identification, and report of all targets in the shortest possible time. PsychoPy software was utilized for the construction of experimental scenarios and the collection of performance data. The experimental platform, as depicted in Figure 1, consists of three main panels: the top half is an image panel dis- playing images captured by the unmanned vehicles; the bottom left corner features a map panel showcasing a floor plan of the rooms and the current positions of the unmanned vehicles; the bottom right corner is a control panel. Participants can issue movement commands to multiple unmanned vehicles simultaneously by clicking on the corresponding numbers on the scale, mov- ing the vehicles to the positions denoted by the numbers on the map. During vehicle movement, the image panel indicates “Position Moving”. The display time for new images equals to the distance between two locations divided by the speed of the unmanned vehicles (UAV: 12 m/s; UGV: 3 m/s).
Figure 1: Unmanned vehicle system control panel.
The experiment included eight scenarios, equally divided into two differ- ent task complexity level (4-targets or 8-targets). Each scenario comprised a varied layouts house consisting of six rooms. Each target to be searched was a cube with sides measuring 1 meter, available in four colors (red, yellow, blue,
green). The top or side of each cube featured a capital English letter with an underscore. The positioning of the targets, their colors, and the orientation of the letters were randomly distributed across the scenarios.
Dependent Measurement The dependent variables in the experiment included task performance and subjective workload perception. Task performance was evaluated by the recognition accuracy, completion time, total number of UV movements, and the UV utilization rate. Subjective workload was assessed by using the NASA- TLX, a widely employed mental workload rating scale with six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration level.
Procedure Firstly, participants were asked to sign an informed consent form and complete a questionnaire including their basic demographic information. Secondly, they were told the experimental tasks and the operation of the experimental platform, and provided with four practice scenarios under four different unmanned vehicle combinations. Thirdly, participants were required to sequentially complete two sets of search tasks at different com- plexity levels, each set comprising four scenarios using various unmanned vehicle combinations. A Latin square experimental design was utilized for two sets of tasks and four scenarios to eliminate learning effects. In each sce- nario, participants were asked to make a mark at the corresponding positions on a paper blank map with the color of each cube and the letter on its surface. Finally, after completing each search scenario, participants were required to complete a NASA-TLX rating scale to measure their mental workload during the previous experimental scenario.
Data Analysis Since each participant engaged in experiments across eight different scenar- ios, the method based on repeated measures analysis of variance (ANOVA) could be utilized to evaluate the differences of searching performance among four unmanned vehicle combinations. For results with significant differences, the Bonferroni correction was employed to calculate t-statistics and p-values for each group in post-hoc tests. All analysis processes were conducted using the statistical software SPSS 27, with the significance level set at 0.05.
RESULTS
Table 1 illustrates the task performance of participants across different unmanned vehicle combinations under two task complexity levels. The results of the recognition accuracy indicated that there were no significant differences between the combinations for both 4-target (F (3,45) =0.508, p = 0.603, η2=0.033) and 8-target search tasks ((F (3,45) =1.231, p = 0.309, η2=0.076), with the accuracy rate of search exceeding 90% in all scenar- ios. However, the completion time for tasks increased with the number of unmanned vehicles in both 4-target (1UAV + 1UGV: M = 86.77, SD = 39.52;
Combinatorial Effects of Unmanned Vehicles 87
1UAV + 2UGVs: M = 90.09, SD = 28.84; 2UAVs + 1UGV: M = 94.14, SD = 25.93; 2UAVs + 2UGVs: M = 99.69, SD = 32.52) and 8-target search tasks (1UAV + 1UGV: M = 175.43, SD = 55.28; 1UAV + 2UGVs: M = 190.17, SD = 55.95; 2UAVs + 1UGV: M = 195.80, SD = 64.21; 2UAVs + 2UGVs: M = 209.84, SD = 65.94). Behavioral analysis of opera- tors indicated that there was a significant difference in the total number of UV movements across the four unmanned vehicle combinations levels during completing 4-target search tasks ((F (3,45) =4.091, p = 0.019, η2=0.214). Post-hoc analysis in Table 3 revealed that the 1UAV + 1UGV combina- tion had fewer search movements with respect to the 2UAVs + 2UGVs combination (t (15) = −2.78, p = 0.084, d= −0.851), but there was no statistically significant difference with respect to other combinations. Accord- ing to results in Table 1, it was observed for UV utilization rate to have a significant difference across combinations for both 4-target tasks ((F (3,45) =7.387, p = 0.002, η2=0.330) and 8-target tasks ((F (3,45) =11.544, p < 0.001, η2=0.435). Post-hoc analysis results (Table 4) demonstrated that the UV utilization rate of the 1UAV + 1UGV combination (4 Objects Task: M = 0.83, SD = 0.20; 8 Objects Task: M = 0.76, SD = 0.14) was higher than 1UAV + 2UGVs (4 Objects Task: M = 0.64, SD = 0.24, t (15) = 3.92, p = 0.008, d = 0.940; 8 Objects Task: M = 0.62, SD = 0.16, t(15) = 4.87, p = 0.001, d = 0.913), 2UAVs + 1UGV (4 Objects Task: M = 0.65, SD = 0.21, t(15) = 3.66, p = 0.014, d = 0.819; 8 Objects Task: M = 0.64, SD = 0.18, t(15) = 3.81, p = 0.010, d = 0.678), and 2UAVs + 2UGVs (4 Objects Task: M = 0.62, SD = 0.23, t(15) = 5.69, p < 0.001, d = 0.891; 8 Objects Task: M = 0.54, SD = 0.20, t(15) = 5.19, p < 0.001, d = 1.115) for each complexity levels.
Table 1. Task performance at different task complexity levels.
The analysis of the NASA-TLX scale revealed that there were signifi- cant differences in mental demands among combinations during the 8-target search tasks ((F (3,45) =7.174, p = 0.001, η2=0.324) (Table 2). In addi- tion, the post-hoc analysis in Table 5 indicated that the mental demand for the 1UAV + 1UGV combination (M = 62.50, SD = 16.33) was marginally or significantly lower than 1UAV + 2UGVs (M = 69.69, SD = 19.96, t(15) = −2.93, p = 0.063, d = −0.440), 2UAVs + 1UGV (M = 70.31, SD = 17.56, t(15) = −3.65, p = 0.014, d = −0.478), and 2UAVs + 2UGVs (M = 75.50, SD = 15.28, t(15) = −4.25, p = 0.004, d = −0.796) combinations. No other significant differences were found in the research.
Table 2. Mental demand scores at different task complexity levels.
DISCUSSIONS Carrying out the search tasks, participants failed to achieve 100% recogni- tion accuracy in most scenarios except 2UAVs + 2UGVs combination for 4-target search tasks. This phenomenon suggests that recognition accuracy is influenced by both task complexity and the combination of unmanned vehi- cles. Under low-complexity scenarios, where the number of targets searched is four, multiple unmanned vehicles can provide participants multidimen- sional images of a single room, and assist them in determining the position, color, and letters in each cube accurately. However, in high-complexity sce- narios, some specific rooms may contain up to four cubes simultaneously. An excess of unmanned vehicles increases the mental workload of participants in processing spatial information (Cummings et al., 2007), which leads to a decrease in accuracy for the 2UAVs + 2UGVs combination during 8-target search tasks.
For the 4-target and 8-target search tasks, their completion time increased to varying degrees with the increase of the number of units in the vehi- cle combination. Under high-complexity tasks, the difference in comple- tion time between the 1UAV + 1UGV and 2UAVs + 2UGVs combi- nations (1M = 34.41s) was greater than that in low-complexity tasks (1M = 12.92s). More frequent shifts of attention from UAVs to UGVs cre- ated a “disruption” effect and led to a phenomenon of change blindness, which caused participants taking more time to regain situational aware- ness (Parasuraman et al., 2009). Therefore, the efficiency of task execution is influenced by an increased number of controllable unmanned vehicles, and this effect is amplified as task complexity increases. Additionally, the completion time is also affected by the composition of unmanned vehicles. Compared to the 1UAV + 1UGV, an increase in the number of UAVs had a greater impact on completion time (4-target search tasks: 1M = 7.37s; 8- target search tasks: 1M = 20.37s) than an increase in the number of UGVs (4-target search tasks: 1M = 3.32s; 8-target search tasks: 1M = 14.74s). Since UAVs and UGVs operate for image capture by using distinct frames of reference (Chen, 2010), the higher dimensionality of information provided by UAVs might require participants to spend more time processing than that for UGVs.
Combinatorial Effects of Unmanned Vehicles 91
The participants’ task execution strategies were also impacted by the num- ber of controllable unmanned vehicles. In low-complexity 4-target search tasks, an increase in the number of unmanned vehicles led to more inef- fective movements and a lower utilization rate. The excess of information required participants to spend more time on information integration, ulti- mately increasing the task completion time. Subjective ratings of mental workload suggest that, in high-complexity 8-target search tasks, participants have used more cognitive resources to process the information within a sin- gle image. The addition of unmanned vehicles did not improve the efficiency of existing information processing but instead introduced more information and led to increase mental workloads. Due to the rise in mental demand and the switch cost, participants reduced the number of units they practically con- trolled at one time. Suppose that the total number of movements remained unchanged, then the completion time increased and corresponded to a lower performance for searching task.
CONCLUSION
Advancements in artificial intelligence technology have introduced more interactions between humans and unmanned vehicle systems. However, based on literature reviews, it is not clear about the effect of UAV and UGV combinations on the mental workload and search task performance. Experi- ments have been designed in this paper with two complexity levels (search tasks with 4 targets and 8 targets). Data on mental workload and task performance have also been collected from 16 participants who completed search tasks using four combinations of unmanned vehicles: 1UAV + 1UGV, 1UAV + 2UGVs, 2UAVs + 1UGV, and 2UAVs + 2UGVs. The experimental results indicate that a moderate increase in the number of unmanned vehi- cles does not significantly affect the recognition accuracy, but reduces the utilization rate of unmanned vehicles. This phenomenon leads to prolonged task completion times, especially in scenarios of searching tasks with higher complexity. Due to the information processing limitations, an increase num- ber of unmanned vehicles may lead to more frequent shifts in attention. The phenomenon of change blindness occurs during the transition between dif- ferent types of unmanned vehicles, which causes the “disruption” effect of attention and ultimately leads to an increase in the completion time of search- ing tasks. The results in this study have significant practical implications for optimizing the interface design of unmanned vehicle systems and enhanc- ing the task performance of operators. They are also helpful for exploring a better human-computer interaction technology under the era of artificial intelligence.
REFERENCES
Asadi, K., Kalkunte Suresh, A., Ender, A., Gotad, S., Maniyar, S., Anand, S., Noghabaei, M., Han, K., Lobaton, E., & Wu, T. (2020). An integrated UGV- UAV system for construction site data collection. Automation in Construction, 112, 103068. https://doi.org/10.1016/j.autcon.2019.103068
Baber, C., Morin, C., Parekh, M., Cahillane, M., & Houghton, R. J. (2011). Multimodal control of sensors on multiple simulated unmanned vehicles. Ergonomics, 54(9), 792–805. https://doi.org/10.1080/00140139.2011.597516 Bommer, S. C., & Fendley, M. (2015). Assessing the Effects of Multimodal Com-
munications on Mental Workload During the Supervision of Multiple Unmanned Aerial Vehicles. International Journal of Unmanned Systems Engineering., 3(1), 38–50. https://doi.org/10.14323/ijuseng.2015.4 Cain, B. (2007). A review of the mental workload literature. DTIC Document.
https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=7367184a149 e20107188cbaaf034fcd3c3b241ea Calhoun, G. L., Ruff, H. A., Behymer, K. J., & Frost, E. M. (2018). Human-autonomy teaming interface design considerations for multi-unmanned vehicle control. The- oretical Issues in Ergonomics Science, 19(3), 321–352. https://doi.org/10.1080/ 1463922X.2017.1315751 Chen, J. Y. C. (2010). UAV-guided navigation for ground robot tele-operation in a
military reconnaissance environment. Ergonomics, 53(8), 940–950. https://doi.or g/10.1080/00140139.2010.500404 Cummings, M. L., Bertucelli, L. F., Macbeth, J., & Surana, A. (2014). Task Versus Vehicle-Based Control Paradigms in Multiple Unmanned Vehicle Supervision by a Single Operator. IEEE Transactions on Human-Machine Systems, 44(3), 353–361. https://doi.org/10.1109/THMS.2014.2304962 Cummings, M. L., Bruni, S., Mercier, S., & Mitchell, P. J. (2007). Automation
Architecture for Single Operator, Multiple UAV Command and Control. https: //dspace.mit.edu/handle/1721.1/90285 Dadashi, N., Stedmon, A. W., & Pridmore, T. P. (2013). Semi-automated CCTV surveillance: The effects of system confidence, system accuracy and task com- plexity on operator vigilance, reliance and workload. Applied Ergonomics, 44(5), 730–738. https://doi.org/10.1016/j.apergo.2012.04.012 Eggemeier, F. T., Wilson, G. F., Kramer, A. F., & Damos, D. L. (2020). Work-
load assessment in multi-task environments. In Multiple task performance (pp. 207–216). CRC Press. https://www.taylorfrancis.com/chapters/edit/10.1201/ 9781003069447-12/workload-assessment-multi-task-environments-thomas-egg emeier-glenn-wilson-arthur-kramer-diane-damos Ghamry, K. A., Kamel, M. A., & Zhang, Y. (2016). Cooperative forest monitoring and fire detection using a team of UAVs-UGVs. 2016 International Conference on Unmanned Aircraft Systems (ICUAS), 1206–1211. https://doi.org/10.1109/ICUA S.2016.7502585 Hobbs, A., & Lyall, B. (2016). Human Factors Guidelines for Unmanned Aircraft
Systems. Ergonomics in Design: The Quarterly of Human Factors Applications, 24(3), 23–28. https://doi.org/10.1177/1064804616640632 Hooey, B. L., Kaber, D. B., Adams, J. A., Fong, T. W., & Gore, B. F. (2018). The Underpinnings of Workload in Unmanned Vehicle Systems. IEEE Transactions on Human-Machine Systems, 48(5), 452–467. https://doi.org/10.1109/THMS.2017. 2759758 Li, K. W., Lu, Y., & Li, N. (2022). Subjective and objective assessments of mental
workload for UAV operations. Work, 72(1), 291–301. https://doi.org/10.3233/ WOR-205318 Long, P., Fan, T., Liao, X., Liu, W., Zhang, H., & Pan, J. (2018). Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learn- ing. 2018 IEEE International Conference on Robotics and Automation (ICRA), 6252–6259. https://doi.org/10.1109/ICRA.2018.8461113
Combinatorial Effects of Unmanned Vehicles 93
Parasuraman, R., Cosenzo, K. A., & De Visser, E. (2009). Adaptive Automation for Human Supervision of Multiple Uninhabited Vehicles: Effects on Change Detec- tion, Situation Awareness, and Mental Workload. Military Psychology, 21(2), 270–297. https://doi.org/10.1080/08995600902768800 Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2008). Situation Awareness,
Mental Workload, and Trust in Automation: Viable, Empirically Supported Cog- nitive Engineering Constructs. Journal of Cognitive Engineering and Decision Making, 2(2), 140–160. https://doi.org/10.1518/155534308X284417 Schmitt, F., & Schulte, A. (2015). Mixed-Initiative Mission Planning Using Planning Strategy Models in Military Manned-Unmanned Teaming Missions. 2015 IEEE International Conference on Systems, Man, and Cybernetics, 1391–1396. https: //doi.org/10.1109/SMC.2015.247
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-14-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional