Elizabeth Frost, Heath Ruff, Kyle Behymer (2022). Lessons Learned and Future Direction of a Teaming Interface for Unmanned Vehicle Tasks. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 57, pp. 86–91. https://doi.org/10.54941/ahfe1002314
Having multiple operators requires aspects critical to teaming, such as coordination and team awareness, to be considered during system design. A Task Manager interface was developed that supports shared awareness across team members by summarizing the relative priority, recency, assignment, and completion status of mission tasks. While the original design provided information essential to the operator, evaluation results indicated that critical information needed to be more accessible. Primarily, important unmanned vehicle (UV) task details should be available at the higher level without the need to “drill down” into the task. Evaluation results informed a Task Manager redesign that does not remove any functionality but altered how information is represented. The goal of these modifications is to improve awareness for the UV operators and support more efficient teaming between operator/autonomy teammates. This new design will be evaluated in future research, and those results will then inform future designs using an iterative design and evaluation process.
Disclaimer: The views expressed are those of the authors and do not reflect the official guidance or position of the United States Government, the Department of Defense or of the United States Air Force.
INTRODUCTION
A control station prototype referred to as “IMPACT” (Intelligent Multi-UxV Planner with Adaptive Collaborative/Control Technologies) was designed to flexibly team a single human operator with autonomous decision aids while performing a base defense mission (Draper et al., 2017; Draper et al., 2018; Figure 1). IMPACT enables a single operator to control multiple heteroge- neous (air, ground, and sea surface) unmanned vehicles (UVs). To support efficient tasking of the UVs, IMPACT uses a “play-calling” method (Miller, Goldman, Funk, Wu, and Pate, 2004) that enables the operator to develop and execute a plan quickly by leveraging the autonomous aids. For exam- ple, when an IMPACT operator calls a play to achieve air surveillance on a building, an intelligent agent recommends a UV (based on estimated time en route, fuel use, etc.), a cooperative control algorithm provides an optimal
Lessons Learned and Future Direction of a Teaming Interface 87
Figure 1: IMPACT control station prototype.
route to get to the building (taking into account no-fly zones, etc.), and an autonomics framework monitors the play’s ongoing status (e.g., alerting the operator if the UV won’t arrive on time).
An IMPACT operator’s primary responsibility is to complete tasks related to base defense mission events. Plays are often called to complete tasks. A suite of play-based interfaces supports calling plays (Calhoun, Ruff, Behy- mer, and Mersch, 2017). However, the operator also needs to manage and maintain awareness of their higher-level tasks.
As autonomous capabilities continue to improve and mature, future ope- rations will likely include teaming between multiple operators (human and autonomous), each with their own set of UVs and tasks. When increasing the number of operators collaborating on a mission, coordination and com- munication for task completion will be vital. The interfaces need to support awareness of the operator’s tasks and information regarding their teammate’s status while mitigating data overload (Wilson, Salas, Priest, & Andrews, 2007).
COLLABORATION INTERFACE TO SUPPORT TEAMING BETWEEN UV OPERATORS The Task Manager interface was developed to support shared awareness across team members by summarizing the relative priority, recency, assign- ment, and completion status of mission tasks (Figure 2). A brief overview of the original Task Manager design is provided here (see Frost et al., 2019, for a complete description).
The main window (on the left) contains a list of the operator’s tasks. Each task has a corresponding mission-coded icon that has been determined in previous research to be intuitive and discriminable (Bartik et al., 2017). Each of the task icons is presented within a circle. The circle’s line coding designates if that task needs to be completed by the operator (solid line), a teammate (dotted line), or both (dashed-dotted line). The tasks are placed into rows based on their priority, with the top row being the highest priority.
Figure 2: Original task manager design.
The selection of a task in the main window brought up a pop-up window that contained the task details. This included: the chat message that triggered the task, the time since the task was created, the list of subtasks, the com- pletion status of the subtask, and the ability to assign/trade a subtask with a teammate.
Review and Feedback The Task Manager was evaluated as part of a study evaluating teammates (12 participants worked with a human teammate and 12 participants worked with an autonomous teammate) working together in various team structures. Each teammate had their own set of UVs to task to complete a base defense mission (Frost, 2019). The Task Manager was a critical aspect of this research because it was essential for coordination between the teammates. Participants used a 5-point Likert scale (with five being the most positive) to rate the Task Manager on several parameters. They agreed that the Task Manager’s overall look and feel was effective (M = 4.38, SD = 0.49), icons were intuitive (M = 4.04, SD = 0.81), tasks were appropriately prioritized (M = 4.46, SD = 0.66), and it was clear which steps they needed to take (M = 4.42, SD = 0.65).
While the necessary information was available, feedback revealed that par- ticipants wanted to access information more efficiently. Specifically, three improvements were identified:
Availability of important task details at a higher level (i.e., the main window) without the need to “drill down” into the task. Especially information regarding the task completion status.
More salient connection between tasks and plays. Improve awareness of which plays are connected to which task.
Task assignment more quickly identifiable. Desire to see their task wor- kload and the task distribution at a ‘quick glance’ without checking the circle around each icon.
Lessons Learned and Future Direction of a Teaming Interface 89
Figure 3: Enhanced task manager design.
ENHANCED TASK MANAGER An Enhanced Task Manager (Figure 3) was developed to address each of the requested improvements listed above. All the original functionality was maintained; however, the representation of the information was altered. Each task is still represented by a mission-coded icon and placed into the rows based on priority. Also, the pop-up window is still accessible by selecting a task icon.
Additional task information is shown at a higher level by representing the number and completion status of subtasks in the main task manager window (instead of bringing up the task detail window to access subtask information). A bar was added underneath each task icon, and it is divided into sections based on the number of subtasks (e.g., a task with two subtasks has a bar divided into two sections). A bar section is shaded when the subtask is com- pleted (e.g., a task with the first and third subtasks has the first and third sections of the bar shaded.) Additionally, each section can be clicked to bring up the details for that subtask without opening the pop-up window.
Each task now has a unique color to increase the connection between tasks and the plays called in response. These plays will then have the same color as the task. Plays in the IMPACT system already used colors to differenti- ate them and help connect play information across the displays. Assigning color based on the task will provide the same awareness across displays and connect multiple plays in response to the same task.
Lastly, to increase the saliency of task assignment, the circles around each icon were removed, and tasks are now placed into columns based on assi- gnment. Although there was mixed feedback regarding whether or not the teammate’s tasks should be shown in the Task Manager, it was unanimously agreed that it was valuable to be able to access information about their teammate’s tasks. It was decided that although this is important information to have accessible, having the tasks mixed in the rows causes unnecessary confusion. Therefore, an ‘expanded’ view was designed (Figure 4).
Figure 4: Expanded view of the enhanced task manager.
By default, the Task Manager’s main window only shows the ‘mine’ and ‘shared’ columns. The operator selects the ‘person’ icon in the upper right- hand corner to show/hide columns that contain their teammates’ tasks. The previous approach of a dotted circle only indicated that it belonged to a “teammate.” Utilizing columns instead enables scalability to multiple team- mates and allows the operator to differentiate between individual teammates’ tasks.
NEXT STEPS/CONCLUSION The modifications made to create the Enhanced Task Manager aim to incre- ase the accessibility of task information (particularly task status), improve the connection between tasks and plays, and provide a clearer view of task assignment and division across teammates. Future research will evaluate this redesign; one possible study design is comparing participant performance with the previous version (Figure 2) to participant performance with the Enh- anced Task Manager. We hypothesize that participants will be more mission effective and prefer the Enhanced Task Manager as compared to the original. Additionally, we plan on investigating the utility of the expanded Task Mana- ger view (Figure 4) when used during multiple-operator multiple-autonomy teaming missions. Potential variables to be examined include the number of teammates and mission complexity. We also plan to explore using the Task Manager to facilitate the transfer of assets between teammates (e.g., Opera- tor 1 requests to “borrow” Operator 2’s UV for a specific period of time or a specific task).
Overall, the Task Manager can be a vital tool for play-based multi-UV management by expediting retrieval of necessary actions and providing a mechanism for sharing the workload across team members. Thus, increa- sing mission awareness for the UV operators and supporting more efficient teaming between multiple operators.
Lessons Learned and Future Direction of a Teaming Interface 91
ACKNOWLEDGMENT
The U.S. Air Force Research Laboratory supported this effort. Special thanks to Sarah Hill at Mile Two, LLC for creating a wireframe of the Enhanced Task Manager.
REFERENCES
Bartik, J., Ruff, H., Behymer, K., Frost, E., Calhoun, G., Spriggs, S., & Hammack, T.
(2017) Mission-coded map icon decision aids for play-based multi-unmanned veh- icle autonomy delegation. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61(1), 237–241. Los Angeles, CA: SAGE Publications. Calhoun, G., Ruff, H., Behymer, K., & Mersch, E. (2017). Operator-autonomy teaming interfaces to support multi-unmanned vehicle missions. Advances in Human Factors in Robots and Unmanned Systems, 113–12. Draper, M., Calhoun, G., Hansen, M., Douglass, S., Spriggs, S., Patzek, M., ... &
Frost, E. (2017). Intelligent multi-unmanned vehicle planner with adaptive col- laborative/control technologies (IMPACT). In 19th International Symposium on Aviation Psychology (p. 226). Draper, M., Rowe, A., Douglass, S., Calhoun, G., Spriggs, S., Kingston, D., ...
& Frost, E. (2018). Realizing autonomy via intelligent hybrid control: Adapta- ble autonomy for achieving UxV RSTA team decision superiority (also known as Intelligent Multi-UxV Planner with Adaptive Collaborative/Control Techno- logies (IMPACT)). (AFRL-RH-WP-TR-2018-0005). Wright-Patterson Air Force Base U.S.: Air Force Research Laboratory. Frost, E. (2019). Creating a well-situated human-autonomy team: The effects of team structure (Doctoral dissertation, Wright State University). Frost, E., Calhoun, G., Ruff, H., Bartik, J., Behymer, K., Spriggs, S., & Bucha-
nan, A. (2019). Collaboration interface supporting human-autonomy teaming for unmanned vehicle management. Proceedings of the International Symposium for Aviation Psychology. Dayton, OH. Miller, C., Goldman, R., Funk, H., Wu, P., & Pate, B. (2004). A playbook approach to variable autonomy controls: Application for control of multiple, heterogeneous unmanned air vehicles. Proceedings of FORUM 60, the annual meeting of the American Helicopter Society, June 7–10, Baltimore, MD. Wilson, K. A., Salas, E., Priest, H. A., & Andrews, D. (2007). Errors in the heat of
battle: Taking a closer look at shared cognition breakdowns through teamwork. Human Factors, 49(2), 243–256.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-33-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional