INTRODUCTION
In the present article, we focus on the interconnectedness of humans and animals and how human-animal teams have allowed us to advance together. We also discuss basic concepts within traditional human teaming and how they can be adapted to human-autonomy teaming. Further, we describe how these human-animal teams, and the complex relationships, attachments, and challenges can serve as a model for other types of human-non-human teams; namely human-robot or human-autonomy/AI teams. The human- animal team analogy is particularly useful as a model for human-robot teams because there are numerous similarities. Both animals and robots cannot communicate information to their human counterparts via natural language. Mental models, knowledge, and capabilities of animals and robots are both limited and work differently when compared with human-human teams. Additionally, issues of trust, interdependence, and information sharing are comparable between animals and robots. Thus, human-animal team analogs
© 2023. Published by AHFE Open Access. All rights reserved. 97
98 Lum
can be leveraged to foster veridical mental models of robots that provide more accurate representations of their near-future capabilities. Some robotic analogs of human-animal teams currently exist but are often incomplete, or do not fully replicate the full capacity of existing human-animal teams. Therefore, we will focus on issues within and surrounding the current mod- els of and components associated with traditional human-human teams, how they relate to human-non-human teaming, and how we may exploit aspects of human-animal teaming to derive a more effective analogous model for human-autonomy teams.
HUMAN-NON-HUMAN TEAMING Most teaming research has focused on dynamics between human team mem- bers. However, teams with at least one or more non-human team members have become increasingly popular and relevant, especially in hazardous, or even impossible, situations for a humans presence, such as Mars exploration. In this section, we will discuss human-agent teams, with a focus on humans teaming with autonomous systems (embodied, for our purposes), which we will refer to as human-autonomy teaming (HAT) and then posit the use of other types of human-non-human teaming to create better HAT models. The future of human-autonomy teams shows a strong trend toward incorporat- ing features to allow the human to engage with their robotic counterparts in a more natural way. Norman (2004) suggests that “products and sys- tems that make you feel good are easier to deal with.” As the interfaces of robots, computers, and inanimate objects are designed to be more “intelli- gent,”humans may adapt the way they interact with, communicate, and think about such technology, treating objects more like humans as anthropomor- phism asserts itself. Humans (and many other animals) display a remarkably flexible and rich array of social competencies, demonstrating the ability to interpret, predict, and react appropriately to the behavior of others, as well as to engage others in a variety of complex social interactions. Developing computational systems that have these same sorts of social abilities is a criti- cal step in designing robots, animated characters, and other computer agents that appear intelligent, that can cooperate with people as capable partners, that are able to learn from natural human instruction, and that offer intuitive and engaging interact for humans teammates.
Human-Agent teams vary greatly in their complexity and the role that agents play in the team. Agents range from software automation to fully autonomous embodied anthropomorphic entities (Fiore, Rosen, Garfield, & Finkelstein, 2005). As computer and robotic technologies have improved, agents have taken on many more roles and are capable of more autonomous roles. Sycara and Lewis (2004) argued that these roles fall into three major categories: 1) those that support individual team members and the individual tasks they complete, 2) those acting as a team member, and 3) those support- ing the team as a whole. Within these categories there may be great variation. For instance, those agents acting as team members may be permitted the priv- ileges of human team members, or they may be limited to tasks requiring less responsibility (Groom & Nass, 2007).
Human-Animal Teaming as a Model for Human-AI-Robot Teaming 99
The team structure, goals, and dynamic dictate this difference, with HAT defined as an autonomous system that works alongside humans as a team- mate and participates in teamwork and taskwork to achieve common mission goals (McNeese et al., 2018). There has been much debate on what makes an autonomous agent a teammate rather than a tool, but most agree that the critical distinction in the continuum between automation and increasingly autonomous systems. Autonomy refers to autonomous teammates that can behave with intention, set their own goals, and can respond to situations with greater independence and even without human direction (Shively et al., 2016; USAF, 2013). In contrast, automation describes a system that will do what it is programmed to do without independent action (Demir et al., 2017; Vagia et al., 2016). Autonomous systems are non-deterministic, whereas automation, given the same inputs, will offer identical outcomes every time (Brill, Cummings, Evans, Hancock, Lyons, & Oden, 2018). Like human- human teaming, human and autonomous teammates dynamically coordinate, communicate, and collaborate in decision-making, planning, and task execu- tion. This includes monitoring, backing each other up, making suggestions, and making decisions together through continuous communication. More- over, teaming requires teammates to perform complementary, non-redundant functions towards a common goal (Brill et al., 2018). In the next sections, we will focus on the specific components that comprise a team and how they can contribute to mission success or failure.
Team Components The team dynamic is an integrated and complex web of several different factors, such as communication (both verbal and overt), interaction with technology, team roles, and team knowledge. Team cognition and team per- formance are areas of interest for numerous fields of study. It is important to assess what characteristics of individuals and teams help distinguish success- ful versus unsuccessful teams. Typically, effective teams are characterized by the amount of coordination, cooperation, and communication exhibited on both individual and team levels (Fiore et al., 2005). More recently, focus has turned to the importance of team knowledge and team situation awareness (Cooke et al., 2013).
Warner, Letsky, and Cohen (2005) have identified four stages of collabora- tive teamwork: 1) Knowledge Construction, 2) Collaborative Team Problem Solving, 3) Team Consensus, and 4) Outcome Evaluation and Revision. Each of these stages has the potential for adoption for team process involving non-human agents. Knowledge Construction consists of numerous sub-tasks, including identification of required domain-specific knowledge, selection of key personnel (team members), creation of communication pathways, development of team member mental models, and knowledge acquisition. Software agents and remotely operated robots may also be used for com- munication of key knowledge, particularly in cases where data collection is too dangerous for a human team member. Although agents are mostly used for the first stage of team collaboration, there are possible future roles reflecting the other three stages. Collaborative Team Problem Solving is the
100 Lum
stage in which most of the actual “teamwork” takes place. Here, solutions are generated and evaluated. To this end, agents may serve as providers of domain-specific knowledge. The third stage, Team Consensus, focuses on the way in which a team negotiates and presents a mutually agreed upon solu- tion (Warner, Letsky, Cowen, 2005). The final stage of team collaboration is Outcome Evaluation and Revision. This post-consensus stage requires team members to evaluate their solution and its ability to solve the problem at hand. At this stage, team members may rework solutions and generate new solutions when a previously agreed upon solution has failed.
Currently, automation may be used to generate simulations of teams solv- ing a given problem. Because this automation works faster than real time, evaluation and revision may be examined in an iterative process that would normally have taken much longer. Although this use of agents is largely as a glorified computer terminal, it may become increasingly possible for agents to participate in the type of negotiations that during consensus making and solution evaluation. As noted under the process of team consensus, many attributes of both the agent and the humans might affect the degree to which the agent’s evaluation is influential in the final evaluation of the team’s earlier decisions. Current research on HATs suggests that agents can be programmed to anticipate the needs of a team as it develops a mental model, and that these teams perform better than those without such assistance (Yen, Fan, Sun, Hanratty, & Dumer, 2006).
Sharing Unique Knowledge & Uncertainty Reduction An important and dynamic aspect of a team involved sharing of unique knowledge. Within the context of teamwork, sharing unique knowledge involves the transfer of mission- or goal-relevant information held by one teammate to another. In a human-agent team, this type of information shar- ing may be even more important due to unique differences inherent in the teammates. Sharing context-relevant information should be equally easy for humans and agents, irrespective of whether sending or receiving. In this arena, non-human agents may augment human knowledge, possibly due to an agent’s ability to assess a situation in a unique and different way than a human typically does. Specifically, an intelligent agent could be able to act as an information provider, ensure the exchange of relevant knowledge among all members, and communicate intent behind commands to allow lower and reiterate the priorities of the situation (Lenox, Lewis, Roth, Shern, Roberts, Refalaski & Jacobson, 1998). Agents, with their large processing capabilities, may be able to ingest vast datasets for near instantaneous situation assess- ment„ rapidly determining which knowledge is most relevant to share to a team and important for trust. Of course, human experts may be able to perform similar functions, though at far slower speeds.
Knowledge Interoperability, Sharing, & Transfer This process involves individual team members sharing knowledge and infor- mation with one another until all team members agree on the importance of identified problems and issues. Research has shown that, when programmed properly, groups of intelligent agents present information in a way that
Human-Animal Teaming as a Model for Human-AI-Robot Teaming 101
resembles a human, and are more likely to be trusted (Li, Montazemi, & Yuan, 2006). These processes roughly correspond to traditional cognitive views of a problem space with an initial position, a goal position, and a set of operators that may be used to solve the problem (Letsky, Warner, Fiore, Rosen, & Salas, 2007).
Knowledge Sharing and Transfer is a process by which team members pass their individual understanding to other team members. This process is mea- sured in terms of both the number of exchanges between team members and the quality of the information that is passed (Letsky, Warner, Fiore, Rosen, & Salas, 2007). Because of their ability to process and transmit a great deal of information quickly, agents may excel in this area. However, the quality of transmissions will likely depend upon whether the agent has been pro- grammed with an appropriate mental model (Gentner & Gentner, 1983) and whether the team members respond to the characteristics of the particular agent (Kang, 2007).
Visualization and Representation of Meaning Visualization and representation of meaning is a straightforward concept involving the presentation of information in a way that is more coherent and easily accessible than in raw data form. In situations with time constraints and large data sets, agents will far surpass humans in the ability to provide quality visualization of information. Agents process raw data in a systematic way with unparalleled speed, and when programmed well, are able to com- municate the information in a way that is easy for humans to understand (Chalupsky et al., 2001). Humans on the other hand, may be constrained by data analysis techniques, involved mathematical formulas, and particularly with difficulty figuring out the appropriate way to display the knowledge in a meaningful way to their teammates. If time is not an issue, then humans may use computers to derive the meaning of a problem, and visualization aids to display that information in an adequate and meaningful way.
As the team conceptualizes a problem, it is important to know which team members possess which pieces of information. Information held by all mem- bers of the team is referred to as congruent knowledge. However, in some cases, each team member may hold different “pieces of the puzzle,” known as complementary knowledge. Given their large memory capacity and inabil- ity to be easily swayed, agents may be particularly useful for keeping track of who has which pieces of information.
Intuitive Decision Making, Solution Option Generation & Appraisal Intuitive decision making involves a process by which an individual makes a decision by “collecting and analyzing information to come up with a solu- tion with no specific solution mentioned” (Letsky, Warner, Fiore, Rosen, & Salas, 2007). Letsky et al. purport this process involves little effort and is usu- ally accomplished without conscious awareness, and thus based on a “gut feeling.” Intuition may be fine-tuned through years of expertise, but some speculate that intuition is “hard-wired”(Sauter, 1999). Many problems faced by teams are vague and ill-defined, making fertile ground for expert human decision makers. However, ambiguity in problem structure runs directly
102 Lum
contrary to agents’ strengths, namely calculation and the rote execution of commands (Woolridge & Jennings, 1995). Furthermore, there seems to be human reluctance to relinquish full autonomy to non-human intelligent agents. Agencies, including DARPA, have established rules restricting agent abilities (Sierhuis et al., 2003). These restrictions govern the reasoning and behavior of agents. Even without these restrictions, computer science is still years away from creating intuitive beings (Woolridge & Jennings, 1995).
After information has been collected, the team generates a set of viable solutions. Often, many possibilities are generated so that the team has alter- natives should it decide that an initial solution is no longer viable. Further, given an agent’s ability to be impartial, it is likely that less popular, but effec- tive solutions will be retained for future review. Research has shown that when an agent fulfils a “clipboard function” to keep track of solutions on a team level, the team performs more effectively (Lenox et al., 1998).
One important distinction between an effective and ineffective team is a proper appraisal of an outcome of a task (Cannon-Bowers & Salas, 2008). This encompasses a dynamic evaluation of objectives that were met in a task, and how the outcome that was influenced by the solution options selected. Often this may be accomplished through feedback given by an external source such as an experimenter or commander. The use of feedback and its struc- ture will be discussed further in the next section. Both the human and agent teammate may be able to appraise the outcome equally well, depending on the situation. If the outcome is based on a quantitative measure of effective- ness, then an agent may be more adept to appraise the situation. However, if the outcome is based on a qualitative and more subjective assessment, then the human might have the upper hand. This, of course, depends on how the agent is programmed to assess such outcomes. As Rao and Georgeff (1995) proposed, it is necessary for a system to have all of the information about the objectives to be accomplished. If the agent hasn’t been programmed with this knowledge or does not have the capability to acquire the knowledge, then it will be unable to appraise the outcome at any level.
MODELING HUMAN-AI TEAMS FROM HUMAN-ANIMAL TEAMS Humans and animals have co-evolved for millions of years. The animal con- nection began with the exploitation and observation of animals by humans. Over time, regular social interactions were incorporated into the animal con- nection. This connection has also allowed us to utilize humans to help support and augment our skills and abilities - physically, emotionally, and cognitively. Of course, human-animal social relationships have changed over time as our connection and understanding of these animals’ capabilities has evolved, as well as through the co-evolution of our species. However, not all teams are successful, and failures often come at a high cost. Why this is important is that humans often do not ascribe the same intelligence, consciousness, or abil- ities to animals as they do to humans, and therefore, may be less apt to get frustrated when it does not perform as expected. Also, understanding the rel- ative strengths and weaknesses of each team member will ultimately facilitate team success. Although animal-inspired designs have led to improvements in
Human-Animal Teaming as a Model for Human-AI-Robot Teaming 103
robotic movement and manipulation, we maintain that inspiration provided by human-animal teaming can provide similar gains in human-robot interac- tion and HAT concepts. As most people have far more experience interacting with animals than with robots, they are generally more able to recognize limi- tations in an animal’s ability to complete a task (Phillips, Ososky, Swigert, & Jentsch, 2012). In consequence, robotic designs inspired by human-animal relationships can lead to faster acceptance while fostering more effective interactions between humans and robots, as humans tap into well-established mental models, promote better understanding of near-future robots, and thus appropriately calibrate trust in near-future robotic teammates. As a result, they frustrate us, and we quickly dismiss them even though they can be use- ful. It may instead be more useful to consider how humans interact and work with their animal counterparts. Like anthropomorphism, zoomorphism cen- ters on attributing qualities to non-sentient beings, but in this case it focuses on animal-like characteristics (Karanika & Hogg, 2020). In many contexts, human-animal teams are capable of solving complex problems well beyond the capacity of any one individual team member (Salas, Rosen, Burke, & Goodwin, 2009).
Although some aspects for HAT are inspired by similarities with human– human teaming, the teammate-like relationship of HAT may also be inspired from human-animal teaming. Human-animal teams can work on complex tasks using the complementary capabilities of humans and animals working together. In human–animal teaming, trust relies on knowing how your team- mate will respond and interpreting your teammate’s behavior (Billings et al., 2012; Phillips et al., 2016). The human can provide instructions or guidance to the animal team member and receive alerts or signs from the animal team member (e.g., working with a canine to search for narcotics) (Phillips et al., 2016). However, the human teammate should be aware that animal reactions are based on instinct and training (Billings et al., 2012).
Building mutual trust between teammates depends on communication and levels of interdependence (Phillips et al., 2016). Communication relies on conveying commands clearly and understanding the animal’s behavior (Kuhl, 2011). Maintaining effective communication entails frequent interac- tion and training. Performing independent tasks requires teammates to rely on each other. Similar to an animal teammate, an autonomous teammate can support the human teammate by extending their skills and abilities to achieve the mission goals together (Billings et al., 2012). The approach taken in this work relies more on the human-human team subset of human-autonomy teaming metaphors.
CONCLUSION
Over the course of the human-autonomy teams research, the human role evolved from control, to supervision, to collaboration (Parasuraman, Sheridan, & Wickens, 2000; Kelley and McGhee 2013; Chen et al., 2018). Systems that were manually controlled with automated components are evolving towards autonomy in the sense that these systems can be aware of their environment, react to change, and alter their abilities when necessary to
104 Lum
achieve their prescribed objectives (Russell and Norvig 2009). To be success- ful, human-autonomy teams require a collaborative relationship between the agent and its human partner predicated on mutual transparency and bidirec- tional communication (Chen et al., 2019). Initial forays into HAT research exposed problems as well as provided solutions.
While in many regards, human-autonomy teams appear to be an opti- mal solution—joining human ingenuity with computer processing capabil- ity to increase the human’s scope of effectiveness and the team’s overall efficiency—familiar human factor issues with automation surfaced—most notably human out-of-the-loop and automation bias (Parasuraman & Riley, 1997; Wright, Chen, Barnes, & Hancock, 2017). Partnership between human and agents presents problems as well as advantages: the two types of intel- ligence are not symmetrical (Kahneman 2011). This model of agents is not only constrained by its software underpinning but also by its difficulty in adjusting to novel events and its limited ability to anticipate human informa- tion requirements in a dynamic environment (McNeese, Mustifa, Cooke, & Myers, 2017).
In many contexts, teams can solve complex problems well beyond the capacity of any one individual team member (Salas, Rosen, Burke, & Goodwin, 2009). Also, understanding what different strengths and weak- nesses each team member possesses will ultimately allow that team to be more successful. Although animal-inspired designs have aided in improved robotic movement and manipulation, we maintain that design inspired by human- animal teaming can provide similar gains in robotic development, especially as it concerns improved human-autonomy team interaction. It is, therefore, advantages to further examine the interconnectedness of a working animals with their human and the ways in which this can serve as a better model for human-autonomy teaming.
ACKNOWLEDGMENT
The author would like to acknowledge the Air Force Research Laboratory and Dr. J. Christopher Brill for allowing Dr. Lum to take part in the Summer Faculty Fellowship Program. During this time, she had the opportunity to speak to researchers regarding human-human, human-animal, and human- autonomous teaming.
REFERENCES
Brill, J. C., Cummings, M. L., Evans III, A. W., Hancock, P. A., Lyons, J. B., & Oden, K. (2018). Navigating the advent of human-machine teaming. In Proceed- ings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 62, No. 1, pp. 455–459). Sage CA: Los Angeles, CA: SAGE Publications. Cacciabue, P. C. & Hollnagel, E. Simulation of Cognition: Applications. Exper-
tise and Technology: Cognition and Human-Computer Cooperation, J. M. Hoc, P. C. Cacciabue, and E. Hollnagel, eds., Lawrence Erlbaum Associates, 1995, pp. 55–73. Cannon-Bowers, J. A., & Salas, E. (2008). Reflections on shared cognition. Journal of Organizational Behavior, 22 (2), 195–202.
Human-Animal Teaming as a Model for Human-AI-Robot Teaming 105
Chalupsky, H., Gil, Y., Knoblock, C. A., Lerman, K., Oh, J., Pynadath, D. V., Russ, T. A., & Tambe, M. (2001). Electric Elves: Applying agent technology to support human organizations. American Association for Artificial Intelligence, www.aaai.org. Chen JY, Lakhmani SG, Stowers K, Selkowitz AR, Wright JL, Barnes M. Situa-
tion awareness-based agent transparency and human–autonomy teaming effec- tiveness. (2019). Theoretical Issues in Ergonomics Science, 19(3); 259–282, doi: 10.1080/1463922X.2017. 1315750. Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J. L. (2013). Interactive team cognition. Cognitive Science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009 Demir, M., McNeese, N. J., & Cooke, N. J. (2017). Team situation awareness within
the context of human-autonomy teaming. Cognitive Systems Research, 46, 3–12. https://doi.org/10.1016/j.cogsys.2016.11.003 Fiore, S. M., Rosen, M., Garfield, K., & Finkelstein, N. (2005). Developing an interdisciplinary language for human-agent teams. Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society (pp. 2149–2153). Santa Monica, CA: Human Factors and Ergonomics Society. Gentner, D. & Gentner, D. R. (1983). Flowing water or teaming crowds: Men-
tal models of electricity. In D. Gentner & A. L. Stevens (Eds). Mental Models. (pp. 99–129). Hillsdale, NJ: Lawrence Erlbaum Associates. Graham, J., Liyazheng, & Gonzalez, C. (2006). A Cognitive Approach to Game Usability and Design: Mental Model Development in Novice Real-Time Strategy Gamers. Cyberspyschology & Behavior, 9, (3), 361–366. Groom, V., & Nass, C. (2007). Can robots be teammates?: Benchmarks in human–
robot teams. Interaction studies, 8(3), 483–500. Kahneman D. Thinking, fast and slow. New York (NY): Farrar, Straus & Giroux;
2011. p. 499. Kang, M. (2007). The effects of agent activeness and cooperativeness on team decision efficiency: A computational simulation study using Team-Soar, International Journal of Human-Computer Studies, 65, 497–510. Karanika, K., & Hogg, M. K. (2020). Self–object relationships in consumers’ spon-
taneous metaphors of anthropomorphism, zoomorphism, and dehumanization. Journal of Business Research, 109, 15–25. Kelley TD, McGhee S. (2013). Combining metric episodes with semantic event concepts within the symbolic and sub-symbolic robotics intelligence control system (SS-RICS). SPIE Defense, Security, and Sensing; Anaheim, CA. p. 87560L. Lenox, T., Lewis, M., Roth, E., Shern, R., Roberts, L., Rafalski, T., & Jacobson,
J. (1998). Support of Teamwork in Human-Agent Teams. Systems, Man, and Cybernetics, 2, 1341–1346. Letsky, M., Warner, N., Fiore, S. M., Rosen, M. A., & Salas, E. (2007). Macrocognition in complex team problem solving. Proceedings of the 12 International Command And Control Research and Technology Symposium (12th ICCRTA), Newport, RI, June 2007. Washington D. C.: U. S. Department of Defense Command and Control Research Program. Li, X., Montazemi, A. R., & Yuan, Y. (2006). Agent-based buddy-finding methodol-
ogy for knowledge sharing. Information & Management, 43, 283–296. Lyons, J., Sycara, K., Lewis, M., & Capiola, A. (2021). Human-autonomy teaming: Definitions, debates, and directions. Frontiers in Psychology, 12: 589585. https://doi.org/10.3389/fpsyg.2021.589585 McNeese, N. J., Demir, M., Cooke, N. J., & Myers, C. (2018). Teaming with a syn-
thetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262–273. https://doi.org/10.1177/0018720817743223
106 Lum
Norman, D. A. (2004). Emotional design: Why we love or hate everyday things.
New York: Basic Books. Parasuraman R, Sheridan TB, Wickens CD. A model for types and levels of human interaction with automation (200). IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans; 30(3), 286–297. Phillips, E., Ososky, S., Swigert, B., & Jentsch, F. (2012). Human-animal teams as
an analog for future human-robot teams. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 1553–1557. https://doi:10.1177/ 1071181312561309 Rao, A. S. & Georgeff, M. P. (1995), “BDI Agents: From Theory to Practice”, In Proceedings of the 1st International Conference on Multi-Agent Systems (ICMAS-95), San Francisco, USA, June, 312–319. Russell S, Norvig P. Artificial intelligence: a modern approach. 3rd ed. Upper Saddle
River (NJ): Prentice Hall Press; 2009. Salas, E., Rosen, M. A., Burke, C. S., & Goodwin, G. F. (2009). The wisdom of collectives in organizations: An update of the teamwork competencies. In E. Salas, G. F. Goodwin, & C. S. Burke, Team effectiveness in complex organi- zations: Cross-disciplinary perspectives and approaches. New York: Taylor & Francis Group. Sauter, V. L. (1999). Intuitive decision-making Communications of the ACM, 42 (6),
p. 109–115. Shively, R. J., Brandt, S. L., Lachter, J., Matessa, M., Sadler, G., & Battiste, H.
(2016). Application of human-autonomy teaming (HAT) patterns to reduced crew operations (RCO). In International Conference on Engineering Psychology and Cognitive Ergonomics (pp. 244–255), Toronto, Canada, July 17–21, 2016. https://doi.org/10.1007/978-3-319-40030-3_25 Sierhuis, M., Bradshaw, J. M., Acquisti, A., van Hoof, R., Jeffers, R., & Uszok, A.
(2003). Human-Agent Teamwork and Adjustable Autonomy in Practice. Proceed- ing of the 7th International Symposium on Artificial Intelligence, Robotics and Automation in Space: i-SAIRAS 2003, NARA, Japan, May 19–23, 2003 Sycara, K., and Lewis, M. (2004). Integrating intelligent agents into human teams. In E. Salas and S. Fiore eds. Team cognition: Understanding the factors that drive pro- cess and performance (pp. 203–231). Washington, DC: American Psychological Association. USAF. (2013). Autonomy science and technology strategy. Dayton. Vagia, M., Transeth, A., & Fjerdingen, S. (2016). A literature review on the levels on
automation during the years. What are the different taxonomies that have been proposed? Applied Ergonomics, 53(2016), 190–202. https://doi.org/10.1016/j.ap ergo.2015.09.013 Warner, N., Letsky, M., & Cowen, M. (2005). Cognitive model of team collaboration: Macrocognitive focus. Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society. Santa Monica, CA: Human Factors and Ergonomics Society. Wooldridge, M and Jennings, N. R. (1995). Intelligent agents: Theory and practice.
The Knowledge Engineering Review, 10(2):115–152. Wright JL, Chen JY, Barnes MJ, Hancock PA. Agent reasoning transparency: the influence of information level on automation-induced complacency. Aberdeen Proving Ground (MD): Army Research Laboratory (US); 2017. Report No.: ARL-TR-8044. Yen, J., Fan, X., Sun, S., Hanratty, T., & Dumer, J. (2006). Agents with shared mental
models for enhancing team decision makings. Systems, 41, 634–653.