Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Cognitive Engineering in Training: Monitoring and Pilot-Automation Coordination in Complex Environments

1 NASA Ames Research Center, Moffett Field, CA 94035, USA ↗
2 San José State University Foundation, San Jose, CA 95112, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1003006

Abstract

This paper reports on a Cognitive Engineering approach to identify untaught skills and knowledge and to support design of learning tools. We investigate flightpath (FP) monitoring and the interaction of pilot and automation implicated in monitoring. We interviewed experienced pilots to understand the knowledge and skills underlying effective monitoring, and we developed an example learning environment to improve these skills. We explore how design of pilot training and learning, like the design of interfaces and of the underlying automation, benefits from cognitive engineering methods and perspective. In aviation, monitoring and managing FP are critical activities, affected by automation, control actions by the pilot, and by external factors, including weather and Air Traffic Control (ATC). Effective piloting depends on strategies for noticing, understanding, and anticipating these influences to monitor and manage FP. Although flightdeck automation is intended to aid pilot understanding and prediction, the Fight Management Systems (FMS) can also mislead the pilot, particularly when depending on old or incomplete information. Understanding such vulnerabilities is an important part of pilot-automation coordination. We identified skills and knowledge learned from experience but not from training; for less experienced pilots these are likely knowledge gaps and potential targets for learning. Using learner-centered principles, we developed a learning environment designed to help pilots build skills and knowledge for proactive FP monitoring. We consider how a broad cognitive engineering approach might inform both the "what" and "how" of learning in dynamic work domains.

INTRODUCTION

Our Problem: Understand Automation-Intensive FP Monitoring Piloting an airliner is fundamentally cognitive work. A crew of a pilot moni- toring (PM) and a pilot flying (PF) uses automation, namely, the autoflight system, to aid monitoring and managing the FP. The autoflight system inclu- des the autopilot, the flight director, an autothrottle, and a flight management system. When engaged, the autopilot flies the trajectory generated by the FMS using targets from the flight plan or entered by the pilots. Operation

© 2023. Published by AHFE Open Access. All rights reserved. 66

Cognitive Engineering in Training 67

depends on a rich body of knowledge and skills. We suspected that details of the needed skills and knowledge are not thoroughly documented or trained. Indeed, inadequate FP monitoring has been identified as a factor contribu- ting to multiple accidents and incidents, particularly on descent (Active Pilot Monitoring Working Group, 2014). We wanted to better understand moni- toring, particularly monitoring FP and its underlying knowledge and skills. In turn, such understanding might be used to improve learning.

Our Approach: Cognitive Engineering Cognitive Engineering (CE) is a framework that helps design systems for human use, accomplishing the intended function, within the relevant con- straints and resources, when human cognition is a critical component. CE provides methods to analyze the constraints required by the work and how work is currently done, when relevant work exists (Roth, Patterson, & Mumaw, 2002). This can identify current tasks, strategies, and the underlying skills and knowledge. CE methods include Cognitive Work Analysis (CWA), Cognitive Task Analysis (CTA), and Critical Incident Reports (Klein, Calde- rwood, & Macgregor, 1989; Schraagen, Chipman, & Shalin, 2000; Vicente, 1999). Results of an analysis can then guide design of artifacts, such as auto- mation, and other aspects of the sociotechnical system -- including training -- throughout the lifecycle (Sanderson, Naikar, Lintern, & Goss, 1999).

We applied this framework after the physical and software components had been built and were in use. Specifically, we investigated how to support learning the cognitively demanding task of monitoring and managing the FP of airliners. We investigated both what content was missing from current training and how that content could be effectively delivered.

We wanted to know whether understanding current activities used for FP monitoring and management by effective pilots might inform design of lear- ning, which might in turn improve performance. When Cognitive Engineering

Figure 1: Blue boxes and arrows show a simplified schema of a frequent role of CE methods: inform the design of the software and hardware. This influence can then flow through the later-developed parts of the sociotechnical system. Green dashed boxes and arrows show a feedback path “retrofitting” training and learning for an existing system using information about operational use, learning from experience, and a structured method for applying learner-centered design process and principles to content.
Original PDF Figure 1:

methods such as Cognitive Work Analysis (CWA) are effectively applied, they frequently are “upstream” in the development process, as illustrated in Figure 1. Ideal use of CE is presented as being carried out prior to the design and development of automation and the larger sociotechnical system and then results can be used in all later phases. Broadly, the hardware and software design can drive the procedures, which then drive training design. In our case, we were looking at a fully operational system to understand how highly skilled pilots monitor FP, to identify knowledge and skills likely learned from experience rather than training, and to use this as feedback to design the learning environment and activities.

IDENTIFYING CONTENT TO BE LEARNED USING CE INTERVIEW METHODS Process: Opportunistic CWA

Our high-level goal was to investigate how pilots carry out FP monitoring and management in the existing, highly automated systems. Ideally, the air- plane follows the FP programmed into the FMS. Also, ideally, other factors, such as traffic or weather, do not force the airplane into a tactical change from the FMS flight plan. We wanted to understand the departures from this ideal scenario and how pilots monitored to anticipate and respond with pilot intervention to alter automation flight modes and FP targets as appropriate.

Our investigations were informal, constrained by what was feasible, and informed by CE methods. Our focus for investigation was descent using Standard Terminal Arrival Routes (STAR). A STAR specifies the airplanes required vertical and lateral path as a series of waypoints with required alti- tudes and, often, airspeeds. A STAR typically takes an airplane from top of descent (TOD) to an approach to the runway. Our inquiry methods were interviews, and our interviewees were experienced line pilots.

Focus Phase of Flight. Descent, particularly in an arrival done on a STAR, can be a challenging, complex phase of flight constrained by external and internal requirements. Externally, the physics of flight and the specific air- plane performance-limits impose energy management requirements that must be met while also meeting the STAR specifications. Internally, the FP origi- nally specified in the FMS may need changes when the current context differs from what was programmed in the FMS computer. Managing the automa- tion requires timely monitoring of the autoflight modes, their targets, and the resulting FP. Pilots may need to change the automation modes and tar- get. Further, the autoflight system may make uncommanded mode changes. Understanding these constraints and how they make the work difficult is an important part of work analysis. Inadequate monitoring in descent has con- tributed to accidents and incidents which suggests that understanding this phase of flight will be particularly useful.

Interviewees: Selection criteria for pilot subject matter experts (SME) were informal. We sought out experienced pilots and looked for those who were also good at introspecting and articulating piloting processes. We intervie- wed a dozen pilots, half individually and half in a group setting. This was a

Cognitive Engineering in Training 69

convenience, network-based sample and should not be thought of as repre- sentative. Some pilots had existing ties with NASA and previously involved as participants or SMEs. Others were contacted through those pilots with an existing relation. At least one was previously involved in developing airline training, though not on the topic in focus here. These pilots flew for 6 US air carriers.

Inquiry Methods. Observation in-flight or in simulators was not feasible. We used interviews drawing on CWA and CTA methods. Interviews were primarily face to face, but some were remote. Some made use of pictures of flightdeck displays or flight charts both as prompts and to identify what the SME pilot was referring to. Interviews followed topics introduced by our SME pilots. Our interviews asked three broad types of questions. 1) What do you do on a routine STAR descent? Follow up questions asked about decisions made, the information needed and its source, and the strategies or heuristics used. 2) What is an example of a challenging descent (i.e., a critical incident) or a descent that makes it a ‘hard day’? Follow-ups asked what was hard, what factors made it difficult, and how the situation was managed. 3) We asked about types of difficult descents generalizing from examples pro- vided. These might be described in terms of the disruptive factor(s) and its impact based on time of occurrence. To check our understanding of what the pilot SME reported, we built graphical representations that abstracted details to show how a described type of event might unfold over time, including decision points (or “gates”) and outcomes.

In questions of type 2 and 3, we worked to identify what the constraints on successful descent were, including the timing and interactions among events. We did not push for reports of monitoring for exceptional, emergency con- ditions. It may be difficult to accurately recall high-stress, unusual situations. Questions included asking about what indications were monitored but shif- ted to discussion about the types of difficulties encountered. Questions were often reformulated; because we were asking about routine, though challen- ging, events, it sometimes took alternative framing to clarify our interest in descriptions of “just doing my job.”

Product: Emerging View of FP Monitoring These interviews showed monitoring to be an active process of observing and making sense of the situation. We provided a sensemaking model of monito- ring to capture the types of behaviors our SME pilots described. To monitor, pilots build up and make use of a model of the situation; this guides attention and provides reference values for comparing current with expected values. The situation model includes relevant components of knowledge from long- term memory about how systems work, such as models of how different autoflight modes work or how air traffic control (ATC) usually manages dif- ferent traffic flows. The situation model includes anticipation of what will happen, planning of needed pilot intervention, and prediction of the effects of different actions. For example, effective monitoring prior to the top of descent recognizes that the current mode is VNAV ALT, that the airplane will not descend when cleared for descent in this mode, that the mode needs to

be manually changed to VNAV PTH, and that this should be stated to ensure both pilots have a shared model of the situation.

Monitoring involves a sensemaking cycle of:

  • prioritizing a question or relevant information to obtain,
  • obtaining and assessing the selected information, and
  • identifying whether and what actions need to be taken.

Communication is needed both to report updates to the situation model and to coordinate on the monitoring process itself. This sensemaking model treats monitoring as an active cycle of inquiry and is presented in brief (Billman, Mumaw, & Feary, 2020) and more extensively (Mumaw, Bill- man, & Feary, 2020). While the model does not address all aspects of monitoring, it provides a foundation for identifying many recurring com- ponents of monitoring that can be easily described and may be particularly learnable.

In addition, these interviews highlighted ATC changes to FP as a particu- larly prevalent source for adding complexity to the work of monitoring. ATC clearances change the FP to be followed from what was initially specified in the flight plan for the automation to execute. Wind might also vary from the value that was programmed initially. We heard about the interactions with automation such changes might produce. These changes may require entering new target values in the autopilot system, assessing whether it is feasible for the aircraft to comply with a clearance, whether mode should be changed, and whether the intended changes have in fact been entered and are in effect. Indeed, it’s an important decision whether to enter changes into the FMS flight plan or to revert to a less automated mode to avoid those automa- tion management activities. In addition, the automation may change modes, without the pilot entry, adding complexity to monitoring.

We also obtained reports of various strategies for monitoring in parti- cular types of situations. Critically, pilots stated that the strategies they were reporting had not been trained. Rather, these had been learned from experience.

These reported strategies varied greatly in specificity. They included quite general strategies about communicating expected upcoming events to the other pilot. Moderately general strategies addressed changing the autofli- ght modes and targets in a variety of conditions; modifying the automation was much more common than simply “turning it off”, that is, switching to completely manual flight. (Turning off the autoflight system may be reported more often for more serious threats and may be produced by pilots less fami- liar with the details of automation than our pilots.) A route-specific strategy directed the pilot to “find the ribbon of red lights,” which is the Long Island Expressway, as a cue that you are nearing JFK. One flight-specific strategy was asking the Pilot Monitoring to make sure the Pilot Flying (PF) turned off the landing lights because the PF had forgotten this on the previous leg. Of course, this very specific strategy could be generalized to asking for special monitoring attention for any vulnerability specific to the current flight, but it was presented as a one-time occurrence.

Cognitive Engineering in Training 71

The understanding developed from these interviews was used in an explo- ratory training study (Billman, Zaal, Mumaw, Lombaerts, Torron, Jamal, & Feary, 2021), which found modest but reliable effects of training in a sim- ple test-retest design. Further, it set the stage for a more principled design of learning materials.

BUILDING A LEARNING MODULE USING LEARNING-DESIGN METHODS Linking Results of the Cognitive Work Analysis With Principles of Learning Design

Basing the design of a learning module on principles of learning is a form of cognitive engineering. This includes drawing on the results of the cognitive work analysis to identify and organize content for training. Successful pilo- ting - with current automation systems in the operational context - requires a range of skills, knowledge, and strategies that pilots say were not acquired in training. In addition, we used a learner-centric method to design a trai- ning module, so the process of learning is structured to respect principles of human learning.

Process: Method for Learner-Centric Design Using a systematic design for learning, we produced a web-based training module designed to help pilots perform adaptive FP monitoring behaviors. This system organizes the content for learning into a hierarchy of intellectual skills. The system provides a flexible platform for integrating a task analysis, learner analysis, and goal analysis to attain a terminal goal of demonstrating proactive pilot monitoring behaviors in a specified context. We applied the Dick and Carey design model to integrate three proactive flightpath moni- toring activities into a learner-centric intervention (Dick, Carey, & Carey, 2015). The design uses interactive webtools to

1) Permit integration and practice of the Sensemaking Model for monito- ring and managing FP, 2) Practice a strategy for analyzing standard terminal arrivals (STARs) for flightpath management threats, and 3) Demonstrate measurable crew communication behaviors that support proactive and adaptive monitoring activities.

Taking the learners’ cognitive and affective characteristics into account, the current design provides context and practice for behaviors that let opera- tors adapt to automation and to an operational environment that were only partially designed with human factors in mind. Focusing on learning goal- oriented actions, and not procedural steps, creates a human-centered learning system with multiple paths to the final learning objective. Each sub-skill is measurable and can be adjusted to the target audience. As the operational context evolves, such a systematic design enables modifying the hierarchy to address the revised problem space. Finally, since the learning system measu- res performance outcomes at each sub-component in the hierarchy, gaps in

performance can be linked to specific areas where the learning intervention needs improvement.

Content of Learning This learning intervention addresses monitoring in the specific context of flying STARs. It begins by demonstrating the benefit of anticipatory work to develop adaptive pilot monitoring behaviors. In the second section, the learners demonstrate that they recognize monitoring is in part an anticipa- tory behavior. In a narrated real-world scenario, the learners practice asking questions about the future state of the aircraft. (Note similar questions are often asked about diverse automation). For example, learners are presented with a task of setting a target for FP monitoring against which to compare the aircraft state as the situation evolves. If they select an option that does not involve the future state of the aircraft, they receive corrective feedback. The third section presents a simplified model for making sense of an evolving situation This specifies an iterative, three-component process:

1) Asking questions about the future state of the aircraft, 2) Gathering relevant information to answer these questions, 3) Deciding on an action that best responds to the situation as the crew understands it.

The learner applies this model to a flightpath monitoring situation invo- lving internal and external cueing of flightpath management challenges. The fourth section conveys a strategy to analyze a STAR by identifying any flight segments that can pose an energy management threat. Finally, the module conveys three general communication behaviors that enable a crew to col- laboratively and proactively perform the monitoring strategies. The design scaffolds the content in a set of real-world challenges familiar to transport category pilots.

Learners work through a progression of intellectual skills to recognize, demonstrate, analyze, and apply subordinate skills. They emerge with an adaptive monitoring strategy to improve FP monitoring performance in dif- ferent flight conditions. Planned web-based and simulator studies will assess the impact of these strategies and explore the possibility of transfer across contexts.

Principles Supporting the Learning Process To promote learning by activating relevant cognitive structures (Merrill, 2009), learners interact with the module to recall and demonstrate prior knowledge associated with the targeted intellectual skills. Using interactive software embedded in the website, presentations pause at strategic points within the real-world scenarios and present the learner with questions about their own experiences and attitudes relating to the scenario. The question sets provide feedback that varies depending on the learner’s response. These acti- vities are designed to integrate these mental models into the learner’s personal experience.

Cognitive Engineering in Training 73

Example Activities Illustrating Learning Principles In accordance with Mayer’s Redundancy, Voice, and Personalization Princi- ples (Mayer, 2009), a natural voice narration accompanies simple graphics depicting the vertical path of aircraft as pilots progress along a STAR (see Figure 2).

Two examples illustrate how the module implements some of the lear- ning principles. Throughout the module, questions are designed to promote learner reflection on their own experiences in accordance with Merrill’s Inte- gration Principle. Real-world scenarios allow users to recall prior knowledge and integrate the intellectual skills into their own experiences. In the section designed to have the learners understand the value of anticipatory behavi- ors when preparing for a complex flight operation, the learner steps through and is asked about events in a scenario when ATC issues a clearance that adds energy to the flight by delaying descent. As the narrative of the flight unfolds, the video pauses at key points to ask questions such as: “What will the FMS do to attempt to comply with the constraint?” followed by: “Is there anyth- ing the crew could have done in advance help the auto flight system comply with the constraint?” And later: “As PM, have you ever waited too long for the PF to address a FP management issue?”

A later section depicts another real-world scenario that activates the lear- ners’ prior knowledge, while helping to build the appropriate mental model (Merrill, 2009). As this scenario evolves the presentation pauses, and the learner is asked what question the crew could ask about the future state of the aircraft to understand the impact of an ATC clearance. The learner faces three options familiar to inquisitive pilots: Is a tailwind impacting them? Are they encroaching on preceding traffic? The third, most-relevant option asks what altitude the flight must cross a downrange waypoint in order to meet a final crossing restriction. If the learners choose the distractor questions, they receive feedback describing why the chosen answer does not help predict the future state of the aircraft, and an explanation why the final selection creates a FP monitoring objective useful in analyzing the flight’s progress relative to the desired state.

Figure 2: Animation showing vertical profile of airplane FP with STAR altitudes and distances between way points.
Original PDF Figure 2:

SUMMARY AND CONCLUSION Summary

Our problem concerned analysis and design for learning, when the socio- technical system was already built and in use. We drew on the user-centered framework of Cognitive Engineering both to identify what needed to be lear- ned and how to support learning. Observing actual or simulated flight was not feasible. However, discussion with a number of pilots provided valua- ble insight to the processes and strategies used in monitoring FP. Indeed, for highly cognitive work it may be particularly valuable to have self-report, not only observational data.

Interaction with automation was a pervasive element of these experts’ descriptions. The prevalence of anticipation, comparison, and planning for possible events highlighted the proactive nature of monitoring. The inte- rviews informed a model of monitoring that characterizes the proactive, investigatory nature of FP monitoring. Pilots commented that they learned about monitoring though experience, not just training. This provided encou- ragement that this examination of what pilots knew could inform design of a learning environment.

This work carried through to build a learning module. and its design was guided by an established, validated method for design for learning. This provided a method for identifying and organizing the content, or learning objectives, and providing a set of vetted principles that facilitate learning and can be used in design.

Limitations Empirically, we were limited in lack of access to real-time performance. In particular, we did not gain insight into the timing of perceptions and actions as they might affect monitoring. Further our scope of inquiry focused on one phase of flight and emphasized the specific context of descent using STARs. While this context is particularly complex, and effective monitoring of FP particularly challenging, there are many other situations for monitoring FP. We were also limited in the resources for developing the learning module. In particular, future revisions might benefit from additional development of interactives and feedback.

Future Work

Summative evaluation is planned for web-based and then simulator-based delivery. Evaluation will include cognitive and affective domain components, exploring what participants learned by measuring how well and how often the participants applied the content in the learning objectives. Simulator eva- luation offers a broader range of observable measures and has an expanded scope compared to the web-based learning evaluation. Observable behavior markers include intra-crew communication, control actions, and autoflight selections. These activities drive aircraft path outcomes as the crew responds to external influences such as ATC clearances, challenging arrival geometries,

Cognitive Engineering in Training 75

and winds that complicate FP management goals. In turn, evaluation of effectiveness of the learning modules will inform further development.

CONCLUSION

This work contributes to an expanding view of monitoring, which emphasi- zes the proactive, anticipatory process of building, sharing, and updating a situation model. It provided a less common application of cognitive enginee- ring approaches, where we looked at design of learning environments for an in-operation system, rather than early input to design of software or hardw- are. This work also contributes an expanded view of technology support for learning. It adopts a learner-centered perspective of designing for learning, rather than the institutional perspective implicit in training. It also provides a method for identifying skills and knowledge of expert users and provides learning opportunities to speed the development of expertise across the pilot community and across monitoring challenges.

REFERENCES

Active Pilot Monitoring Working Group. (2014). A practical guide forimproving Flight Path monitoring. Final Report of the Active PilotMonitoring Working Group. Alexandria, VA: Flight Safety Foundation. Billman, D, Mumaw, R., & Feary, M. (2020). A Model of Monitoring as Sensema-

king: Application to Flight Path Management and Pilot Training. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 64). SAGE Publications. Billman, D., Zaal, P., Mumaw, R., Lombaerts, T., Torron, I., Jamal, S., & Feary, M.

(2021). Training Airline Pilots for Improved Flight Path Monitoring: The Sense- making Model Framework. In Proceedings of the 21st International Symposium on Aviation Psychology, 18–21 May, 2021, pp. 403–408. Dick, W., Carey, J. O., & Carey, L. (2015). The Systematic Design of Instruction.

Merrill Pearson. Klein, G. A., Calderwood, R., & Macgregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on systems, man, and cybernetics, 19(3), 462–472. Mayer, R. E., (2009). Multimedia Learning. (2nd ed). Cambridge University Press. Merrill, M. D. (2009). First Principles of Instruction. In C. M. Reigeluth & A. Carr

(Eds.), Instructional Design Theories and Models: Building a Common Knowledge Base (Vol. III). New York: RoutledgePublishers Mumaw, R., Billman, D., & Feary, M. (2020). Analysis of pilot monitoring skills and a review of training effectiveness. NASA/TM–20210000047. Roth, E. M., Patterson, E. S., & Mumaw, R. J. (2002). Cognitive engineering: issues

in user-centered system design. Encyclopedia of software engineering, 1, 163–179. Sanderson, P., Naikar, N., Lintern, G., & Goss, S. (1999, September). Use of cognitive work analysis across the system life cycle: From requirements to decommissi- oning. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 43, No. 3, pp. 318–322). Sage CA: Los Angeles, CA: SAGE Publications. Schraagen, J. M., Chipman, S. F., & Shalin, V. L. (Eds.). (2000). Cognitive task

analysis. Psychology Press. Vicente, K. J. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. CRC press.

Cover of the proceedings volume Neuroergonomics and Cognitive Engineering

Explore the complete digital volume

Neuroergonomics and Cognitive Engineering

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-78-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →