Diana Paola Moreno-Alarcon, Corinne Bieder (2022). Co-Development Approach integrating Training into the Design Process. In: Training, Education, and Learning Sciences. AHFE Open Access, vol. 59, pp. 99–108. https://doi.org/10.54941/ahfe1002389
Since the introduction of advanced automation technology within highly dynamic and complex systems, traditional approaches to training no longer appear to be adequate in preparing end-users of such technologies (Carroll and Olson, 1988). Advances in design processes have resulted in the creation of user-centered designs that employ an iterative design process that follows a cyclical pattern, where the prototype is designed, created and evaluated, throughout multiple loops. Once the final design solution is validated, a training session is organized to prepare the end-user for the Human in the loop Simulation. Since the training sessions and the preparation of these training sessions are conceived after the completion and validation of the design; any issues with the design often result in extended training sessions to compensate for human factors that were overlooked in the initial design. Nonetheless, such enhanced training “cannot and should not be a fix for bad design” (Sarter, Woods & Billings, 1997.p.1936). This research intended to consider the training from the onset of the design in order to avoid transferring the burden of a poor design onto the training. This was not fully possible in the framework of the SAFEMODE project, where this research was conducted, as some design iterations had already taken place prior to the implementation of the proposed training considerations. Instead, a training approach involving future users as part of the system design, hereafter referred to as “Co-Development”, was proposed and tested during the creation of a safety net or alert for air-traffic controllers. Building on this added value to user-centered design, we investigated: To what extent does the co-development of the training with end-users improve the training, the design, and the acceptability of the designed alert?This paper presents this Co-Development approach and the outcome of its first implementation; particularly in terms of its impact on the training, the design, and the acceptability of the alert. Specifically, this system design approach allows for human factor issues to be identified and hence corrected early in the design stage, thus yielding a potential impact on the acceptability and usability of the alert, while simultaneously affecting safety through an improved user-centered design experience. This paper also proposes a future research direction that consists of a joint training and design approach that considers training right from the onset of the first iteration of the design. As a result, the training is conceived jointly and evolves simultaneously with each design loop, further improving the design, and thus reducing the burden placed onto the training due to design shortfalls and impossible design challenges that are seemingly improbable to meet.
Since the introduction of advanced automation technology within highly dynamic and complex systems, traditional approaches to training no longer appear to be adequate in preparing end-users of such technologies (Carroll and Olson, 1988). Advances in design processes have resulted in the creation of the user-centered design process that employs a cyclical pattern, where the new technology is planned, analyzed, designed, prototyped, and evalua- ted, throughout multiple design iterations (cf. Figure 1). Once the designed solution is in its final iteration and the designed solution has been valida- ted, a training session is organized to prepare the end-user for the Human in the loop Simulation. Since the training sessions and the preparation of these training sessions are conceived after the completion and validation of the design; any issues with the design often result in extended training sessions to compensate for human factors that were overlooked in the initial design. Nonetheless, such enhanced training “cannot and should not be a fix for bad design” (Sarter et al., 1997, p. 1936). Moreover, theoretical and practical training/simulation sessions in aviation (and other high-risk sectors) con- sume extensive organizational, financial, operational and temporal resources,
Figure 1: Co-Development approach that integrates Training within the user-centered design process.
therefore it is critical to minimize them through system and training design right the first time (or as right as possible the first time).
Previous studies (Daniellou, 2021; Moreno Alarcon and Bieder, 2021) highlight the importance of integrating HF experts “as full members of the design team” to co-construct efficient HF recommendations that would othe- rwise be produced independently or autonomously and then “exported to the design team”. This integration of HF experts reduces the potential counter- productive effect of considering the integration of HF at an advanced stage of design where changes may no longer be economically or technically feasible to accommodate; thereby transferring the burden to training to compensate for the lack of HF considerations at the design stage. This paper aligns with Moreno Alarcon and Bieder’s (2021, p. 236) request to “evaluate how an early integration of HF in the design impacts the training needs and global efficiency” by proposing a framework for a human-centered joint training and design approach that considers training from the onset of the first itera- tion of the design. As a result, the training is conceived jointly and evolves simultaneously with each design loop, further improving the design, and thus reducing the burden placed onto the training due to design characteristics1 or design shortfalls making an efficient training extremely challenging if not impossible.
This research intended to consider the training from the onset of the design, to avoid transferring the burden of design shortfalls onto the training, during the creation of a safety net or Wake Vortex alert for air-traffic controllers (ATCOs). This was not fully possible in the framework of the SAFEMODE
1Certain design characteristics may be interpreted as design shortfalls from the training perspective as their use can lead to training inefficiencies. For example a designer may conceive access to tools through different sub-menus which appear intuitive to the designer but not to the user; thus resulting in prolonged training sessions to ensure the user comprehends which tools are categorized under which submenu, and how to implement them.
Co-Development Approach Integrating Training into the Design Process 101
project, where this research was conducted, as some design iterations had already taken place before the implementation of the proposed training con- siderations. Instead, a “Training Development Workshop” to “co-develop” the training, was proposed and tested with retired ATCO professionals post- simulation2 during the first iteration of the Training and Simulation Process (see Fig. 1). Improvements made to both the design and training were evalua- ted during the second iteration of the Training and Simulation Process with actively employed ATCO professionals from the same European Air Naviga- tion Service Provider (ANSP). Building on this added value to user-centered design, the initial research question aimed to resolve: To what extent does the co-development of the training with end-users improve the training, the design, and the acceptability of the designed alert?
However, due to the operational and organizational constraints associated with implementing training from the onset and inconclusive results due to the extensive experimental variables3 with the initial experimental design, this paper broadens the initial research question to: How can the operational efficiency of the system be improved, both in terms of how the system is designed and how the system is applied?
This paper is organized into three sections: the first presents the state-of- the-art of training in User-centered Design, the second presents the proposed Co-Development Approach, the third discusses organizational and cultural considersations to keep in mind when implementing the approach, while the final section provides concluding remarks and future perspectives for its implementation.
TRAINING IN USER-CENTERED DESIGNS In automation literature, human-centered automation aims at designing systems that cooperate or support the user. As a result, “the notion that peo- ple should be expected, instead, to conform to automation, […],is antithetical to human-centered automation or user-centered design”(Karsh, 2009, p. 23). In fact, “while increasing automation might relieve the operator of some tasks, they are likely to create new and more complex tasks that require more, not less, training” (Lee and Seppelt, 2012, p. 1618). Additionally, increased automation has changed the role of the user/controller and led to a new form of “technology-induced human error”, where the response to these errors has primarily been to alter the interface, increase the role of the automation, and enhance the training (Leveson et al., 1998). Such errors are further acer- bated by “designer bias” or cognitive inconsistencies between what makes common sense to the designer (who has intimate knowledge of the inner workings of the system) and what makes sense to the user (Karsh, 2009).
2Due to organizational and design constraints, the Training Development Workshop was conducted at the earliest possibility (post-simulation during the initial simulation iteration) instead of during the intended Design Process Phase. 3Extensive experimental variables including the timing of the Training Development Workshop (post- simulation instead of during the design process), a reduced number of participants (n = 6, n = 6), participant familiarity with the alert prior to the training and simulation due to their participation during the design phase (n = 2), etc.
102 Alarcon and Bieder
Therefore, a quality design, not only requires integrating end-users into the design phase to reduce “designer bias”, but also requires training to ensure the user develops an accurate model of how the system functions and derives its recommendations (Lehner and Zirk, 1987).
Leveson et al., (1998) propose an approach to detect error-prone automa- tion features early in the development process while changes to the conceptual design can still be made. This approach aims to identify automation design characteristics that increase cognitive load, allowing the design team to rede- sign the automation without reducing system capacities. By knowing what factors impact cognitive load and potential problematic design features, desi- gners can predict where errors could occur making changes to the interface and the training more effective (Leveson et al., 1998).
Nonetheless, while effective training is crucial to use the system, “it can- not completely compensate for a poor design” (Karsh, 2009, p. 33). Schutte (2017, p. 245), further affirms that “excessive training- which is costly” is often the antidote for poor designs that “require significant support from the [user] to overcome deficiencies that depend on the [user] for memory, vigila- nce, computation, precision etc.”. Specifically, “in the modern aircraft, we are seeing this antidote being applied widely—so much so that more time may be spent on training the automation rather than training on accomplishing the mission regardless of the level of automation” (Schutte, 2017, p. 245).
Specifically, with regards to training, FonSCI (Foundation for an Industrial Safety Culture) recommends that in the planning and design of high-risk systems, “the training needs of the future [users] should be anticipated to enable the effective appropriation of the new [system]” (Daniellou, 2021, p. 53). Furthermore, “entire training process [should not be]concentrated at the end of the design stage and during the implementation of the [..] pro- ject” but rather there should be a “progressive organization of training, from the detailed design phase […] planned a long time in advance” (Daniellou, 2021, p. 54). As a result, this study responds to a call for research by Jent- sch et al., (2002, p. 123), indicating that “ future studies should […]examine how training solutions targeting at the earlier stages (i.e. sensation and perce- ptual) […] might have a cascading effect in relation to the elimination of problems further along in the processing model”. As indicated by the previ- ous literature review, the lack of a joint design and training development can result in issues to the detriment to both the design and the training aspects. Furthermore, although scholars raise the importance of addressing training early on, limited applied research has been conducted to assess the legiti- macy of such an approach. The following section defines and illustrates the Co-Development Approach, where designers and trainers work collectively to develop the design and training collaboratively.
CO-DEVELOPMENT APPROACH: TRAINING AS PART OF THE ITERATIVE DESIGN PROCESS The Co-Development Approach is a joint training and design approach that integrates training support(s) into the user-centered design process allowing the training support to be conceived jointly with each design and therefore
Co-Development Approach Integrating Training into the Design Process 103
evolving simultaneously with the new prototype at each design loop. By superimposing a training loop onto the iterative design process, this joint training and design process shifts the training preparation and planning step that usually occurs after the design into the design process by anticipating the training support. Specifically, this joint training and design approach depicted in Figure 1 below, begins with the User-centered Design Process (plan, analyze, design, prototype, evaluate) throughout multiple iterations, and then proceeds to the Training and Simulation Process (software deve- lopment in the simulator, training, human-in-the-loop Simulation) that may also follow multiple iterations that loop back through to the User-centered Design Process.
The Co-Development framework integrates several training-related mile- stones at different levels of the User-centered Design process. The first training milestone consists of Training Requirements at the Plan Stage deri- ved from Human Factor (HF) considerations and design requirements, which will influence the design of the system from the initial conception of the ini- tial prototype. These training requirements encourage the integration of HF into the design thereby making the system more efficient, intuitive (afforda- nce) and potentially requiring limited training. The second training milestone occurs at the Prototype Stage and consists of Training Development Work- shops that address HF considerations of the learning characteristics and capabilities of users which in turn influence the design and the content of the training support. Such HF considerations may include the number of training/simulation sessions (sessions/day), the duration of the training sessi- ons (hours/day), or the quantity of information conveyed. The co-developed training support is updated with each design iteration and includes elements necessary during the theoretical/practical training and simulation sessions such as presentations, videos, operating procedures, simulation scenarios, etc. The third training milestone occurs at the Evaluate Stage and consists of Usability Testing based on the Training Support, conducted by HF Experts and Trainers. Here, the prototype is tested using the training support from the latest iteration, identifying any training support/design challenges to be addressed with the next design iteration. Given the iterative nature of most designs, an equal number of training supports as design iterations (prototype visions) are anticipated.
There are numerous benefits to training support that evolves with each new design, as incorporating training-related milestones within the design loop helps identify overlooked human factors or user-friendly elements that should be incorporated within the next design. Specifically, a feedback loop exists between design and training as changes to the design (from the Design Thinking Workshop) are integrated into the training support and the les- sons learnt from Training Development Workshops are considered in the next design phase. Additionally, the integration of personnel from the trai- ning domain during the design (thinking) process fosters a training-bound perspective, group cohesion and innovative problem solving amongst all sta- keholders. From a human and economic costs point of view, this approach saves time and costs, particularly in design projects where the training requi- rement is high, as trainers can participate in the design process from the onset;
104 Alarcon and Bieder
or in the case of significant time constraints, they can participate during syn- thesis presentations of for example the user research step to understanding the critical pain points, objective statements and personas. Moreover, the usabi- lity testing based on the training support milestone evaluates the operational efficiency of the system by simultaneously evaluating the congruency betw- een how the system is designed and how the system is applied. This system evaluation assists in identifying and preventing potential Technical and Ope- rational issues or incompatibilities early at the Evaluation/ Validation Stage of the iterative design; enabling their correction at the design stage. Often, when the training process is conceived after the final design, these technical and operational issues are identified during the training sessions or the human in the loop simulations, making any alterations to the final design futile. Instead, by anticipating Technical and Operational issues by the training-related mile- stones within the User-centered Design Process, the training and simulation process may become a validation rather than an improvement step that would otherwise lead to another full design, training and simulation iteration.
Table 1 below presents the stages of the User-centered Design process and the Training and Simulation process, detailing how the training and design aspects integrate within the Co-Development Approach.
DISCUSSION: APPLICATION OF THE CO-DEVELOPMENT APPROACH The application of the Co-Development Approach combines qualitative and quantitative feedback via questionnaires, debriefs, and training workshops. Several organizational and cultural considerations should be contemplated during the implementation of this novel approach as organizations may resist deviating from traditional protocols. As a result, the following suggestions arise from the lessons learned following the first implementation attempt of the proposed training considerations for a safety net for Wake Vortex with Air Traffic Controllers (ATCOs) from a European Air Navigation Service Provider (ANSP). Several organizational aspects arise: first, the required collaboration between training and design professionals (often from diffe- rent departments) in a process that is not natural or intuitive. Second, this approach requires pre-identifying the trainer before commencing the design process. This was not possible during the design of the safety net for Wake Vortex where the identity of the trainer was not confirmed until the end of the design stage. Third, sufficient institutional resources are required to ensure the number and status of participants are representative of the whole spe- ctrum of end-users. This was not the case with the first implementation were due to institutional limitations, where ATCOs varied in professional status (retired professionals vs. actively employed) resulting in less than optimal testing conditions due to small and unrepresentative sample size, yielding potential experimental design biases. Finally, cultural aspects should also be considered including, for example, the existence of an organizational design culture whereby the design is central to the designers or design team and does not naturally receive input from the trainers. As a result, there is a strong need for upstream preparation to ensure that all stakeholders comprehend
Co-Development Approach Integrating Training into the Design Process 105
Table 1. Definition and inter-related design and training elements of every stage of the co-development approach.
Table available in the original PDF paper.
Stage Definition and Inter-related Design/ Training elements of every stage of the Co-Development Approach Plan Develop a deep understanding of the challenge Define user requirements and training requirements derived from HF considerations (with designers, end-users, HF experts, and trainers). Trainers contribute as Subject Matter Experts (SMEs) to the understanding of the problem from a trainer’s perspective. Analyze Define and articulate the problem to solve Conduct User Research with Target users using qualitative and quantitative measures to find insights, determine users’ motivation, needs and pain points in order to guide successful design. Trainers are involved as SMEs to support HF in structuring the Research protocol and provide user-based insight from their experience (as trainers). Design Ideate or generate concepts which are then selected and designed according to Research insight Transform user-based insights and needs into early-stage concepts that will be further refined. Develop Design wireframes that consider HMI interactions (input/output), on-screen navigation and information hierarchy during Design thinking workshops. Trainers contribute as SMEs to the design by considering potential training challenges of the design.
User-Centered Design Process Prototype Develop dynamic prototypes for usability evaluation As the design of the future solution becomes more mature, designers create a dynamic prototype that will allow users to test the relevance of the proposed design. In parallel, trainers organize Training Development Workshop with users, to address HF considerations of the learning characteristics and capabilities of users and co-develop the training support. Trainers (along with end-users, and HF experts) support the designers in the development of the prototype and update the training support (theoretical, practical, simulation scenarios, etc.) with each design iteration. Evaluate Ensure design pertinence by performing review and testing sessions with users Trainer and HF organized sessions to assess the usability of the design solution. These sessions can be structured as a cognitive walkthrough with Trainers or Usability Tests based on training support with users. Software Dev. In the Sim.
Develop and test Simulation software based on selected simulation scenarios Design hand-over and follow-up to Simulation Technicians/Engineers. Trainers prepare and plan the training based on the HF training requirements, HF training considerations and validated training support.
106 Alarcon and Bieder
Table 2. Continued.
Table available in the original PDF paper.
Stage Definition and Inter-related Design/ Training elements of every stage of the Co-Development Approach Training Administer Theoretical and Practical Training for simulation use Trainers conduct Training sessions with users based on the validated Co-Developed training support. Simultaneous HF observations by HF experts assess if training, design and safety elements were overlooked. Human-in- loop Simulation
Real-time simulation with human interaction HF experts and trainers conduct the Simulation session with simultaneous HF observations. Simulation Synthesis integrates user feedback from the simulation/debrief sessions, which are integrated into the Analyze Phase of the next iteration.
Training & Simulation Process
the process, the resources required and what the approach entails - do not underestimate the reach of the proposal.
CONCLUSION
Designs with perceived affordance are simple to use and often require fewer training constraints than poorly constructed ones. In contrast, misconstrued designs often require more training components or operational procedures to compensate for the design’s limitations. The Co-development Approach allows for human factor issues related to both the design of the system itself and the required training (both being critical to ensure operational efficiency) to be identified and hence corrected early in the design stage, thus yielding a potential impact on the usability of the design, while simultaneously affecting safety through an improved user-centered design experience. This approach integrates the training milestones into the design phase and co-develops the training with experienced end-users allowing these actors to form a mental model of the system, actively examine the accessible settings and dynamics of the design; thereby preventing prolonged and complex training sessions that take on the burden of a poor design. As a result, this proposed appro- ach contributes to the training, design and human factors safety literature by integrating training solutions at the earlier stages of the design (Jentsch et al., 2002) and evaluating the impact of integrating early HF considerations on the training and design ( Moreno Alarcon and Bieder, 2021).
As a design evolves over multiple design loops, it can fluctuate in usability. By superimposing the training loop onto the design process, it is possible to monitor the evolution of the design’s usability, as the design’s evolution will be reflected in changes to the training support. As a result, the training can serve as an indicator of the design’s usability by promoting an “affordant” design concept through training requirements that derive from HF considerations, and by improving the training efficiency through HF considerations related to
Co-Development Approach Integrating Training into the Design Process 107
the learning characteristics of humans. The heuristics of the experienced end- user may also provide designers with insight into the current designs’ in-use capabilities and limitations, as well as the circumstances under which errors in-use are more likely to occur. Additionally, this process allows experienced end-users and designers to: identify overlooked human factors that should be incorporated into the next design, prevent potential Technical and Ope- rational issues or incompatibilities, and propose corrective mechanisms for these potential design complications prior to the validation stage of the itera- tive design. Particularly, as such in-use elements identified later on during the training sessions or the human in the loop simulations (once the final design solution is validated), make any alterations to the final design futile.
The first implementation of the proposed training considerations for a safety net for Wake Vortex, within the framework of the SAFEMODE pro- ject, was impossible due to organizational restrictions, cultural constraints and experimental design limitations, partly related to the novelty of the sequence proposed between system design and training design. Nonetheless, future research perspectives of the approach under minimal experimental restrictions (large sample size, participants with similar professional sta- tus and experience, design at the planning stage, multiple design iterations, etc) in organizations that expedite organizational change and implementa- tion of new processes, can provide future insight into the impact of the Co-Development Approach on the training efficiency, the design perception, the acceptability of the new design, and on safety through improved user experience.
ACKNOWLEDGMENT
This publication was conducted as part of the SAFEMODE Project, a Euro- pean Commission H2020 project (grant agreement 81496) that aims to develop a Human Risk-Informed Design (HURID) framework that facilita- tes the accessibility of HF data and information, essential in the design and implementation of HF recommendations, by Safety Managers, Regulators, and System and Operation designers. The views and opinions expressed in this publication are those of the authors and do not represent the opinions nor position of the SAFEMODE Consortium, or any of the individual partner organizations.
REFERENCES
Carroll, J.M., Olson, J.R., 1988. Mental models in human-computer interaction.
Handb. Hum.-Comput. Interact. 45–65. Daniellou, F., 2021. Taking account of human and organisational factors in planning and designing a high risk system. Jentsch, F., Hitt, J.M., Bowers, C., 2002. 6. Identifying training areas for advanced
automated aircraft: application of an information-processing model, in: Advances in Human Performance and Cognitive Engineering Research, Advances in Human Performance and Cognitive Engineering Research. Emerald Group Publishing Limited, pp. 123–156. https://doi.org/10.1016/S1479-3601(02)02008-8
108 Alarcon and Bieder
Karsh, B.-T., 2009. Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. Rockv. MD Agency Healthc. Res. Qual. 200943. Lee, J.D., Seppelt, B.D., 2012. Human Factors and Ergonomics in Automation
Design, in: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics. Wiley, pp. 1615–1642. https://doi.org/10.1002/9781118131350.ch59 Lehner, P.E., Zirk, D.A., 1987. Cognitive Factors in User/Expert-System Interaction.
Hum. Factors J. Hum. Factors Ergon. Soc. 29, 97–109. https://doi.org/10.1177/ 001872088702900111 Leveson, N.L., Pinnel, D., Sandys, S., Koga, S., Reese, J. “Analyzing software specifications for mode confusion potential.” In Proceedings of a workshop on human error and system development, pp. 132–146. Glasgow Accident Analysis Group, 1997. Moreno Alarcon, D.P., Bieder, C., 2021. Producing Human Factor Recommendations
in the Aviation Sector, in: Arezes, P.M., Boring, R.L. (Eds.), Advances in Safety Management and Human Performance, Lecture Notes in Networks and Systems. Springer International Publishing, Cham, pp. 228–236. https://doi.org/10.1007/ 978-3-030-80288-2_27 Sarter, N.B., Woods, D.D., Billings, C.E., 1997. Automation surprises. Handb. Hum.
Factors Ergon. 2, 1926–1943. Schutte, P.C., 2017. How to make the most of your human: design considerations for human–machine interactions. Cognition, Technology & Work, 19(2), pp. 233–249.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Training, Education, and Learning Sciences
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-35-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional