Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Human Factors in Discovery Phase of TRLs and HRLs

Pacific Northwest National Laboratory, Richland, WA 99354, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1003582

Abstract

With rapid growth in technology, there has been a corresponding growth in research focused on the ways that human-machine interactions can be improved. As part of that work, researchers have explored how human expertise can inform technology design and evaluation. For example, interaction with subject matter experts (SMEs) or end users can help to design and enhance a machine. The human factors of technology release can be divided into five steps: discovery, planning, development, evaluation, and deployment. This framework is a higher-level abstraction of the Human Readiness Levels for technology use and adoption (See et al. 2018). In this exposition, we discuss how human factors methodologies, principles, and practices can be realized in the first phase, Discovery, of the technology development process.

INTRODUCTION

The Technology Readiness Level (TRL) scale is a widely adopted measure- ment system to assess the technological maturity of new equipment, software, applications, and concepts. Many industries and institutions provide gui- des and walk-throughs to aid researchers, developers, engineers, and project managers to ensure meeting project expectations with respect to technology goals. For example, in 2008 the Department of Energy adopted a DOE-wide model that refers to an assessment guide (Frank, 2011) to consistently assess new technologies. Many of these guides neglect to account explicitly for the human component of many new technologies. As a result, the Human Rea- diness Level (HRL) scale was developed and has been recently approved as an American National Standards Institute standard (See et al., 2021; See, 2019). The ANSI/ HFES 400–2021 guide cites the importance of coordinated advancement of technology along both HRL and TRL scales. In addition, the guide maps the HRL’s to existing processes for TRL advancement. In particu- lar, most processes categorize the levels into 3–5 broad areas. Here, we further the work advocating for human-centered technology development and pro- vide another view on technology maturity, guided mostly by the wave of machine learning applications (or promise thereof). Our five-step categoriza- tion (see Figure 1) explicitly marries the technology readiness with the user/ human readiness in the context of a human user working with new (poten- tially intelligent) technologies in a domain where the human component is

© 2023. Published by AHFE Open Access. All rights reserved. 243

244 Jefferson and Baweja

typically the authority. In this exposition, focus on the discovery phase of the process and provide guides for explicitly accounting for human experts while in early stages of identifying technology needs and desires.

HUMAN FACTORS IN DISCOVERY The involvement of subject matter experts and end users in the technology advancement workflow provides enhanced design and performance in mach- ine learning tools (Amershi et al., 2014). When considering the 9-level TRL and HRL scales, there is opportunity for user involvement at each step as (1) an explicit evaluation of the human’s preparedness for new technology and (2) the utilization of human expertise, judgment, and operational knowledge to move technology to more advanced stages. Damacharla discusses the deli- neation of metrics for human-machine teams in terms of human-centered, machine-centered, and teaming-centered metrics (Damacharla et al., 2018). Together with methods and principles from classic human factors research and practices, there is a holistic approach and categorization of techno- logy progression and evaluation. Human factors facilitates reducing human error, increasing productivity, and enhancing safety and comfort with tech- nology (Wickens et al., 2004). At each stage of the technology lifecycle (from discovery to deployment), human factors can serve to improve the efficiency and effectiveness of not only the human-technology interaction, but also the process to see that technology is ultimately adopted.

The human factors of technology deployment can be divided into five steps, displayed in Figure 1. This framework is a higher-level abstraction of the Human Readiness Levels for technology use and adoption, as shown in the lower part of the graphic (See, et al., 2018). Steps in the human factors of

Figure 1: Human factors of technology advancement.
Original PDF Figure 1:

Human Factors in Discovery Phase of TRLs and HRLs 245

technology development include discovery, planning, development, evalua- tion, and deployment. Notably, this framework applies to technology that is intended for operational use, generally within a job, occupation, or task, not those technology projects focused primarily or exclusively on research, because in those projects there is no planned end user. In discovery, human factors methods can help to determine what should be built, how it should work, and the requirements for that technology. In planning, human factors can help to verify that the proposed technology will support the human needs and integrate with the workflow. During development, human factors can be integrated iteratively to ensure that the technology continues to meet human needs as it matures. Similarly, human factors is central to the evaluation process. During deployment, human factors methods can be used to assess, periodically, the success of the technology at meeting potentially changing human needs.

First, it is important to understand the constraints of the environment as well as the current tools, materials, and procedures. The discovery phase provides an opportunity for development teams to conduct an investigation into the kinds of tools and materials that are already used during the task ((Dul et al., 2012) (McDermott et al., 2018)). This information can help guide teams to determine whether their proposed or planned technology is an improvement on the status quo (i.e., a replacement) or supplement to the existing tools. Second, human factors methods in the discovery phase can help to identify the current workflow that exists for a given job or task. The application of human factors methods during the discovery process can help to design a tool that operates well within the current workflow. The task analysis process can also identify challenges faced by a human in a given domain. Human factors techniques can also help development teams to reco- gnize areas within a process or organization that might benefit from change or enhancement (Liebowitz et al., 2000), and thus, contribute to innovation. Third, in addition to understanding the general workflow and task, human factors can also advise on the allocation or assignment of tasks through the requirements writing process. The goal of writing requirements from the task analysis is to clearly and simply state the goals of the users and the ways that the technology will contribute to achieving the goals. Finally, having determined the allocation of tasks and the application of the technology, human factors can help to examine the potential ethical and trust-related considerations of the technology.

Overall, the discovery phase lays the groundwork for ensuring that the development teams understand the challenges faced that technology might solve, the current operations of the end user, and the tools and references that those users already have. This information will help the team to create a better end product that is informed by the needs, operations, and tools that already exist. Thinking in this way will support improved technology and advancing technology maturity; it can also help to avoid investing resources in a technology that will not work for the proposed end users within their current task. To this end we propose a checklist to guide developers and teams in this phase of the advancement process.

246 Jefferson and Baweja

DEVELOPMENT READINESS CHECKLIST The development readiness checklist is designed to be used by the project manager in collaboration with a human factors expert to guide designers (often data scientists) as they complete these discovery-oriented tasks and design their new technology. There are items that development teams should review when determining if they are ready to proceed to the planning and development stages. The four themes of the checklist include components, data collection and task analysis, system functions and requirements, and considerations (see Figure 2); the checklist is also presented in full in Appen- dix A. Each element of the checklist should help the development team to identify gaps in their knowledge and serve as a pointer for more in-depth investigation. Upon completion of the checklist, the team should be prepared to generate a concept of operations that describes the purpose of their pro- posed technology, when and how it will be integrated into a workflow, and the main trust and ethical considerations in its application.

Components Before beginning technology development, it is important that teams under- stand what components of the job might need to be examined during the task analysis phase, including the tools used, procedures followed, or sof- tware available. Many of the potential components of a job are likely to be discussed during a task analysis process and are a key focus of human factors and ergonomic methods (e.g., the physical environment; (Stanton et al., 2013)). The components section of the checklist outlines some boun- daries under which the technology will be operating, including the existing materials, the required processes, and the visibility of that technology to end users. To understand the constraints of the work environment, elements of cognitive work analysis (Vicente, 1999) may be useful, as cognitive work analysis focuses on identifying the boundaries that exist within the work environment.

Figure 2: Elements of the advancement readiness checklist.
Original PDF Figure 2:

Human Factors in Discovery Phase of TRLs and HRLs 247

The components section of the development readiness checklist serves two functions: first, it can help to guide teams as they review task analysis meth- ods to determine which questions might be relevant to ask experts or which activities might be useful to observe. Second, it helps to guide the remainder of the checklist: the subsequent items on the checklist for each element are marked by a tag that indicates the component to which the item refers. If a particular component is not relevant to the technology, it may be ignored.

Data Collection and Task Analysis Considerable attention has been devoted to the ways that systems engineers, data scientists, or machine developers can conduct task analysis to integrate into technology design and development (e.g., (Amershi et al., 2014;Rosson & Carroll, 2002; Seymoens et al., 2019)). In general, these methods strive to understand the nature of expert knowledge, the primary components of task completion, and user or operator workflow in order to design systems that are better integrated into that workflow (e.g., (Annett, 2003; Stanton et al., 2013; Wickens et al., 2004)). There are a number of methods that have been used to contribute to the understanding of human decision-making and, sub- sequently, to inform system design; some example methods are summarized in Table 1. These methods use data collection techniques like observation, interviews, focus groups, or surveys to complete a task analysis. The task analytic methods, in turn, help to generate information that can be used to later describe the responsibilities and roles for humans and technology in the proposed application (as described in the next section of the checklist).

Task analysis and cognitive task analysis techniques are applied to identify components of a task and the cognitive elements (e.g., expertise) required to complete it (e.g., (Annett, 2003; Militello & Hutton, 1998)) by leveraging data collection through interviews, focus groups, or observation. Two com- mon methods of cognitive task analysis include the critical decision method ((Klein et al., 1989); an extension of the critical incident technique; (Fla- nagan, 1954)) and the knowledge audit (Liebowitz et al., 2000; Militello & Hutton, 1998). The knowledge audit was recently adapted for a human- machine teaming (HMT) context, focusing on the ways that task analysis can help to form better human-machine teams (McDermotta et al., 2017; McDermott et al., 2018).

These task analysis methods can support system developers and engine- ers to create machines that work better with human partners (McDermott et al., 2018) (Lyons et al., 2019). However, they can also be fairly extensive in the time commitment required to complete them. For example, when using interviews as the data collection method, McDermott et al. (2018) suggested 90-minute interviews with between three and five SMEs per role or job to understand the nature of the task and the associated knowledge, skills, and abilities of the experts. Of course, practitioners can select the number that they have available and focus their questions to reduce the time required; nonetheless, the need to interview, analyze, and translate the task analysis into requirements can be daunting. It is important to note that SMEs likely vary in their approach to work. Therefore, a larger sample of SMEs (i.e., >2)

248 Jefferson and Baweja

will likely capture important variability in strategies and challenges needed to fully understand the task or work environment under investigation.

In 1998, Militello and Hutton attempted to streamline the process of cognitive task analysis in the Applied Cognitive Task Analysis (ACTA) method (Militello & Hutton, 1998). We revisit this method here because of its practitioner focus; although far more comprehensive guides have been written to support the completion of task analysis, the goal of the checklist presented here is to translate these into elements that system designers can use to determine if the goals of the data collection and task analysis have been met. Briefly, Militello and Hutton (1998) describe three phases to task analy- sis: a task diagram interview, knowledge audit, and a simulation interview. The task diagram phase asks SMEs to generate a broad overview of the task and to identify the difficult cognitive elements of that task. This creates a roa- dmap to the tasks that can guide the later interviews. In the knowledge audit, the primary emphasis is to understand the areas of expertise that are requi- red to complete the task. It is this knowledge audit phase that McDermott et al. (2018) recently expanded to an HMT context, creating an extensive guide for systems engineers and human factors professionals to explore the elements of expertise and knowledge required for a specific domain. In the final phase of ACTA, the simulation interview, the SME is asked to identify major judgments and decisions that are made in the context of a task or inci- dent, such as critical cues, actions, assessments, and potential errors. This helps to provide a picture of how an expert is assessing a situation and where that judgment might go awry.

The second section of the development readiness checklist is intended to support teams as they complete the data collection and task analysis process and begin to develop functional requirements for the technology. The items should help to determine whether they have sufficiently explored the tasks, workflow, and tools of human users to identify the tasks and functions that could be accomplished by the proposed technology.

System Functions and Requirements After completion of the data collection and task analysis, the results will need to be translated into system requirements (McDermott et al., 2018) (Wickens et al., 2004). The purpose of this process is to categorize and understand the information that the SMEs provided to create actionable guidance for the developers. In order to complete this step of requirements writing, it may be helpful to create a task diagram to ensure that the workflow is clear (Stanton et al., 2013) (Wickens et al., 2004)). This helps the development team to identify areas where the human may have challenges or where errors may be more likely to occur. Then, the precise actions for developers can be identified. Some potential methods for mapping the results of a task analysis and using it to inform system requirements are discussed.

Ultimately, task analysis and the requirements writing process helps to define what the machine should do—the function that it should serve. Pro- cess charts and event trees lay out the elements of task completion in a linear flow chart or in a decision tree (Kirwan & Ainsworth, 1992). This helps to

Human Factors in Discovery Phase of TRLs and HRLs 249

visually depict the actions a human is taking and can help development teams to assess at what point during task completion the technology might be used. In allocation-of-function analysis, results of a task analysis are used to deter- mine whether a specific task should be assigned to a human user or to the technology (Philip Marsden, & Kirby 2004). This functional analysis should consider the relative capabilities of the human and the proposed technology to help determine what the technology will do (and thus, the requirements to build it).

Task analysis can also inform a scenario to be used in the design process. In scenario-based design, a story or scenario description is generated to describe a proposed function of the technology (Rosson & Carroll, 2002). This is done during the early phases of development to illustrate how the human will use the technology within their job. This is one method of more interactive design that incorporates the human user into the process to help envision the functions and operations of that technology (an early form of system requirements).

In addition to functional or process maps, one common framework for presenting requirements is user stories, which typically take the form, “As a [type of user], I want to [complete some action] to [achieve a goal]”(Lucassen et al., 2016; McDermott et al., 2018). By translating the inputs from SMEs into requirements, potentially expressed as user stories, the developers can understand what tasks the human and the machine should be completing to achieve task goals. It helps to clarify the capabilities that will be provided and how it will contribute to task performance. In some ways, this might be considered a type of gap analysis, where the developers compare what the experts would like to do with the limits of what is currently possible to identify those areas where a machine could best contribute.

Although not discussed in detail here, in addition to the functional requirements of the planned technology, teams will also need to consider performance requirements (e.g., computing resources, speed) and interface requirements (e.g., the inputs/outputs that need to be displayed). Human factors methods like task analysis and subsequent functional analysis are par- ticularly informative for the functional and interface requirements but may also be helpful for understanding the performance required of the system. For instance, if time pressure is a significant consideration, the system must operate quickly to be useful.

Once the team can clearly state the functional requirements of the propo- sed technology and outline its role within task completion, the final section of the development readiness checklist prompts teams to consider the poten- tial implications for trust and ethics that result from the technology and its application.

Trust and Ethical Considerations Although this has been less elaborated in research, recent work has empha- sized the need to consider ethics in technology development (Smith, 2019). Similarly, Ososky et al. (2013) discuss the need to develop appropriate trust depending on the effectiveness of the technology and the nature of the task (in

250 Jefferson and Baweja

contrast to simply increasing machine trust). Both of these aspects of develo- pment relate to the appropriateness of the task allocation: are the tasks that are planned to be assigned to the technology appropriate ones, and will the human be able to understand when to trust the machine (or not)?

Inherent in this question is a need for the planned tool to consider observa- bility sufficiently so that the human can, in fact, calibrate trust appropriately (Ososky et al., 2013). Put simply, the human must have sufficient insight to know when the technology can be relied on or when it may fail (McDermott et al., 2018). Although this is also an aspect of good design (after develo- pment has begun), it nonetheless represents part of the planning process: the team should consider observability as part of its planning, and only include those features that are observable and explainable to the human, particularly in applications where the result has real-world consequences (McDermott et al., 2018; Smith, 2019)).

In addition to these considerations, Smith (2019) describes some key com- ponents of trustworthy or ethical artificial intelligence, including the ability for humans to monitor performance, override all outcomes, and remain responsible for any impactful decisions. Before beginning development, teams should consider the benefits and damages that might result from the use of the planned technology and consider whether the potential risks outweigh the possible costs (Smith, 2019). Thus, this final section of the development readiness checklist focuses on considering the potential implications of task assignment for machine trustworthiness and ethics.

CONCLUSION

The purpose of this checklist is to provide a roadmap for technology develo- pment teams as they navigate the human factors involved in the technology discovery process to use their resources most efficiently. The checklist is inten- ded to be used by the project manager in collaboration with a human factors expert. The intent is to help the team to effectively utilize methods to inform the development of requirements for the technology. In addition, it provi- des some important considerations regarding trust and ethics that can be addressed prior to beginning the planning and prototyping process.

Our checklist is proposed to aide in understanding the nature of exi- sting tasks, the overall workflow, and the skills necessary to complete tasks successfully. While machine teammates are often desired, researchers have highlighted that the integration of algorithms into human workflow remains a substantial challenge to AI: developers may not fully consider the impact that the AI has on human processes, particularly if the end users are alre- ady under substantial cognitive load (e.g., in healthcare settings; (Asan & Choudhury, 2021)). These challenges are not unique to AI, but they may be exacerbated by it; as the technology becomes increasingly opaque, com- plex, and autonomous, the importance of understanding the human tasks and workload increases.

It must be stressed that integration of the human users into technology advancement is not complete after the discovery phase has ended. Inte- ractions with the end users of a proposed technology should be iterative;

Human Factors in Discovery Phase of TRLs and HRLs 251

development teams should continue to vet the proposed technology to ensure that it will be functional and useful for the proposed application. Human users are critical to discovery, planning, development, evaluation, and deploy- ment; the application of human factors methods to understand those users is a critical part of increasing the likelihood of successful technology deployment.

APPENDIX

Does the domain involve this Component? Responses cpnt. Item (Y) (N) Not Sure

Job The technology will be deployed within a specific job (i.e., occupation) or role. Materials and References

There are materials or references that are used in the job (e.g., manuals). Procedures There are procedures/ processes that apply to the job, role, or task. Physical Environment

The job, role, or task being undertaken by the human and technology involves a physical environment with characteristics that might impact the technology functionality. Software There are software tools current deployed in the job, role, or task. Visibility The human user will be aware of the technology and its role in task completion. Ethics The job, role, or task may have an impact on risk, safety, or other human outcomes.

Can you answer yes to these? Responses cpnt Item (Y) (N) N/A

Job There is a clear understanding of the current workflow of humans in the job, and that workflow is generalizable to more than one user (i.e., is not idiosyncratic). Job The major tasks and steps have been decomposed can be clearly stated. Job Major challenges and limitations in the workflow have been identified and considered — for example, equipment difficulties, challenges interpreting or receiving information, or anomalies that arise during task completion. Materials and References

Documentation or references that are used in the job have been identified and reviewed. Procedures The procedures used in task completion have been identified and are understood. Physical Environment

There is a clear understanding of the nature of the physical environment where the machine will be deployed.

252 Jefferson and Baweja

Can you answer yes to these? Responses cpnt Item (Y) (N) N/A

Software Any software used in the job has been identified and reviewed to understand its functions and purpose. Job It is clear how and when the technology will be incorporated into the human workflow. Job It is clear what tasks the human user and the technology will be accomplishing. Job The technology fills a specific need or role within the completion of the task. Job The roles of the technology and the human user are well-defined. Job The tasks that the technology will be completing can be clearly and simply articulated. Materials and References

Gaps in the tools and resources available to the human user have been identified. Physical Environment

There is an understanding of the physical environment where the technology will be deployed. Software Benefits and limitations in the current software available have been identified and are understood. Job It is clear when the technology can be relied on to complete a task and when the human will need to be involved in task completion. Visibility There will be sufficient insight into the system functions that trust can be calibrated appropriately. Ethics The human knows when to trust the technology and when not to do so. Ethics The human user is able to monitor and control potential risks. Ethics The tasks that are planned to be assigned to the human user and the technology are appropriate—that is, humans are responsible for decisions that impact other humans.

ACKNOWLEDGMENT

The authors would like to acknowledge Dr. Corey Fallon for discussions.

REFERENCES

Amershi, S., Cakmak, M., Knox, W. B. & Kulesza, T., 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine, 35(4). Annett, J., 2003. Hierarchical Task Analysis. In: E. Hollnagel, ed. Handbook of

Cognitive Task Analysis. 1 ed. Boca Raton: CRC Press, pp. 17–35. Asan, O. & Choudhury, A., 2021. Research Trends in Artificial Intel- ligence Applications in Human Factors Health Care: Mapping Review. JMIR Human Factors, 8(2). Damacharla, P., Javaid, A. Y., Gallimore, J. J. & Devabhaktuni, V. K., 2018. Common Metrics to Benchmark Human-Machine Teams (HMT): A Review. IEEE Access, Volume 6. Dul, J. et al., 2012. A strategy for human factors/ ergonomics: developing the

discipline and profession. Ergonomics, 55(4), pp. 377–395.

Human Factors in Discovery Phase of TRLs and HRLs 253

Flanagan, J. C., 1954. The Critical Incident Technique. Psychological Bulletin, 51(4), pp. 327–358. Frank, M., 2011. DOE G 413.3–4A Chg 1 (Admin Chg), Technology Readiness

Assessment Guide, s.l.: U. S. Department of Energy. Kirwan, B. & Ainsworth, L. K., 1992. A Guide to Task Analysis. s.l.: Taylor & Francis. Klein, G. A., Calderwood, R. & MacGregor, D., 1989. Critical decision method for

eliciting knowledge. IEEE Transactions on Systems, Man, & Cybernetics, 19(3), pp. 462–472. Liebowitz, J. et al., 2000. The Knowledge Audit. Knowledge and Process Management, 7(1), pp. 3–10. Lucassen, G., Dalpiaz, F., Werf, J. M. V. D. & Brinkkemper, S., 2016. Improving

agile requirements: the Quality User Story framework and tool. Requirements Engineering, Volume 21, pp. 383–403. Lyons, J. B., Wynne, K. T., Mahoney, S. & Roebke, M. A., 2019. Trust and Human- Machine Teaming: A Qualitative Study. Artificial Intelligence for the Internet of Everything. Marsden, P. & Kirby, M., 2004. Allocation of Functions. In: Handbook of Human

Factors and Ergonomics Methods. Boca Raton: CRC Press. McDermott, P. et al., 2018. Human-Machine Teaming Systems Engineering Guide, Bedford, MA: MITRE Corporation. McDermott, P. L. et al., 2017. Quenching the Thrist for Human-Machine Teaming

Guidance: helping Military Systems Acquisition Leverage Cognitive Engineering Research. Bath, UK, s.n. Militello, L. & Hutton, R. J., 1998. Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), pp. 1618–1641. Ososky, S., Schuster, D., Phillips, E. & Jentsch, F., 2013. Building Appropriate

Trust in Human-Robot Teams. s.l., Association for the Advancement of Artificial Intelligence. Rosson, M. B. & Carroll, J. M., 2002. Usability Engineering: Scenario-Based Development of Human-Computer Interaction. s.l.: Morgan Kaufmann. See, J. E. et al., 2018. Incorporating Human Readiness Levels at Sandia National

Laboratories. Journal of Human Performance in Extreme Environments, 14(1). See, J. E. et al., 2021. ANSI/HFES 400-2021, Human Readiness Level Scale in the System Development Process, Washington DC: Human Factors and Ergonomics Society. See, J. E., 2019. Convincing Systems Engineers to Use Human Factors During Process

Design. In: T. Z. Ahrem, ed. Advances in Artificial Intelligence, Software and Systems Engineering. Cham: Springer International Publishing, pp. 133–145. Seymoens, T. et al., 2019. A Methodology to Involve Domain Experts and Machine Learning Techniques in the Design of Human-Centered Algorithms. s.l., Springer, Cham. Smith, C. J., 2019. Designing Trustworthy AI: A Human-Machine Teaming Frame-

work to Guide Development. s.l., Software Engineering Institute. Stanton, N. A. et al., 2013. Human Factors Methods: A Practical Guide for Engineering and Design. 2 ed. London: CRC Press. Vicente, K., 1999. Cognitive Work Analysis: Toward Safe, Productive, and Healthy

Computer-Based Work. 2 ed. s.l.: CRC Press. Wickens, C. D., Gordon, S. E., Lee, J. D. & Liu, Y., 2004. An Introduction to Human Factors Engineering. 2 ed. s.l.: Pearson Prentice Hall.

Cover of the proceedings volume Human Factors and Simulation

Explore the complete digital volume

Human Factors and Simulation

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-59-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →