Terence Andre, Rod Ford, Ryan Meyer, Angelia Sebok (2018). A Performance Support Tool for Human Factors Design and Evaluation. In: Advances in Human Factors, Software, and Systems Engineering. AHFE Open Access, vol. 4. https://doi.org/10.54941/ahfe100122
H-FAST (Human Factors Analysis Support Tool) is a research and development project with NASA to increase human factors awareness among design engineers, facilitate communication between human factors engineers and design engineers, and promote the application of human factors best practices earlier in the design cycle. H-FAST reduces the amount of in-person time needed for evaluations by Human factors specialists by organizing their body of knowledge and presenting it in a way that is usable by engineers throughout the design process. H-FAST also provides detailed guidance regarding human factors evaluations and the capability to store data and provide feedback on the results of these evaluations. H-FAST will improve the engineering design process by providing engineers with easy access to detailed human factors methods, relevant research, and subject matter expert contact information. This will empower engineers to create more usable systems, thus reducing the number of design iterations and resulting in higher-quality products. In this paper, we describe the identification and development of relevant human factors information that was included in the tool, discuss the methods and results of a user test at NASA, and provide an overview of how H-FAST can be extended to other domains such as commercial aviation, homeland security, and product design.
Large-scale systems engineering projects involve multiple teams of engineers working in parallel. These projects typically include numerous human factors challenges, many of which first become evident during the integration stage. Human factors evaluations are essential in gathering human performance data and analyzing the usability of new design concepts. These evaluations are generally carried out by human factors experts due to the level of expertise required. However, many projects may not have a dedicated human factors expert available for consultation throughout all design phases. In these cases, non-human factors engineers could derive value from a tool that locates relevant human factors design resources and guides them through conducting simpler, formative human factors evaluations to gather feedback earlier in the process. The challenge for the design engineer is in finding the appropriate human factors design guidance relevant to a specific project. There is often a considerable body of human factors knowledge available in the form of design standards (e.g., MIL-STD-1472, NASA-STD- 3001, FAA Human Factors Design Standard), handbooks, reference documents, and lessons learned, which have built up over time. Much of this body of human factors knowledge is very useful, but exists in various departments and in different forms (e.g., electronic and hardcopy). In the end, many design engineers are overwhelmed with the corpus of human factors knowledge. In addition, they are not sure of when and how to consult with a human factors specialist, so they often do not get the needed help until later in the design process when components are evaluated in an integrated test environment. This challenge often leads to costly rework.
To help solve this problem, TiER1 Performance Solutions and Alion Science and Technology developed H-FAST (Human Factors Analysis Support Tool). The H-FAST solution was directed at solving the above challenges at NASA where there are hundreds of engineers working on various projects who need foundational human factors content to do early design and evaluation activities. The design and structure of H-FAST allows engineers to (1) determine which program-level human factors requirements apply to their design project; (2) understand how to apply human factors guidelines to their design goals; (3) select, perform, and analyze small-scale human factors evaluations earlier in their design process; and (4) gain access to the NASA human factors body of knowledge in a way that is consistently organized and relevant to their current design issues.
As shown in Figure 1, the system concept for H-FAST is to synthesize the human factors body of knowledge into a catalog of design aids that are searched using a filtering process based on specific design project information. Based on the overall requirements and system concept, our team began the process of prioritizing specific requirements with NASA, iteratively designing the tool, and conducting formative evaluation to improve its usefulness and usability.
Figure 1. The H-FAST concept for cataloging design aids.
TOOL DEVELOPMENT PROCESS Based on initial stakeholder input, our team began to develop an architecture to support the original design aid catalog concept shown in Figure 1. Figure 2 shows the architectural structure for the categorization of design aids (e.g., research documents, subject matter experts, and design patterns) by tagging against multiple taxonomies (e.g., NASA-STD-3001, user tasks, systems area, and design phase). The goal of this tagging was to utilize taxonomies that provide a common language that can be shared between human factors specialists and design engineers. By selecting which tags within the taxonomies applied to their current project, the human factors resources can be automatically filtered to a reduced, highly applicable set.
Once the content architecture was defined, document templates were created for each of the design aid content types. The initial corpus of design aids, and associated tags, was developed from these templates.
Our design of H-FAST focused on providing access to information in one of two main ways as shown in Figure 3: either through a specific project, or through a generic “design aid” repository. H-FAST users can create a project and define the specific characteristics of the system or product they are designing. Based on user chosen design issues (e.g., end user tasks, operating environments), H-FAST directs the user toward certain types of guidance and evaluation methods. In addition to the “project-specific” aspect, H-FAST also includes a diverse repository of human factors guidance, including guidelines, design templates, and detailed methods for performing human factors evaluations. Further, H-FAST offers users easy access to NASA lessons learned, reports and experimental studies, and contact information for human factors subject matter experts. The key benefits of H-FAST are that it provides a tool for supporting human factors in the design process, facilitating communication between engineers and human factors specialists, and keeping records of project-related evaluations and results.
Figure 2: Design project filtering applied to the taxonomy within H-FAST.Figure 3. Overview of H-FAST: Key elements and benefits.
The information model of H-FAST consists of three high-level object types that were modeled in the system. These are described below.
High-Level Object Types There are three high-level object types modeled in H-FAST: Design Aids, Taxonomies, and Design Project Objects. These correspond to the elements in Figure 3 in the following manner: Design Aids object types are “Design Aids” (green box) in the figure. Taxonomies correspond to relationships (red box) among elements in the system. Design Project Objects are a subset of the Project (light blue box) elements in H-FAST.
Design Project Objects Design Projects are used by engineers to collect and organize human factors information as it applies to their component or system design. A project can reference subsets of the taxonomies that apply specifically to their design requirements and challenges. This provides a first-level filter to help engineers narrow their search for relevant Design Aids. In addition, engineers are able to "bookmark" Design Aids within their project, and add some contextual information about how it will be utilized.
H-FAST also allows engineers and HF specialists to perform and document the results of human factors evaluations. All information that is captured as part of the evaluation, and individual findings, are stored within the project as well. Table 1 gives a summary of the high-level object types in H-FAST and how they are utilized.
Table 1: High-level object types and usage Object Type Engineer Usage HF Specialist Usage
Design Aids Design Aids are essentially documents which organize human factors information and best practices. They are typically created and maintained by human factors specialists, and utilized by engineers to gain insight into how they can apply human factors principles to their designs. There are several types of Design Aids, such as a method for performing evaluations, a subject matter expert, or a link to a useful research article. Each Design Aid type has its own template, to ensure information is gathered and presented consistently.
Taxonomies Taxonomies are hierarchically-organized structures of "tags,” which are used to organize and filter Design Aids. Each tree structure represents a classification mechanism for a single aspect of human factors knowledge. Examples of taxonomies include the NASA-STD-3001 guidelines, types of user tasks, and types of NASA engineering systems. Each Design Aid can be tagged (or untagged) with one or more items from any of the taxonomies. This allows complex searching by filtering to view only Design Aids associated with a user-selected subset of taxonomy items. Taxonomies are typically defined during the design of the overall application, as they do not change as frequently as other information. However, system administrators have the capability to add and edit taxonomy items and complete structures as needed.
Principle-Based Guidance It was important that H-FAST provide and recommend NASA-approved procedures and processes for ensuring usable design. To address this concern, the H-FAST team used the NASA-developed Human Interface Design Handbook (HIDH) as the basis for many of the human factors principles content types. Specific methods (e.g., how to conduct different types of usability evaluations) are taken from well-known, widely-accepted human factors methods such as Nielsen and Mack (1994).
In parallel, our team of software engineers developed the functionality to support the user requirements using Microsoft’s ASP.NET and SQL Server. At specific intervals during the development cycle, we reviewed the tool with human factors and design engineers for formative feedback that was used to modify the course of the development. We used formative evaluation techniques where a small subset of representative users were shown early concepts of the system prototype. In formative evaluation, the amount of data collected and the replicability of the data collection methods are less important than the insights gained (Kies, Williges, & Rosson, 1998). Based on several cycles of formative evaluation, we created a fully functional system known as H-FAST 1.0. Figure 4 shows two screenshots of H-FAST 1.0 where a user enters specific characteristics about a particular project (top of Figure 4) and browses design aids associated with that project (bottom of Figure 4). We then used H-FAST 1.0 in a pilot study with an engineering group at NASA.
Figure 4. Screenshots from H-FAST 1.0.
PILOT STUDY The user testing plan included an actual NASA engineering team working with the tool for an extended period of time (e.g., 3 months) and providing feedback on their experiences. The Flight Deck of the Future (F.F) group at NASA JSC was an ideal candidate for this pilot study. The F.F group is responsible for identifying and testing innovative, early design concepts. These typically include numerous human factors concerns, such as wearable computing, robotic system control design, planning and designing stowage locations, and speech-input devices. The F.F group employs engineers and interns who have a technical background but not necessarily education in human factors.
Steps completed as part of the user test included the following: (1) providing an in-person, face-to-face kick-off meeting, in which the users were introduced to the tool and the purpose of and plans for the user test, and (2) providing access to the H-FAST web-based application and technical support to users. Follow-up phone meetings were arranged approximately once every three weeks to gather data from users on their experiences with the tool. We provided the participants with sample questions that were used in the first sets of interviews.
During the course of the user evaluation, one intern left NASA and two new interns arrived. This provided an opportunity to identify the types of problems that new users would experience when using H-FAST. We held a web meeting in which we presented the tool and asked users to perform a few key tasks. These tasks were sent via email just as the web meeting was initiated, so the users did not have a chance to review questions or learn how to perform these tasks before the session. The two new F.F users quickly and accurately performed basic tasks such as entering projects and searching for information.
During the follow-up meetings, we found that users did not typically work with H-FAST on a frequent basis; in fact, the email to arrange a meeting was the prompt that reminded the F.F users to “check” H-FAST. Despite this infrequent use, the H-FAST users did remember how to perform tasks with the tool.
Our main findings to date are presented in Table 2, where several key concerns have been identified.
Table 2: Key concerns identified in the user test Issue Category Examples
As shown in Table 2, some H-FAST users were unable to provide information when they were unfamiliar with specific terminology. Examples include the systems engineering design phases, questions about the eventual use cases, and identifying relevant NASA standards. Second, the material included in H-FAST is not always relevant to the users, either because the methods identified must be performed by human factors specialists, or because the F.F engineers do not find relevant material for their particular project area. In addition, while users understand how to search for information, they do not always understand the results of a search. This problem was related to the links (or “tags”) among the different types of content in the tool and occasional slow system responses. Finally, some users expressed concerns about hyperlinks that did not work and page layout issues (e.g., where headers are visually emphasized over the meaningful content).
In addition to the concerns, users identified several positive features of H-FAST. According to the users, the tool was generally well organized and had an easy-to-understand interface. H-FAST F.F users were able to find information and input their projects. The capability to input a project provided support to educate engineers on a human-centered approach to system design.
When the H-FAST user enters a project, H-FAST provides a form that asks about end users, the environment in which the system will be used, what protective clothing the end users might be wearing, and what tasks the end users will perform. While additional support is needed to help engineers better understand and answer these questions, all users commented that simply presenting these questions will trigger the engineer to think about how the end users will work with the system they design.
F.F users indicated they believe the tool will help engineers better understand and appreciate human factors considerations. In addition, F.F users indicated that tailor-made human factors guidelines, principles, and references (developed specifically for the F.F team) were particularly useful.
The need for custom-created content is an expected issue with any new knowledge management system, but once the content is developed and included, it is available and searchable for any future projects and all future users.
Based on this feedback, the H-FAST team has developed plans forward. Several of the concerns (e.g., non- functioning hyperlinks, page layouts) can be addressed by updating the tool. Similarly, more targeted human factors information has already been provided to meet the requirements of the F.F team. In addition, the questions that F.F users found difficult to answer (e.g., about phases in the systems engineering lifecycle) can be readily addressed by providing text to explain or define these phases.
On the other hand, some issues require further consideration and planning. The issue that H-FAST users might not have a good understanding of the end users' environment and use cases can be addressed through a variety of means. These include providing an additional menu option (e.g., “not yet known”), giving additional information to explain how to learn about the usage environment and define use cases, and providing occasional “reminders” to review these assumptions. Further, the concern that many of the methods identified in H-FAST require a human factors specialist to perform them can be addressed by including direct feedback to contact a human factors specialist and by providing additional resources and techniques that can be implemented by engineers. These types of issues require further consideration and development of a unified approach to ensure that H-FAST adequately and appropriately supports engineers and human factors specialists in developing more usable systems.
FUTURE CAPABILITIES As part of the effort in developing H-FAST, our team is also incorporating additional functionality that will allow users to diagnose usability problems and conduct some evaluations with automated support.
Usability evaluation methods, such as the ones we are including in H-FAST, are designed to help engineers identify design errors and inefficiencies. The usability evaluation process typically involves three steps: (1) presenting an interface and tasks to a prospective user, (2) observing and documenting the issues encountered, and (3) analyzing those documented findings to generate redesign recommendations to refine system designs. Usability analysts attempt to apply this process carefully, in order to ensure that evaluation studies address valid user tasks, elicit evidence of usability failures, and support design recommendations. However, research by John and Marks (1997) has shown that this process alone is not always effective, especially when non-human factors experts do not have the appropriate support to conduct the evaluation. To solve this challenge, our team is enhancing the basic H-FAST capability by developing augmented support for usability analysis, diagnosis, and reporting. Our enhancement work will include refinement and extension of the User Action Framework (UAF)—a theory-based framework of usability concepts for applied usability engineering (Andre, Hartson, Belz, & McCreary, 2001). The UAF supports individual classification by providing a consistent framework that helps usability evaluators identify the true root cause of a usability problem, and therefore helps designers fix the true underlying problem. Key advantages of applying the UAF in H-FAST are that it will support both engineers and human factors specialists in identifying true root causes of usability problems and in developing solutions that actually address those problems.
As NASA engineers develop and implement systems for use on spacecraft, there is a need for consistency and adherence to standards and best-practice human factors design practices across these systems. This situation is typically addressed by having human factors specialists participate on design teams, collaborating with the teams to develop project-specific style guides. H-FAST helps further support the design process from a human factors perspective by providing NASA engineers with access to lessons learned, human factors guidelines, and “best practice” methods. In addition, there is also a need for tools that support designers in predicting potential difficulties and suggesting remedies during the design phase. To support this need, we are integrating AGAT (Automated Guidance Assessment Tool)—an electronic, automated tool, to assess compliance with human factors design principles. AGAT will assess interface design in terms of how well the displays, controls, and layout are expected to support (end-user) operator performance. A likely scenario with AGAT is an engineer stepping through a series of modules, entering information on the current system. This would include the layout of displays and controls; the importance, frequency of use, and sequence of use of displays; the types of controls; and the procedures for interaction. Upon receiving feedback and suggestions from AGAT, the designer might choose to implement these recommendations and re-evaluate, or the designer might decide to develop several potential approaches and compare them.
ACKNOWLEDGMENTS
This material is based upon work supported by NASA under Contract No. NNX11CB40C. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of NASA.
CONCLUSION
H-FAST has been developed to help guide design engineers as they conduct informal evaluations. It is also intended to support collaboration between design engineers and human factors specialists on more formal evaluations. H- FAST incorporates human factors knowledge, resources, and expert input needed to effectively design human- system technologies, saving time by solving problems before they start. Results from our pilot study show that H- FAST is a usable tool and is easy to learn how to use. Input from users has been incorporated into a new version, H- FAST 2.0.
The template-based design approach of H-FAST and associated tagging of key terms allows for easy translation to other organizations with a set of published standards as guidelines. For example, the FAA Human Factors Design Standard (HF-STD-001) is structured such that content could replace what is currently in H-FAST from the NASA- STD-3001. This inherent tool architecture provides the needed flexibility for use across many federal organizations involved in human-systems research, design, and evaluation.
REFERENCES
Andre, T. S., Hartson, H. R., Belz, S. M., & McCreary, F. A. (2001). The user action framework: A reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54(1), 107-136. John, B. E., & Marks, S. J. (1997). Tracking the effectiveness of usability evaluation methods. Behaviour and Information
Technology, 16, 188-202. Kies, J. K., Williges, R. C., & Rosson, M. B. (1998). Coordinating computer-supported cooperative work: A review of research issues and strategies. Journal of the American Society for Information Science, 49(9), 776-779. Nielsen, J., & Mack, R. L. (1994). Usability inspection methods. New York, NY: John Wiley & Sons.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Human Factors, Software, and Systems Engineering
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2096-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional