Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Behavior-based performance optimization in military training environments

Design Interactive, Orlando, FL 32817, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1002420

Abstract

The quality of instruction Warfighters receive can substantially influence mission readiness. As such, it is important that military instructors develop and maintain instructional competencies to adequately promote relevant knowledge and skills among trainees. However, unlike K-12 educators, active-duty military instructors are not often provided opportunities for formal instructional training; that is, training on how to teach (Malone, Vogel-Walcutt, Ross, & Phillips, 2014). While their content, domain knowledge, and leadership skills are strong, military instructors lack “expert instructor techniques” and skill in implementing useful strategies to convey their knowledge to trainees. They are typically not afforded time and resources to develop these skills prior to taking charge of training new members of the Armed Forces. There was a need to design training tools for accelerating mastery among military instructors via rapid acquisition of high-quality instructional skills to ensure effective learning and retention among trainees.To address this challenge, Design Interactive developed an adaptive training tool, the Interactive Military Instructor Training and Assessment Technology (IMITATE). There was a focus on incorporating research-based training interventions such as video self-modeling, prompting, performance reflection, practice and feedback to support accelerated mastery of observable skills. The goal of the IMITATE program was to optimize the learning experience for military instructors with a tool that they could utilize to sharpen and receive interactive feedback on the skills that are crucial to being a good instructor. The system implements customizable rating systems made up of key performance areas (KPAs), broken down into observable behaviors with detailed rating anchors. This creates a competency- based approach for instructors, who through IMITATE, can receive structured, personalized feedback to quickly bridge performance gaps. IMITATE utilizes a three-stage approach for training – Prepare, Practice, and Assess. The Prepare stage provides introductory video-based observation and training that identifies the KPAs and behaviors and illustrates expert performance in the form of lessons. In the Practice stage, instructors practice their skills while receiving standardized, behavior-based assessment through an intuitive rating tool that provides observers with guidance on how to rate, promoting consistency between observers. The final stage, Assess, is where results are captured and displayed for immediate, actionable, structured and personalized feedback. A video-based after-action review offers session playback paired with time-synced ratings to highlight effective or ineffective performance. Built-in analytics enable performance comparisons between individuals and groups over time. Usability for IMITATE is consistently being assessed and refined based on end-user feedback. This iterative, user-centered design process enables a learning tool that is easy to use and applicable across a variety of domains. Integrating these research-based training interventions with a user-centered design process enabled the Design Interactive team to build upon existing military training protocols to enhance instructor competence, reduce instructor development costs, and improve mission readiness. This method is currently informing additional R&D efforts across the Marine Corps and Air Force and has been tested across a wide range of use cases from military, medical, academic, and industrial settings.

© 2022. Published by AHFE Open Access. All rights reserved. 345

INTRODUCTION TO THE MILITARY TRAINING ENVIRONMENT AND PROBLEM The training that warfighters receive is crucial to their preparedness on the battlefield and can substantially influence mission readiness. Therefore, it is important that military instructors develop and maintain instructional com- petencies to adequately relay their knowledge and skills to their trainees. Unlike professional educators, active-duty military instructors are not often provided opportunities for formal instructional training, or training on how to teach (Malone, Vogel-Walcutt, Ross, & Phillips, 2014). Military instru- ctors are highly specialized and knowledgeable in the subjects they instruct; however, new instructors often lack expert instructor techniques and skills in implementing useful strategies to convey their knowledge to trainees. Oppor- tunities for self-directed training, as well as occasions to practice instructional techniques, are limited. There is a need to accelerate novice instructor skill development to ensure their trainees receive the best possible instruction.

These challenges provided an opportunity for the development of a trai- ning tool to benefit the instructional skills of military instructors. The desired tool would accelerate mastery among military instructors via rapid acqui- sition of high-quality instructional skills to ensure effective learning and retention among trainees. The tool must demonstrate an increase in an instru- ctor’s performance over time, while providing them accurate feedback they can use to address any skill deficiencies. The tool must be simple and intui- tive in terms of its usability and versatile enough to be applied in different instructional instances. To meet those needs, Design Interactive develo- ped the Interactive Military Instructor Training and Assessment Technology (IMITATE).

INTRODUCTION TO IMITATE IMITATE is a research-based adaptive training tool that provides persona- lized feedback to the instructor. The goal of instructors using the IMITATE platform is to accelerate acquisition of their instructional capabilities from a novice level to an expert level. IMITATE takes a three-stage approach to trai- ning – Prepare, Practice, and Assess. The Prepare stage provides instructors with lessons on the skills they are trying to develop and master. These lessons come in the form of document (e.g., Word, PDF, PowerPoint) or video-based observation lessons which highlight the desired instructor behaviors and pro- vide examples of what expert performance looks like for each behavior. The Practice stage is where the instructor practices their instructional skills live while they receive standardized, behavior-based assessment through the IMI- TATE rating tool by an observer. In the Assess stage, instructors can review the results of their practice session. The results displayed are personalized, structured, and allow for quick feedback on performance. Performance data are viewable in multiple ways, including graphs and charts that track overall performance and performance over time. If a video was recorded of the pra- ctice session and uploaded to IMITATE, a video-based after-action review is readily available. This feature offers a playback of the practice session,

Behavior-Based Performance Optimization in Military Training Environments 347

with timestamped ratings and comments on the video timeline so the instru- ctor can see what they were doing when they received a specific rating or piece of feedback. IMITATE also includes built-in analytics to track indivi- dual or group performance or make comparisons across different instructors or groups over time.

Research-Based Training Interventions IMITATE is built on research-based training interventions to support the mastery of observable skills. Some of these interventions include video self- modeling, self-reflection, and personalized feedback. These are all evidence- based methods that have been applied in conjunction with the IMITATE software to provide instructors with a high level of instructional skill training and retention.

IMITATE enables evaluators the ability to provide instructors with speci- fic and actionable feedback on performance. This is accomplished through the inclusion of customizable rating systems made up of Key Performance Areas (KPAs), which are broken down into observable behaviors. Each beh- avior is rated on a five-point scale and is further broken down into detailed rating anchor descriptions to facilitate accurate selection of ratings for each behavior the instructor exhibited during their evaluated session. The rating anchors help new raters understand how a rating of a selected value is defi- ned by describing what the behavior looks like at each value. The consistent definition of the rating values for each behavior improves both inter-rater and intra-rater reliability. Ratings are typically conducted on a tablet with IMITATE loaded onto it, or a tablet that is connected to an IMITATE server, by an observer who is watching the instructor live. Alternatively, a video of the instructor may be uploaded to IMITATE and then rated by an observer later. An example of the live rating interface is shown in Figure 1.

At the conclusion of rating sessions, instructors are provided personali- zed feedback on their performance. Performance data is displayed in various ways. Summary data shows average performance across all KPAs and their respective behaviors (see Figure 2). IMITATE highlights the areas where grea- test improvement is needed (lowest three rated behaviors) as well as the areas of highest performance. Tips or reflective questions, adapted based on perfor- mance results, are provided to help improve performance on the lowest rated behaviors. The user also has access to various graphs and charts which display their average performance or performance over time for selected KPAs and behaviors. Instructors may reflect on their personalized feedback, as this is a constructive way they can learn from their own shortcomings.

Video self-modeling is a training technique where individuals record them- selves performing an activity. Studies have demonstrated that individuals benefit from observing their own behaviors (Dowrick, 1999). One of the fea- tures of IMITATE is the option to upload a video. An instructor may upload a video of them practicing their classroom teaching skills and an expert can rate their performance. Or if a person is rated live while being recorded, the video and timestamped ratings can be paired at the conclusion of the session. The result is a video-based after-action review that enables the instructor to

Figure 1: Imitate live rating interface.
Original PDF Figure 1:

see what they were doing at the time they received a given rating and/or com- ment from the evaluator (see Figure 3). The instructor can use this video for self-reflection and as a learning tool to understand what they did well and where they need to focus on improvement.

DEVELOPMENT AND USABILITY OF IMITATE User-centered design is a key component of the success of IMITATE. Fre- quent and efficient collaboration occurred amongst project team members, stakeholders, and potential end users. Each feature or aspect of IMITATE was designed with the user as the focal point.

IMITATE was conceptualized using only preliminary, limited input from end users. However, as soon as the core development effort began, stakehol- ders and end users became heavily involved in the process, and understanding user needs and prioritizing the User Experience (UX) became a primary area of focus. Prior to and during design and development, many interviews were conducted to gather input on desired features, interface design concepts, and the overall user experience. This information was translated into user stories which led to the creation of wireframes. Rapid prototype development was used to quickly develop functional prototypes which could be used for itera- tive evaluation. Evaluations were conducted with different groups of users that included internal personnel, Marine Corps instructors, and industry partners. Internal assessments conducted included cognitive walkthroughs of wireframes, heuristic evaluation of interfaces using Nielsen’s 10 Usabi- lity Heuristics, formal reviews of interface concepts, and formal reviews of

Behavior-Based Performance Optimization in Military Training Environments 349

Figure 2: Summary session results.
Original PDF Figure 2:

functionality. Reviews were held with personnel of various Marine groups to gain insight on specific use cases and to discuss the needs of the different organizations (Campbell, 2018). Topics discussed included understanding the different types of users and how each would interact with the tool, where the software would be used and with what hardware, preferred interface layouts, preferred display of results, pain points in their current processes that the software could address, desired features, and any limitations or restrictions that may affect their use of the software. Marine instructors from multiple schoolhouses were given demos of the IMITATE prototype and, as prototy- pes were developed, they were able to interact with the tool and evaluate the features and functionality. Multiple pilot tests were conducted. Similarly, industry partners received IMITATE demos and contributed to interface revi- ews and pilot tests the system. The early user needs analysis formed the basis for the initial development plan, which was then modified throughout deve- lopment, as were interface designs, based on the continual evaluation and feedback cycle. Even now that IMITATE is in use at schoolhouses, feedback

Figure 3: Video-based after-action review.
Original PDF Figure 3:

is regularly gathered to build the backlog of desired additional features to be implemented in the future.

A strong foundational understanding of user needs and a user-centered design process are crucial for driving the successful adoption and utility of a product. Often, there is a disconnect between business decisions and what developers believe is desired and the real-world use cases and user needs. Bridging this gap allows for smooth product development without having significant rework. Products developed with strict user testing and usability guidelines will save both time and money and ensures the product will be successful when it reaches the end user. The significantly favorable response and interest that has been received for IMITATE is an example of how a focus on user needs and usability can lead to enthusiastic adoption of and desire for a tool, which will contribute to the software’s goal of promoting accelerated instructor skill mastery, leading to better trained warfighters.

IMITATE AS A TOOL FOR MILITARY INSTRUCTORS The IMITATE software has undergone multiple pilot tests that have ranged from single day events to multi-year evaluations. The software has been used at seven United States Marine Corps (USMC) schoolhouses, one Air Force Base, and at instructor workshops, as well as during USMC field exercises. All pilots were kicked off with a short PowerPoint presentation and then operated with little to no support, providing a strong indicator that the user- centered design process was successful in producing an intuitive, easy-to-use tool. While IMITATE has primarily been used for instructor development, it

Behavior-Based Performance Optimization in Military Training Environments 351

has also been used to assess students or individuals in other contexts. Alth- ough work to roll out the software on a larger scale and to more locations is continuing, IMITATE is currently in ongoing use at five different USMC schoolhouses.

A formal effectiveness evaluation has not yet been conducted to measure the degree to which instructor mastery is accelerated with the use of IMI- TATE. However, feedback received indicates instructors are benefiting from the use of the software in multiple ways. In terms of improving the skills of new instructors, users have stated that the structured and detailed feedback provided helps them gain a good understanding of where they are on the mastery continuum (from novice to expert) in the different KPAs for instru- ction and helps them to identify their strengths and weaknesses much more clearly than the typical feedback they receive. Additionally, novice instructors felt the video-based after-action review to be highly effective at helping them identify what and how they needed to improve, since they could see how they were exhibiting behaviors paired with the ratings and feedback. They felt the additional visual observation component was much more powerful than ver- bal feedback alone (e.g., it is one thing to be told that you are fidgeting too much, but it is more impactful to see how distracting your fidgeting truly is). Further stated benefits resulting from use of IMITATE by senior instru- ctors who serve as evaluators include decreased time and cost to create and conduct evaluations, enhanced performance data, and ability to easily track performance over time.

CONCLUSION

It is important for military instructors to develop and maintain instructio- nal competencies so they can successfully transfer their expert knowledge on to their trainees in the classroom or field. IMITATE is a customizable tool that provides standardized, behavior-based assessment and personalized feedback, supporting instructional skill development for military instructors. IMITATE was developed with a focus on the user, resulting in a positive user experience and leading to enthusiastic adoption and ever-growing inte- rest in larger-scale transition. The software continues to undergo testing and evaluation for further improvements as more features are added. The versatility of IMITATE to be applied in a variety of settings is beneficial in solving different challenges and supporting numerous use cases, inclu- ding many contexts outside of instruction. Overall, IMITATE represents a user-centered tool that will benefit military instructors and other indivi- duals as it continues to be applied across a variety of military and other settings.

ACKNOWLEDGMENT

This research and development were sponsored by the Office of Naval Research under contracts N00014-17-C-1031 and N00014-21-C-2037.

Disclaimer: Presentation of this material does not constitute or imply its endorsement, recommendation, or favoring by the U.S. Navy or the Depar- tment of Defense (DoD). The opinions of the authors expressed herein do not necessarily state or reflect those of the U.S. Navy or DoD.

REFERENCES

Campbell, G. (2018). “Usability and Subject Matter Expert Evaluation Report.”

IMITATE Program Technical Report for ONR Contract Number N00014-17-C- 1031. Dowrick, P. W. (1999). A review of self modeling and related interventions. Applied and Preventive Psychology, 8(1), 23-39. Gün, B. (2011). Quality self-reflection through reflection training. ELT Jour-

nal, 65(2), 126–135. Malone, N., Vogel-Walcutt, J., Ross, K., & Phillips, J. (2014). “Literature review of competencies and standards for instructor development.” Program Technical Report for ONR Contract Number N00014-14-C-0106. Zlatkin, A., & Campbell G. (2021). “Pilot Evaluation Report and Resulting Software

Modifications and Requirements.” IMITATE Program Technical Report for ONR Contract Number N00014-17-C-1031.

Cover of the proceedings volume Training, Education, and Learning Sciences

Explore the complete digital volume

Training, Education, and Learning Sciences

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-35-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →