Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Fun as a Strategic Advantage: Applying Lessons in Engagement from Commercial Games to Military Logistics Training

Massachusetts Institute of Technology Lincoln Laboratory Lexington, MA 02421, USA ↗ ↗
Publication year
Digital Object Identifier10.54941/ahfe1002399

Abstract

Games have been identified as a potential solution to improving learning outcomes in educational settings. Game environments offer many elements to augment traditional classroom learning such as lectures and static reading assignments. They enable players to explore concepts through repeat play in a low-risk environment, and can integrate feedback into gameplay to enable students to evaluate their own performance. Commercial games leverage a number of features to engage players and hold their attention; they typically use enticing graphics and visual elements, and break game play down for new players. But do those methods have a place in instructional environments with a captive and motivated audience? Our experience and measures suggest that yes; applying lessons in engagement from commercial games can help students become more invested in their learning. Though the military may not prioritize fun, they are interested in leveraging potentially effective training methods.MIT Lincoln Laboratory worked with the Office of Naval Research Global TechSolutions (ONR Global TechSolutions), the Marine Corps University Expeditionary Warfighting School (MCU EWS), and the Marine Corps Expeditionary Energy Office (E2O) to develop an interactive, web-based serious game prototype that teaches the principles of logistics and their trade-offs. Developed from a proposal by Marine Corps Captains, the game’s overarching objective was to improve the education and training of Marine Corps University students on the topic of energy management and logistics. Throughout development, MIT LL conducted game assessments at regular intervals, both with internal personnel and Marine Corps University students to validate project goals and guide development. A final test was conducted at the conclusion of development to measure usability against earlier results to measure learning outcomes, and examine the impact of engagement on learning outcomes as well as user reported experience. The game was tested with 12 students and 4 non-student personnel, who represented a mix of operations, logistics, and other disciplines. Students were split between “engaged” (7 students) and “de-engaged” conditions (5 students), where the “de-engaged” condition replaced introductory movies with equivalent static content, and removed decorative elements. Game rules, game information, user support information, and user workflows did not change between the conditions. Though testing was conducted throughout with a relatively small sample size, qualitative and quantitative measures suggest results relevant to how game-based and digital learning tools are designed. Reported usability increased considerably throughout development with less coaching and support, including during development phases focused almost exclusively on improving engagement and applying lessons from entertainment games. In the final assessment, those in the “engaged” condition reported higher usability scores as expected, but also reported making less mistakes and finding play easier. Additionally, those in the “engaged” condition reported finding stronger connections between the principles of logistics presented, indicating that there is a connection between engaging features and learning outcomes. Though more research is needed to see if results hold up more broadly, these results indicate that the integration of engaging features can improve engagement and perception as well as potentially improving learning outcomes even with communities that may not traditionally prioritize engagement.

INTRODUCTION

Military logistics planning (getting necessities such as food and fuel to ope- rational forces) has long been a focus of military strategists, educators, and historians alike (Glick and Charters, 1983). Current teaching methods include classroom lectures and discussion, reading of historical impacts of logistics plans, and thought exercises played out on paper. Many members of today’s Marines play engaging digital games in their personal lives. Three such Marine Corps Captains saw potential in bringing the capabilities of digital games to bear on their logistics training at the Marine Corps Univer- sity (MCU). The immediate feedback offered by a digital tool over multiple plan iterations would be a significant improvement to the qualitative feed- back received, sometimes weeks later, from in-class exercises with a single logistics plan.

Gamification also has a long history of being studied as a potential lear- ning tool. Research indicates that gamifying learning goals does help increase student knowledge, while pointing to some aspect of external motivation (as opposed to intrinsic motivation) that is a factor (Buckley and Doyle, 2016). Providing the ability to self-test, or to allow players to gauge and improve

© 2022. Published by AHFE Open Access. All rights reserved. 180

Fun as a Strategic Advantage: Applying Lessons in Engagement 181

on their own performance using digital games has been shown to be an effe- ctive strategy (Denny et al, 2018), highlighting the potential for scoring and meeting scoring thresholds to provide external motivation. Adapting trai- ning materials to consider the learner’s current knowledge and bridging the gap between novice and expert is also a core component to facilitate lear- ning (Case and Bereiter, 1984), particularly where learning environments are tailored to be applicable to actual environments (Blumenfeld, Kempler, and Krajcik, 2006).

Gaming is not a new concept for the military. War games that would be recognized by today’s military were invented by Prussian Army officers in the nineteenth century (Glick and Charters, 1983). Modern war games range from strategy exercises on paper, to activation of military equipment and practice of equipment maneuver in the field. Though they may offer simi- lar lessons in strategic thinking offered by some commercial games, they are a serious study of tactics and readiness, and are focused on the application of official processes and procedures. Many of these games require manual tracking of progress, and qualitative analysis of results. “Fun” in the sense traditionally meant for popular games is at best a low priority. However, some individuals and agencies throughout the military are aware of the promise of digital gaming environments to build intuition and improve learning, with resulting research pointing to the potential of integrating artificial intelligence into war games to improve planning (Allen and Chan, 2017). Serious games embed as much of actual decision-making and operational environments as can be supported, with a focus on balancing game abstraction with real- world fidelity. Commercial games intended for smartphone or tablet use are typically rapid-play games aimed at novice gamers. Our team integrated aspe- cts of commercial rapid-play games into a serious game environment, and found evidence that these commercial engagement elements can positively affect learning outcomes.

CASE STUDY: MCFLIE Over the course of a year, our team built the Marine Corps Field Logi- stics Instructional Environment, (MCFLIE, pronounced “mick-fly”), which is a browser-based gaming environment that tasks players with following a Marine Corps Expeditionary Unit through a multi-stage operation. We wor- ked to develop a widely accessible digital game that would allow logistics students at MCU to investigate the principles of logistics and their trade-offs as described in Marine Corps doctrine (US Marine Corps, 2016), to align with their coursework. All MCU students are military personnel, but they range from having limited knowledge or understanding of military logistics, to being responsible for logistics planning for their unit. Players are given an operational plan they can investigate. The game models a set distribution network (roads) that varied in distance, quality, and security, which would result more or less fuel and ammunition use. Each player’s goal is to place and direct logistics assets to supply operational troops at various nodes with water, food, ammunition, and fuel for the duration of a multi-day mission. Players are given access to details about how much each asset can carry, and

Figure 1: Screenshot of segments of gameplay, showing a logistics plan at left and the logistics scorecard screen at right. Players are expected to flip between the two screens to gauge their progress and assess their plans.
Original PDF Figure 1:

how quickly each unit uses its resources. A digital scorecard is accessible at any point in game play, allowing players to try out strategies and iterate in near-real-time (Figure 1). Our goal was to provide a gaming environment that would take students less than an hour to learn and two hours to play, to better focus their limited time on experimenting with logistical trade-offs.

APPROACH AND EXPERIMENTAL DESIGN We echoed game build processes from the development of rapid-play games to evaluate technology (Seater, Kurucar, Uhmeyer, 2019). Initial development included a multi-day session with four Marine Corps logistics experts to work through a simple operational scenario to identify core components of logi- stics support. Access to planning tools allowed our team to integrate realistic logistic support details, to align with recommendations from Blumenfeld, Kempler, and Krajcik (2006). Based on a meeting with MCU logistics instru- ctors, we developed a scorecard focused on the principles of logistics (US Marine Corps, 2016), allowing students to gauge their own performance (Denny et al. 2018).

Usability considerations were focused on turning gaming and logistics planning novices into confident performers. We leveraged the tiered intro- duction to content found in commercial games aimed at novices, consistent with recommendations from Case and Bereiter (1984) and Alekseyev (2021). We also developed a basic scenario and corresponding step-by-step tuto- rial for students to play through, which introduced initial game mechanics though a focus on the most direct logistics principle (Attainability, or ensuring troops have enough supply). Other major aspects we adopted from comme- rcial games were the simplification of user workflows, engaging graphics, and easily digestible in-screen information served when needed. To aid in fidelity and immersion (Sailer and Homner 2020), introductory cut scene segments were created introducing players to the scenario they would be playing through.

Throughout development, MIT LL conducted game assessments at regu- lar intervals, both with internal personnel and MCU students. As shown in Table 1, the first major test with students was conducted onsite MCU at

Fun as a Strategic Advantage: Applying Lessons in Engagement 183

Table 1. Tests conducted with digital game, with number and type of participants.

Table available in the original PDF paper.

Month Session Type Participants MCU students (12)

5 Onsite MCU, Individual play Verbal tutorial, full scenario

Non-MCU military cadets (3)

10 Virtual, Individual play Click-through tutorial, basic scenario

12 Hybrid, Individual play Click-through tutorial, basic scenario

MCU personnel (16) 7 students, 4 other – Engaged 5 students – De-Engaged

Figure 2: Selection of screens used to support month 12 validation testing. From left, introduction, scenario selection, logistics planning screens, and scorecard. “De-Engaged” interface screens at top, and “Engaged” equivalent screens at bottom.
Original PDF Figure 2:

month five. In this session, participants were given a survey, then gameplay was introduced by a team member talking through a scenario while partici- pants watched. Next, participants were able to play through on their own and ask the development team questions. They played an early digital pro- totype running the full scenario without an integrated scorecard, and took a second survey prior to the conclusion of the session. In month 10, participants were provided with video overviews and a click-through tutorial in lieu of demonstration and accessible development team, then played through a basic scenario that matched the tutorial with a functional scorecard to reference.

Final validation testing was conducted remotely in month 12. The game was tested with 12 MCU students and 4 non-student MCU personnel. We shifted to a between-subjects design, where students were split between “Engaged” (7 students, 4 other MCU personnel) and “De-engaged” conditi- ons (5 students) to test the relative impacts of commercial game engagement features. Students signed up for one session, where each session was assi- gned one condition to eliminate the possibility of seeing both conditions. As shown in Figure 2, the De-Engaged condition removed movies, anima- tion, and visual polish. Game rules, information, and user workflows did not change between the conditions. We also added a paired comparison tool to the pre-and post-survey to assess learning outcomes (Cooke et al. 2003).

Figure 3: Bar chart with variance comparing aggregate reports from tests conducted (left to right) months 5, 10, and 12 (validation testing). Bars without variation in month 10 reports indicate a unanimous rating; month 12 scores are for the Engaged condition participants only.
Original PDF Figure 3:

Sessions were two hours long, and participants were given a $25 gift card for their participation.

Testing received approval from MIT’s Committee on the Use of Humans As Experimental Subjects, with concurrence from the Air Force Medical Readi- ness Agency Human Research Protection Official (HRPO), the Marine Corps HRPO, and the Office of Naval Research HRPO.

RESULTS

We compared participant ratings at months 5, 10, and 12 to investigate the impacts of game improvements (Figure 3). During each session, participants were asked to rate each question from 1 – Strongly Disagree to 5 – Strongly Agree. Looking across aggregate numbers for each session, replacing verbal overviews and development team questions with a simple scenario and a step- by-step tutorial improved overall reports on all measures (month 5 compared to 10). Month 12 shows similar improvement to month 5, though with more variation between participants. User interface scores were similar to Rules and Mechanics scores in month 5, but consistently trend higher in months 10 and 12. When we look at just the aggregate results between Engaged and De- Engaged conditions during month 12 testing (Figure 4), reported Rules and Mechanics scores between the two conditions were similar. One notable exce- ption was “7. The rules did not cause mistakes”, which was rated much lower by participants in the De-Engaged condition. Participants in the De-Engaged condition additionally reported scores noticeably lower User Interface scores across the board, with one particularly large drop reported for “11. I was satisfied with how easy the game was.”

Finally, we used paired comparisons in a grid format to assess participant learning outcomes, where the principals of logistics with definitions were listed in the first row and column (Cooke et al, 2003). Participants were asked to rate similarity of each concept pair in the corresponding cell on a scale of 1 – Not Related to 8 – Extremely Related at the beginning of the session, and

Fun as a Strategic Advantage: Applying Lessons in Engagement 185

Figure 4: Bar chart with variance comparing aggregate participant reports from Enga- ged (left), and De-Engaged (right) conditions, with ratings for rules and mechanics questions at top, and usability and interface components at bottom.
Original PDF Figure 4:
Figure 5: Portion of paired comparison grid addressing principles of logistics cove- red in month 12 gameplay, reported by condition. Positive values indicate stronger reported connections comparing pre-and post-gameplay.
Original PDF Figure 5:

again at the conclusion. Figure 5 shows the results between Engaged and De- Engaged conditions for the principles of logistics taught in the play session. Numbers in each cell indicate the difference between pre-and post-session scores, with darker colors indicating a stronger effect. Participants in the Engaged condition saw stronger connections between the relevant principles after gameplay, while those using the De-Engaged condition did not show strong effect in those areas.

CONCLUSION

Our focus on engaging features and novice training between months 5 and 12 improved overall user experience as well as perception of the game. General usability was reported as moderate to high across all players during the final month 12 testing. These initial results show some support to the effect descri- bed by Norman (2004) where visceral, immediate reactions affect long term perception, even between versions of a game with a captivated and motiva- ted audience. Perhaps surprisingly, while rules and game play were consistent across conditions, players reported making more mistakes due to the rules and being less satisfied with ease of use when working with the less visually engaging version. These players additionally saw less connection between principles of logistics, indicating less cognitive effect from gameplay. Use of the game appears to create engagement with principles of logistics across

play groups, though user experience with visual elements seems to affect user experience and perception of performance, as well as connection to learning goals. These results provide some support to the idea that engagement, even within a gaming format, can affect learning outcomes. Given these findings, we suggest that it may not be enough to convert subject matter into a digital, interactive format with immediate feedback; it is also important to present the content in a visually stimulating way to encourage players to engage with the content. We expect these effects to carry over into any domain, though future research should consider validating these claims further.

ACKNOWLEDGMENT

We would like to acknowledge the support we received from the Marine Corps, Marine Corps University, the Office of Naval Research-Global, other MIT LL staff and management who directly supported this effort, and the many play testers at MIT LL. We would especially like to thank the Marine Corps Captains and Majors that worked directly with us to make this project possible.

REFERENCES

Alekseyev, J. (2020, July). A Framework to Improve Evaluation of Novel Decision Support Tools. In International Conference on Applied Human Factors and Ergonomics (pp. 410–416). Springer, Cham. Allen, G., Chan, T. (2017). Artificial Intelligence and National Security. Study on behalf of the US Intelligence Advanced Research Projects Activity (IARPA). Belfer Center for Science and International Affairs, Cambridge MA. http://www.belfercenter.org/sites/default/files/files/publication/AI%20NatSec%20- %20final.pdf. Accessed 20 Jan 2022 Blumenfeld, P. C., Kempler, T. M., & Krajcik, J. S. (2006). Motivation and Cognitive

Engagement in Learning Environments. The Cambridge Handbook of Learning Sciences. 476–488. https://psycnet.apa.org/record/2006-07157-028. Accessed 20 Jan 2022 Buckley, P., & Doyle, E. (2016). Gamification and student motivation. Interactive learning environments, 24(6), 1162–1175. Case, R., Bereiter, C. (1984). From Behaviorism to Cognitive Behavio- rism to Cognitive Development: Steps in the Evolution of Instructional Design. Instructional Science, 13, 141–158. https://link.springer.com/content/pdf/ 10.1007/BF00052382.pdf. Accessed 20 Jan 2022 Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R., Bowers, C., & Cannon-Bowers, J.

(2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics: Theory, Research, and Practice, 7(3), 179. Denny, P., McDonald, F., Empson, R., Kelly, P., & Petersen, A. (2018, April). Empirical support for a causal relationship between gamification and learning outcomes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Glick, S. P., & Charters, L. I. (1983). War, games, and military history. Journal of

Contemporary History, 18(4), 567–582. https://www.jstor.org/stable/260304 Norman, D. A. (2004). Emotional design: Why we love (or hate) everyday things.

Basic Civitas Books.

Fun as a Strategic Advantage: Applying Lessons in Engagement 187

Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis.

Educational Psychology Review, 32(1), 77–112. Seater R., Kurucar J., Uhmeyer A. (2019) Rapid-Play Games for Evaluating Future Technology. In: Gentile M., Allegra M., Söbke H. (eds) Games and Learning Allia- nce. GALA 2018. Lecture Notes in Computer Science, vol 11385. Springer, Cham. https://doi.org/10.1007/978-3-030-11548-7_42 U.S. Marine Corps. (2016) MCWP 3-40 Logistic Operation.

https://www.marines.mil/Portals/1/Publications/MCWP%203-40.pdf?ver=2017- 03-15-124213-007 Accessed 20 Jan 2022

Cover of the proceedings volume Training, Education, and Learning Sciences

Explore the complete digital volume

Training, Education, and Learning Sciences

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-35-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →