Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Gamification: 20 years on, what have we learned?

QA Ltd, Cheltenham, UK ↗
Publication year
Digital Object Identifier10.54941/ahfe1004776

Abstract

Gamification has gained significant traction and attention over the last decade, though the term goes back over two decades, and the application of the principles themselves likely pre-date the term itself. Establishing a widely accepted definition of gamification and classification of the underpinning principles is still ongoing; this paper considers three proposed models, and their unique contributions to research into gamification, particularly in the cyber environment.We then examine a case study in cyber-security gamification, assessing the performance of the Cyber Explorer’s programme – a UK government sponsored initiative, which aims to utilise gamification to enhance cyber security education in UK based 11–14-year-olds. We examine how the various gamification principles have been applied, their effectiveness and implications for cyber security education.The resulting analysis and discussion highlight a need for more research into the effectiveness of gamification in sub-populations, to examine the impact of gamification elements on learning effectiveness rather than motivation, and to identify the specific gamification mechanisms which are most effective in the cyber security learning arena.

BACKGROUND

Just over two decades ago, the term gamification was first coined by Nick Pelling (Marczewski, 2023), though it would be nearly another decade before it took centre stage in learning and development. In 2011, papers were still proposing a definition for the word, including discussions around the meaning of games versus play or fun. One of the most popular single definitions was “the use of game design elements, in non-game contexts” (Deterding et al., 2011). This definition remains popular, and is clearly sepa- rated from game-based learning, which could be defined as the reverse – the use of learning elements in games. In the interest of simplicity, this paper will use Deterding’s definition, and avoid game-based learning.

Gamification elements and HEXAD profile - Andrzej Marczewski has proposed many theories, models and concepts relating to gamification, including a ‘periodic table of gamification elements’ (Marczewski, 2023). His HEXAD model has been extensively researched and built upon by oth- ers; Tondello et al., 2016 found empirical support for most elements of the HEXAD model, with all categories except player, found to have high test- retest reliability, indicating that these user types are enduring and can be

© 2024. Published by AHFE Open Access. All rights reserved. 167

168 Aldred

reliably identified. Additionally, this research found correlations between the user types and Big Five personality traits, indicating alignment to established psychological concepts. They also established correlation between the user types and his various gamification elements.

This could provide a unique benefit to the cyber industry, in terms of enabling the tailoring of gamification to their unique demographics. For example, the disruptor user profile appears to share core characteristics with certain roles, such as penetration testers, and gamification mechanics and elements which appeal to disruptors, could be most beneficial to engaging this subsection of the cyber workforce, whereas social engineers could ben- efit most from free-spirit and socialiser elements. However, this would need further research to establish evidence-based links, rather than theoretical.

Gamification taxonomy - Toda et al. (2019a) recently developed a tax- onomy of game elements, in collaboration with gamification experts, and building upon prior classification attempts, including the GAME elements proposed by Marczewski (2015) and the framework proposed by Klock et al. (2016). This taxonomy was developed based upon gamification in educa- tional environments, and has subsequently been cited in studies of computer science and cyber-education. This model therefore aligns well with the Cyber Explorers case study, and the paper later discusses the examples from this taxonomy, which were worked into the Cyber Explorers cyber education solution.

Octalysis framework - Mohanty & Christopher (2023) put forward a liter- ature review to provide support towards the popular Gamification Octalysis Framework, developed originally by Chou (2019). This framework outlines eight core drivers for user motivation. This model was one of several cited in research by Qusa and Tarazi (2021), who developed a specific framework for the gamification of cyber security awareness in high school students. It there- fore does appear to have already proven beneficial not only to gamification in general, but specifically cyber gamification. It is positive to see that work has already begun into developing subject-specific gamification frameworks.

Much of the benefit of gamification has been approached from the angle of motivation, which bases the assumption that training or learning is gener- ally seen as laborious rather than fun. However, there is evidence to suggest that gamification may not only be motivational, but effective, in terms of improving retention of information (Putz et al., 2018). Further delving into gamification could reveal whether it contains any ability to improve comprehension of complex concepts, especially within the region of cyber particularly. Additionally, we have seen the beginning of bespoke gamifica- tion frameworks for cybersecurity, which will likely continue to expand over the coming decades.

CYBER EXPLORERS - A CASE STUDY OF GAMIFICATION Background - Cyber Explorers aims to expand the cyber security skills of 11–14-year-olds across the UK. It seeks to reach a wider pool of talent out- side those already interested in technology; engaging girls and young people in hard-to-reach communities. It is an innovative new approach to cyber

Gamification: 20 Years on, What Have We Learned? 169

security learning developed through a collaborative effort led by The Depart- ment for Science, Innovation and Technology (DSIT) and supported by QA Ltd. To date, the platform has reached over 70,000 learners across more than 2,500 schools, learning organisations and home educators. The solution was an entirely online platform containing various learning content. It was designed primarily for use in schools, with all registrations needing to commence with a school or learning organisation and at least one teacher or educator. Learners are then attached to schools and divided into cohorts for specific teachers. In terms of gamification design or elements used within the platform to enhance the experience, Cyber Explorers design concepts can be clearly mapped back to all of the above frameworks.

Table 1. Mapping key cyber explorers design features to the aforementioned frameworks.

Table available in the original PDF paper.

Cyber Explorers Gamification Elements

Gamification Taxonomy

Octalysis Framework

General elements - Narrative

Epic meaning - narrative

Use of interesting career pathways in fictional city, with a range of characters.

Fictional - narrative, storytelling Player type - leaderboard Social - competition Accomplishment - leaderboard

School leaderboards based on engagement and new school competitions.

Player type - badges

Performance - achievement

Learners can earn unique badges by completed certain combinations of content.

Achiever type – boss battles

Accomplishment – badges (achievement symbols) A final challenge to ‘save the city’ which requires mastery of all learning objectives.

Personal - puzzle Accomplishment – boss fights

Narrative / Epic meaning - The primary gamification element used was narrative, with ‘narrative atoms’ which can be combined to provide the user with choices to enable exploration. The various narrative options for the first batch of content were specifically chosen based on the interests of the target audience, and popular career pathways - social media content creation, building your own business, healthcare, sports and the environment. Students can work through the learning objectives in the context of any of these topics, and switch between them as desired.

Leaderboards / Competition - Whilst an individual leaderboard presents personal information challenges, a school leaderboard does not face the same scrutiny. The Cyber Explorers team therefore implemented a school leaderboard, where schools can compete against each other to complete the most content per student. Winners of this are announced publicly, and sent certificates and branded merchandise, such as stickers, for their students. Since the announcement of the competition, which requires participating schools to have completed a substantial portion of the online content to take part, the platform has seen a spike in learner engagement, with course one episode completion increasing by 142% over the 6 weeks following the announcement compared to the same period the prior year.

170 Aldred

Badges - Within the platform, learners are able to achieve and collect var- ious badges for their engagement with the content. Prior research has found that badges are a more effective tool in women, compared to men, (Codish and Ravid, 2017, Toda et al., 2019b) which ties into the Cyber Explorers aim to promote cyber careers to young women. Indeed, looking at badge achievement, female users have achieved more badges between them than the male users, despite there being fewer female learners in total, and overall engagement being greater in male learners. It would suggest female learn- ers are specifically completing content in such a way as to maximise badge collection.

Boss battles / Challenges - The final challenges are not strictly physical fights in the traditional gaming sense, but they represent the same concept – a culmination of all that has been learned so far, a challenge where you must apply the skills you have previously been mastering, in order to suc- ceed and win. This is very much akin to summative assessment, but with narrative context, which appeals to achiever gamer types, providing a feeling of accomplishment.

User types - Assessing Cyber Explorers against the HEXAD of user types, the platform has been best designed for free-spirit and achiever users, it supports the player and philanthropist types to an extent, but has minimal offerings for the socialiser and disruptor. The free spirits can explore content from multiple perspectives, and choose their own pathway through the learn- ing. This was a key element of the team’s design, to promote creativity and diversity. It also benefits the achievers with the various challenges, levels and ‘boss battles’ aka Save The City or City Saved. There are badges and school leaderboard for the players, and the narrative follows an altruistic plot to save others, fulfilling the philanthropists.

OVERALL FINDINGS Engagement - Cyber Explorers is a unique programme, however, compared to other free, online, self-paced learning, the completion rates are very positive. Overall, engagement sits at 15.4% and 32.9% for the two courses, com- pared to a median completion rate of 12.6% across 221 Massive Open Online Courses (Jordan, 2015). These are not perfect comparisons, in part due to the target audience, as well as the learning structure. Most learning programmes are predominantly linear, and have fixed start and end points, whilst the Cyber Explorers platform promotes choice and exploration through a range of non-linear content, and has unusual engagement/completion definitions.

When engagement is analysed by gender, there appears to be a consistent difference in completion rates between those who identify as male, versus those who identify as female. Past research has similarly found that gam- ification is more motivational for male than female users (Jent & Janneck, 2017), however, a study on the impact of gamification on self-efficacy in exer- cise found gamification to be more effective for the female learners. Another study found that when men were in a predominantly male group, their per- ceptions were different from when they were a gender minority within a group (Codish and Ravid, 2017). It therefore may not be straightforward to

Gamification: 20 Years on, What Have We Learned? 171

definitively conclude that gamification is more effective in a particular gen- der, but the specific elements, audience demographics and contextual factors each contribute to outcomes in a complex relationship. Indeed, as previously mentioned, when badge completion data is analysed, the level of engage- ment reverses, and seems to be a greater motivator for young women, which provides more evidence that the individual components of gamification can have different outcomes, and looking at gamification as one concept is far too broad.

Table 2. Cyber explorers content completion by gender.

Table available in the original PDF paper.

Registered Episodes (Course One) Missions (Course Two) 1+ 6+ 7+ 12+ 21+

Male 51.6% 51.5% 53.4% 65.7% 72.9% 83.0% Female 48.4% 48.5% 46.6% 34.3% 27.1% 17.0%

As a government backed programme, the Cyber Explorers programme aimed to encourage cyber education and careers amongst ethnic minorities in the UK, in order to build a more diverse workforce of the future. Engagement data appears to demonstrate differences between broad ethnicity groups, with Asian learners being over-represented in the platform, whilst learners belonging to various Black ethnicity groups, and white minority groups are under-represented. White British learners are roughly proportionally rep- resented, though appear to have higher representation in engaged groups. All other ethnicities have lower engagement representation than registration, suggesting that the gamified content may be appealing more to White British learners than other groups. For the Asian group, their engagement is still above proportional representation, which is positive, whereas the Black eth- nic group appears to be least engaged, and perhaps are less motivated by gamification. However, it is possible there are other influencing factors, such as a lack of interest in cyber in general, or characters or careers being less appealing. Additionally, it is likely there are cultural differences within these broad groups. Due to low rates of learners providing their ethnicity, sam- ple sizes were too small to analyse differences between more specific ethnic groups.

Table 3. Group ethnicity across registered and engaged groups, compared to national representation.

Table available in the original PDF paper.

England & Wales Population

Registered Users

1 or More Episodes

6 or More Episodes

Asian groups 8.9% 13.8% 13.1% 13.3% Black groups 9.3% 5.3% 4.4% 5.1% White minorities 7.4% 7.1% 4.4% 5.1% White British 74.4% 73.7% 78.0% 76.4%

172 Aldred

Unfortunately, the Cyber Explorers programme does not collect informa- tion on user disability or the prevalence of neurodivergence in the audi- ence. This would prove a fascinating addition in future, to compare the effectiveness of gamification in these sub-populations.

Impact - Amongst learners who engaged with the Cyber Explorers content, there have been significant improvements seen to the key measures. There are four main areas in which the intervention aimed to influence learners: cyber security knowledge, cyber security skills, knowledge of cyber related careers, and intentions to pursue digital qualifications at school.

Statistical testing between groups using self-report measures indicates a significant positive difference in engaged learners at post-learning compared to unengaged learners. This was also shown to not be a result of passing of time or completing a second questionnaire. What cannot be determined from these results is whether the positive outcomes are a result of gamifi- cation improving retention or motivation. However, in this specific target audience, motivation is a considerable factor. These results align with other findings that not only is gamification effective, but it is particularly beneficial in younger populations (Jent and Janneck, 2017).

Table 4. Results of impact on key measures – averages before and after completing learning.

Table available in the original PDF paper.

Measurement Average Before

Average After

% Increase Testing Results Intentions to study digital GCSE 2.5 3.0 20% Significant α = 0.05. Cyber security knowledge 2.9 3.7 28% Cyber security skills 2.8 3.6 29% Knowledge of cyber security jobs 2.4 3.1 29%

Across all four measures, male learners scored significantly higher than their female counterparts at the baseline, before engaging with content. How- ever, at follow-up, engaged male and female groups were found to not be statistically different for the questions on cyber knowledge, skills and jobs. For intentions to study a digital subject at GCSE, male learners were still sig- nificantly more likely to score higher than female learners. This could suggest that engagement with the Cyber Explorers initiative has helped to close the gap between male and female learners on three of the four measures.

Results could not be further examined for differences in ethnicity, due to sample size issues.

Cyber Explorers appears to provide yet more empirical support for the effectiveness of gamification, both in general and for cyber security, as well as some indications of the effectiveness of specific game elements in sub- populations, which can be used to target the cyber security community. The gamification concepts which produce improved results in men may be more applicable in the training of the current cyber workforce, which is over-represented by male employees (BCS, 2021). Whereas the findings around badges proving more popular to women, will be key to engagement

Gamification: 20 Years on, What Have We Learned? 173

in the upcoming cyber workforce, should we hope to move towards equal representation.

Additionally, in terms of impact, engagement with the programme does appear to bring female learners up to the same level as male learners, though there remains a distinct gap in desire to pursue related qualifications. This could be explained by the findings of Zahedi et al. (2021) who found that despite improved performance in cyber security, their gamification did not increase enjoyment or interest in the subject in women. This study also found that leaderboards were not considered motivating for female learners, which may link to the tendency for competitive behaviour to be greater in the gender with higher social authority (Gneezy, Leonard, and List, 2009), which would be men, in the still moderately patriarchal UK.

The socialisation elements of gamification have been extensively discussed and found to be effective in other gamified applications, such as Strava or Duolingo (Hyzy & Wardle, 2023). However, this proves a challenging series of mechanics to implement in the target audience for Cyber Explorers due to the age range and data protection considerations. Whilst a leaderboard, where learners could compete against friends from their school, would likely prove motivational, there are limitations in what can ethically and legally be achieved, and collecting the correct permissions for personal information has been reported by Cyber Explorer staff as one of the greatest challenges.

The importance of social aspects may not be immediately apparent within the cyber field, however, human factors indicate that social interactions are often one of the weakest links in the chain when it comes to cyber security, with the entire field of social engineering coming to light in recent decades. According to Havard Business Review, over 80% of cybersecurity incidents are caused by human error (Chamorro-Premuzict, 2023) - any digital system is only as secure as the behaviours of the authorised users.

CONCLUSION

20 years since its official inception, there are many examples of gamification entering the education and learning and development industries. In that time, research has produced clear definitions and beneficial, reliable frameworks which build on existing psychology literature, and appear to be converging towards common themes. Research has covered a large range of principles, elements, populations and explanations for the relatively short period of investigation, and also delved specifically into the use of gamification in the cyber security area, finding it to have positive outcomes (Ros et al., 2020; Ask et al., 2023; van Steen and Deeleman, 2021). Additionally, we have seen the beginning of bespoke gamification frameworks for cybersecurity, which will likely continue to expand over the coming decades.

We can see how the elements of these frameworks are being implemented in cyber solutions such as Cyber Explorers. There is still much work to be undertaken in terms of seeking the underpinning mechanisms which influ- ence retention rather than motivation, and establishing the differences in sub-populations, or personality variances, to enable ideal design for specific target audiences, such as the cyber industry.

174 Aldred

ACKNOWLEDGMENT

The author would like to acknowledge Alex Pani´c (learning specialist) and Karl Aldred (gaming specialist) for their assistance and feedback.

REFERENCES

Ask, T. F., Knox, B. J., Lugo, R. G., Hoffmann, L., & Sütterlin, S. (2023) Gamification as a neuroergonomic approach to improving interpersonal situational awareness in cyber defense. Frontiers in Education, 8, 988043. Frontiers. British Computing Society. (2021, July 07). BCS diversity report 2021: Women in IT.

Available at: https://www.bcs.org/policy-and-influence/equality-diversity-and-in clusion/bcs-diversity-report-2021-women-in-it/ Chou, Y. K. (2019) Actionable gamification: Beyond points, badges, and leaderboards. Packt Publishing Ltd. Chamorro-Premuzict, T. (2023, May 03). Human Error Drives Most Cyber Incidents.

Could AI Help? Harvard Business Review. Available at: https://hbr.org/2023/05/ human-error-drives-most-cyber-incidents-could-ai-help Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: defining” gamification”. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments (pp. 9–15). Gneezy, U., Leonard, K. L., & List, J. A. (2009). Gender differences in competi-

tion: Evidence from a matrilineal and a patriarchal society. Econometrica, 77(5), 1637–1664. Hyzy, M., & Wardle, B. (2023). Gamification for product excellence. Packt publishing. Jent, S., & Janneck, M. (2018). Using gamification to enhance user motivation: the

influence of gender and age. In Advances in The Human Side of Service Engi- neering: Proceedings of the AHFE 2017 International Conference on The Human Side of Service Engineering, July 17–21, 2017, The Westin Bonaventure Hotel, Los Angeles, California, USA 8 (pp. 3–10). Springer International Publishing. Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16(3), 341–358. Klock, A. C. T., Gasparini, I., & Pimenta, M. S. (2016, October). 5W2H Frame-

work: a guide to design, develop and evaluate the user-centered gamification. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems (pp. 1–10). Marczewski, A. (2023) The Gamification Design Handbook: Even ninja monkeys like to play (3rd ed.). Gamified UK. Marczewski, A. (2015). Even ninja monkeys like to play: Gamification, game

thinking and motivational design. Gamified UK. Mohanty, S., & Christopher B, P. (2023). A bibliometric analysis of the use of the Gamification Octalysis Framework in training: evidence from Web of Science. Humanities and Social Sciences Communications, 10(1), 1–14. Putz, L. M., Schmidt-Kraepelin, M., Treiblmaier, H., & Sunyaev, A. (2018). The

influence of gamified workshops on students’ knowledge retention. In GamiFIN (pp. 40–47). Qusa, H., & Tarazi, J. (2021, January). Cyber-hero: A gamification framework for cyber security awareness for high schools students. In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0677–0682). IEEE.

Gamification: 20 Years on, What Have We Learned? 175

Ros, S., Gonzalez, S., Robles, A., Tobarra, L. L., Caminero, A., & Cano, J. (2020).

Analyzing students’ self-perception of success and learning effectiveness using gamification in an online cybersecurity course. IEEE Access, 8, 97718–97728. Toda, A. M., Klock, A. C., Oliveira, W., Palomino, P. T., Rodrigues, L., Shi, L., Bittencourt, I., Gasparini, L., Isotani, S. & Cristea, A. I. (2019a). Analysing gamification elements in educational environments using an existing Gamification taxonomy. Smart Learning Environments, 6(1), 1–14. Toda, A. M., Oliveira, W., Shi, L., Bittencourt, I. I., Isotani, S., & Cristea, A. (2019b).

Planning gamification strategies based on user characteristics and dm: A gender- based case study. arXiv preprint arXiv:1905.09146. Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A., & Nacke, L. E. (2016, October). The gamification user types hexad scale. In Pro- ceedings of the 2016 annual symposium on computer-human interaction in play (pp. 229–243). van Steen, T., & Deeleman, J. R. (2021). Successful gamification of cybersecurity

training. Cyberpsychology, Behavior, and Social Networking, 24(9), 593–598. Zahedi, L., Batten, J., Ross, M., Potvin, G., Damas, S., Clarke, P., & Davis, D. (2021).

Gamification in education: A mixed-methods study of gender on computer science students’ academic performance and identity development. Journal of Computing in Higher Education, 33, 441–474.

Cover of the proceedings volume Human Factors in Cybersecurity

Explore the complete digital volume

Human Factors in Cybersecurity

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-03-8

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →