Yavor Dankov, Boyan Bontchev, Albena Antonova (2021). Gaming and Learning Analytics for Educational Video Games. In: Human Systems Engineering and Design (IHSED2021): Future Trends and Applications. AHFE Open Access, vol. 21. https://doi.org/10.54941/ahfe1001167
The design and development of educational video games, and the realization of a successful video game project, often requires the research and analysis of data as a key element for success. Increasingly companies are integrating analytics tools into their structures, taking advantage of the analytics capabilities. The focus of this paper is on the analytics instruments developed in the APOGEE software platform for educational video games. The paper presents the workflow of using these instruments in the platform for analysis and evaluation. The integration of the analytics tools into the platform will provide the opportunity to monitor all activities carried out in the platform, to process, analyze, and evaluate all available data, user behavior, and user experience in the platform. This will allow for the formation of an overall assessment of the created educational video games and their contents, and the APOGEЕ platform, including evaluations for learnability, playability, and usability.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
INTRODUCTION
Serious games and in particular educational video games are an area of high interest from researchers, teachers, practitioners, and educational institutions. The role of game-based learning and the use of games in the process of learning is increasingly strengthened (Juul, 2010). Explicitly, educational video games provide users with an interactive and modern approach to presenting educational information for perception, in line with modern technologies that are part of people's daily lives. In educational video games, educators and teachers have the opportunity to achieve the set educational goals and objectives with the help of game-based learning.
One of the main challenges for educational video games is their distribution and reaching a wide audience of users with a variety of characteristics and preferences (Dondlinger, 2007). Among the challenges, facing educational video games is their comparison with entertaining video games. Entertaining video games are created for entertainment purposes and to satisfy the user experience of users within the game. On the other hand, educational video games are free to play and designed and created for educational purposes, whereupon making a profit is not a priority. They contain specialized educational content and the element of entertainment and user experience must be accompanied by the realization of educational goals, designed for a specific educational group of learners with different characteristics. X The process of designing educational games often requires the use of specialists from other fields to conduct additional research (studies of conceptual requirements and general requirements for the environment and spaces (Andreeva, 2019a), color and general artistic aspects and solutions of virtual design (Andreeva, 2019b; Deykov and Andreeva, 2017), digitized art in virtual environments (Deykov, 2013), personalized content and so on). to contribute to the fulfillment of all requirements for the creation of the educational game for the specific educational group of learners.X This determines the narrowly specialized audience of educational video games and the potential number of users. Nevertheless, the process of design and development of both entertainment and educational video games involves extensive research and analysis of data as key elements for the realization of a successful video game project (Shute and Wang, 2016). The results of this data processing provide users with valuable information and knowledge on data that would be difficult to obtain without the capabilities provided by the analytics tools (Dankov and Birov, 2018; Paunova- Hubenova et al, 2020).
The focus of this paper is on the analytics instruments of the APOGEE software platform for generating educational video games (APOGEE, 2021). The paper presents the workflow of using these instruments in the platform for analysis and evaluation. The development of analytics tools and their integration into the platform will provide designers of educational video games (educators, teachers, and others) to monitor all activities carried out in the platform, as well as to analyze and evaluate the user behavior (students and players), and the user experience in the platform. The integration of the analytics tools and their functionalities in the APOGEE platform will enable the processing and analysis of all available data in the platform. This will
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 allow for the formation of an overall assessment of both the created educational video games and for the formation of an overall assessment of the APOGEЕ platform, including evaluations for learnability, playability, and usability. The results of these analyzes and evaluations will serve the designers to improve the educational video games they have designed in the platform, as well as to make them more personalized (learning content and gaming content) and adaptable to user behavior. This will contribute to improving the quality of the user gaming experience and user learning experience in the platform, proving the benefits of those instruments, as well as will contribute to the further development improvement of the APOGEE platform, its functionalities, and capabilities.
The paper is organized as follows: Section 2 introduces the APOGEE software instruments, and the focus is on the analytics instruments. Section 3 presents the workflow of using the analytics instruments in the APOGEE platform. The paper ends with a short discussion about technical, methodological, and organizational issues concerning the development of analytics instruments, and a conclusion.
ANALYTICS INSTRUMENTS IN THE APOGEE SOFTWARE PLATFORM The APOGEE platform includes a set of tools that support the processes of designing and managing the design of educational video games in the platform.
Figure 1. Analytics Instruments in the APOGEE Software Platform
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
The software instruments support the processes of analysis and evaluation of game design, and the evaluation of all activities performed in the APOGEЕ platform. These instruments and their functionalities are developed based on the application of Taxonomy of Instruments for Facilitated Design and Evaluation of Video Games for Education (TIMED-VGE) (Dankov and Bontchev, 2020). The main designed software instruments are divided into two main categories: assistive instruments and analytics instruments, each category containing several subcategories of tools (Dankov and Bontchev, 2020; Bontchev et al, 2020) The focus of this paper is on analytics tools. The category of analytics tools consists of three main subcategories, divided as follows: Learning Analytics, Gaming Analytics, and Analytics for Users (Dankov and Bontchev, 2020; Bontchev et al, 2020). Figure 1 illustrates the set of designed analytics instruments of the APOGEE software platform (Dankov and Bontchev, 2021), based on the TIMED-VGE taxonomy.
Gaming Analytics and Game Telemetry Gaming Analytics (or Game Analytics) is an area of constant development. Gaming Analytics tools are analytics instruments that are used in research, development, and analysis of video games and one of the main purposes of Gaming Analytics tools is “to support decision making, at operational, tactical and strategic levels and within all levels of an organization – design, art, programming, marketing, user research” (Seif El-Nasr et al, 2013). On the other hand, Gaming Analytics tools take advantage of the wide range of features that analytics tools have. Gaming Analytics tools are used to process and analyze data, and to extract valuable information and knowledge about the video game industry.
With the incredibly high growth rates of the gaming industry and the variety of users of video games, one of the most commonly used terms is the game telemetry. The term game telemetry is becoming increasingly important, both for video game creators and their users and for all participants in this multimillion-dollar industry. This is because game telemetry (as telemetry is defined as any transmitted signal) represents all available data, regardless of data sources, as long as this data is obtained over a distance (digitally transmitted) and this data is directly related to the game development and research in the field of games (Seif El-Nasr et al, 2013).
Therefore, the game telemetry provides a source of raw unprocessed data for the analytics tools – Gaming Analytics (and for learning analytics tools for game data of educational video games). To be able to process the raw game data from Gaming Analytics tools, it is necessary to have the so-called game metrics of data. A game metric represents “a quantitative measure of one or more attributes of one or more objects that operate in the context of games” (Seif El-Nasr et al, 2013).
In the APOGEE software platform for educational video games, the main sources of game metrics are telemetry data about the player's behavior (Learning Analytics is about the learner's behavior). The analysis and selection of specifically defined metrics to be integrated into the APOGEЕ platform are planned for future research.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
Learning Analytics Instruments Learning Analytics instruments are primarily focused on monitoring all the data that is relevant to the learning process. Both Gaming Analytics and Learning Analytics are a type of analytics tools with a variety of analytics capabilities. As a definition, Learning Analytics could be defined as "measuring, collecting, analyzing and reporting data on learners and their context for understanding and optimizing learning and the environment in which it takes place" (Long and Siemens, 2011; Siemens, Dawson and Lynch, 2013; Alonso-Fernandez et al, 2017). The integration and use of Learning Analytics can contribute to the following areas: as tools for quality assurance and quality improvement; to increase retention and engagement rates; for assessment and action in case of different results among the trainees; as well as a tool for development and introduction of adaptive learning (Sclater, Peasgood and Mullan, 2016). In the APOGEE software platform, we integrate tools for the Learning and Gaming Analytics for educational purposes and for monitoring all data related to the educational processes within the platform. The main sources of metrics are telemetry data about the learner’s behavior. We argue that collecting and analyzing data with the Learning Analytics tools will provide various analytics capabilities to the APOGEE platform. Thus, this will augment the possibilities for measurements and evaluation of the generated educational video games, overall user learning experience, and overall educational capacity of the APOGEE platform.
THE WORKFLOW OF USING THE ANALYTICS INSTRUMENTS OF THE APOGEE SOFTWARE PLATFORM
Figure 2 presents the main workflow of using the analytics instruments of the APOGEE software platform for educational video games. The workflow presents the process of using the three main categories of the analytics instruments, included in the APOGEE platform. The focus of the illustrated diagram is on the two main categories: Learning Analytics and Gaming Analytics. The analytics instruments in the platform are used for monitoring the data flows in the platform and for analysis and evaluation of the data for improving the overall experience of the users in the APOGEE platform. The instruments are used also for analysis and evaluation of the designed and created educational video games. The data results from the Analytics instruments provide the necessary and important information for the users of the platform, including teachers and educators, and creators of educational video games for better understanding the requirements of students, the best personalized educational content for specific students’ groups, and so on. Furthermore, using the Analytics instruments into the educational software platforms such as the free APOGEE software platform for the generation of educational video games will
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 provide the necessary data for improving and enhancing the platform, and also prove the benefits of those instruments, receiving multiple positive feedbacks from the users of the platform.
Figure 1. Domains of human systems integration. (Adapted from U.S Air Force The user has the opportunity to choose to view the data from the analytics tools (view analytics), or to use the full range of analytics capabilities that provide these tools (analysis and evaluation). In both cases, the analytics instruments have a wide range of options for visualizing users' data, using a variety of data visualization
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 techniques and metaphors for data visualization In the APOGEE platform, the user in his role as an educational video game designer is faced with three possibilities: he can take advantage of the designed functionalities of Learning Analytics, Gaming Analytics, and Analytics for Users. In Figure 2, the focus is on two of the main analytics instruments - Learning Analytics and Gaming Analytics.
By choosing to use the Learning Analytics tools, the user can use the following instruments and their designed functionalities in terms of game learning results, game ratings and learning user experience, learner/player models, and learnability evaluation. The user uses these tools, with the main purpose of analysis and evaluation concerning his specific work, after which he has the opportunity to complete the work process by saving the work in the database and returning to the starting position. The use of the Learning analytics tools in the APOGEE platform is directly related to the analysis and evaluation of learning content and learner's behavior. The results of these tools will serve the designers of educational video games, to improve the educational content of the designed video games, as well as to make a learnability evaluation and to improve the learning experience of users in the platform.
All available data in the platform are combined in a single layer of data, presented at the bottom of Figure 2. These data are stored in a user database, a gaming and learning content database, as well as a database for storing all data related to analytics tools. The data layer directly connects and communicates with the three main categories of tools, which include Learning Analytics, Gaming Analytics, and User Analytics, as well as their sub-tools and their designed functionalities.
When choosing the Gaming Analytics instruments, the user can take advantage of the tools and their designed functionalities related to game playing results, game ratings and playing user experience, playability evaluation, as well as additional features such as game adaptation, personalization, and behavior settings of virtual heroes or so-called non-player characters (NPC). Upon completion of the work, the designer has the opportunity to return to his original position and choose to use again one of the instruments of Gaming Analytics. If the user considers that he has finished working with these tools, then the user has the opportunity to save his work in the database and return to the starting position - to select one of the main categories of tools. The use of Gaming Analytics tools in the APOGEE platform is directly related to the analysis and evaluation of game content and player's behavior. The results of these tools will serve the designers of educational video games, to improve the game content of the designed video games, as well as to make playability evaluation and to improve the playing experience of users in the platform.
Figure 2 also presents the instruments Analytics for users, whose main purpose is the monitoring of data related to game creator activity, Game student activity (learner and player), and Analytics for usage. The analysis and selection of specifically defined metrics to be integrated into the APOGEE platform are planned for future research. The integration of the analytics instruments into the APOGEE software platform will allow for the formation of an overall assessment of the educational video games and the formation of an overall assessment of the APOGEЕ platform for learnability, playability, and usability.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
DISCUSSION
In order to build useful analytics instruments for developing and designing educational video games, the following group of issues has to be taken into account:
1. Technical issues – how to identify users within online and desktop game versions; how the system will be made extendable with various analytics (such as additional statistical metrics); how teachers, learners, and administrators will adjust and calibrate different sets of analytical instruments; and finally – how could they extend these instruments with new functionalities; 2. Methodological issues – how to validate UX, playability and learnability by self-reports, how to build self-reporting instruments with less than 20 questions, and how to organize questions both for the maze game and for any mini-game embedded in maze halls; 3. Organizational (process) issues – how to evaluate non-adaptive vs adaptive educational video game, games without NPC vs games offering NPC support to the player, etc., in order to find out potential advantages and drawbacks. Based on the proposed approach, the APOGEE game development engine allows game designers to use a systematic development process. The first version of the games can be further calibrated and improved, based on the first players’ feedback and in-game analytics. This way game designers - teachers and education experts can have the overall control over the game design, better recognizing learners’ and players’ preferences. The analytics instruments can serve as a base for the development of complex and personalized metrics and dashboards of learning/gaming indicators, improving feedback, reflections, and educational value of games. Furthermore, analytics instruments and feedback data can serve for the development of additional smart services (Antonova, Dankov, and Bontchev, 2019) that can further facilitate customization and adaptation of educational video games. While under customization we mean personalization of both the didactic content and feedback provided to the player, the adaptation will include dynamic adjustment of game difficulty, mechanics, and audio-visual effects. Both the customization and personalization are going to be based on player model characteristics like age, gender, learning goals, previous experience, and playing/learning style.
CONCLUSIONS
The paper presented usage of analytics instruments integrated into the APOGEE software platform for generating educational video games (APOGEE, 2021; Bontchev et al, 2020). The offering of learning and gaming analytics tools provide educational video games designers such as educators and with monitoring the player/learner results such as player score (effectiveness), efficiency, playing time, and earned points for each mini-game included into maze halls and, as well, for the
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 whole maze game. Moreover, they will be able to view valuable statistics describing the quality of generated maze games like descriptive statistics of players’ outcomes, T-tests for the significance of means, and correlations and effect size between outcomes and player model characteristics.
As future work, we plan to finalize the inclusion of appropriate metrics for the users of the APOGEE platform. This includes in-game metrics for the behavior of the players and metrics for the behavior of the learners. The metrics will be used for further evaluation of the processes of design, creation, and gameplay of the educational video games within the APOGEE platform.
ACKNOWLEDGMENTS
The research leading to these results has received funding from the APOGEE project, funded by the Bulgarian National Science Fund, Grant Agreement No. DN12/7/2017, and from the National Scientific Program “Information and Communication Technologies in Science, Education and Security” (ICT in SES) financed by the Ministry of Education and Science of Bulgaria.
REFERENCES
Alonso-Fernandez, C., Calvo-Morata, A., Freire, M., Martinez-Ortiz, I. and Fernández- Manjón, B. (2017). “Systematizing game learning analytics for serious games”. In Proc. EDUCON, doi: 10.1109/EDUCON.2017.7942988. Andreeva, A. (2019a). "Conceptual Requirements for Non-Traditional Exhibition Spaces.
Architectural, Design and Art Aspects". Bulgarian Journal for Engineering Design, Issue 40, Mechanical Engineering Faculty, Technical University-Sofia, pp.53-58, ISSN:1313- 7530 Andreeva, A. (2019b). "Colorful and General-Artistic Aspects of Architecture and Design.
Viewpoints”. In book: Aesthetic achievements of the exhibition activities of Technical University-Sofia 2009_2019“, vol. 1, Issue 1, Technical University-Sofia, pp.78-96, ISSN: 2682-9797 Antonova, A., Dankov, Y., and Bontchev, B. (2019). “Smart Services for Managing the Design of Personalized Educational Video Games”. In: Proceedings of ACM Int. 9th Balkan Conf. in Informatics (BCI’2019), Association for Computing Machinery (ACM), New York, USA, Article 20, pp. 1–8, doi: https://doi.org/10.1145/3351556.3351574 APOGEE Project Homepage (2021), APOGEE Web site: http://www.apogee.online/index-
en.html , Accessed 19/04/2021. Bontchev, B., Antonova, A., and Dankov, Y. (2020). “Educational video game design using personalized learning scenarios.” In: Gervasi, O. et al. (eds.) Computational Science and Its Applications – ICCSA 2020. ICCSA 2020. Lecture Notes in Computer Science, vol. 12254. Springer, Cham, pp. 829–845, https://doi.org/10.1007/978-3-030-58817-5_59 Dankov, Y., and Bontchev, B. (2020). “Towards a taxonomy of instruments for facilitated
design and evaluation of video games for education”. In: Proc. of the 21st Int. Conference on CompSysTech 2020, Association for Computing Machinery (ACM), New York, USA, pp. 285–292, https://doi.org/10.1145/3407982.3408010
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
Dankov Y., and Bontchev B. (2021). “Designing Software Instruments for Analysis and Visualization of Data Relevant to Playing Educational Video Games”. In: Ahram T., Taiar R., Groff F. (eds) Human Interaction, Emerging Technologies and Future Applications IV. IHIET-AI 2021. Advances in Intelligent Systems and Computing, vol 1378. Springer, Cham. https://doi.org/10.1007/978-3-030-74009-2_54 Dankov, Y. and Birov, D. (2018). “General Architectural Framework for Business Visual
Analytics”. In: Shishkov, B. (eds) Business Modeling and Software Design. BMSD 2018. Lecture Notes in Business Information Processing, vol 319. Springer, Cham., pp. 280–288, https://doi.org/10.1007/978-3-319-94214-8_19 Deykov, Y. (2013). "Museum of Museums”. Digitized Art in A Virtual Museum Environment.
First International Conference Modern Technologies in Cultural Heritage, Ivanova, M. (eds), vol. 1, Technical University of Sofia, pp. 45-48, ISSN:2367-6523. Deykov, Y., Andreeva, A. (2017). "Current aspects of the virtual design of expo-environment
- gallery, museum, church”. V International Conference Modern Technologies in Cultural Heritage, Ivanova, M. (eds), vol. 5, Technical University of Sofia, pp. 17-22, ISSN:2367-6523 Dondlinger, M. J. (2007). “Educational video game design: A review of the literature”. Journal of applied educational technology, 4(1), pp.21-31 Juul, J. (2010). “A casual revolution: Reinventing video games and their players”. MIT press Long, G., and Siemens, G. (2011). “Penetrating the Fog: Analytics in Learning and Education”.
Educ. Rev. pp.31–40 Paunova-Hubenova, E., Todorova, K., and Kademova-Katzarova, P. (2020). “Learning Analytics – Need of Centralized Portal for Access to E-Learning Resources”. In BdKCSE'2019, IEEE, pp.1-8, doi:10.1109/BdKCSE48644.2019.9010600 Sclater, N., Peasgood, A., Mullan, J. (2016). “Learning Analytics in Higher Education”. Jisc,
UK Seif El-Nasr, M., Drachen, A., and Canossa, A. (eds.) (2013). “Game Analytics: Maximizing the Value of Player Data”. Springer, London, doi: 10.1007/978-1-4471-4769-5_2 Shute, V., Wang, L. (2016). “Assessing and supporting hard-to-measure constructs in video
games”. The handbook of cognition and assessment, pp. 535-562 Siemens, G., Dawson, G., and Lynch, G. (2013). “Improving the Quality and Productivity of the Higher Education Sector”. Policy and Strategy for Systems-Level Deployment of Learning Analytics. Canberra, Australia: Society for Learning Analytics Research for the Australian Office for Learning and Teaching
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Systems Engineering and Design (IHSED2021): Future Trends and Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8987-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional