Alina Marquet, Sven Münker, Antonia Markus, Anas Abdelrazeq, Robert H. Schmitt, Bert Leyendecker (2024). Conceptualization of a Serious Game for Validation and Optimization of a Decision Support System in Disturbance and Error Management on the Shop floor. In: Usability and User Experience. AHFE Open Access, vol. 156, pp. 1–11. https://doi.org/10.54941/ahfe1005409
The integration of digital technologies and production elements has profound implications for disturbance and error management in operational-tactical production areas, driven by demands for flexibility, responsiveness, and resilience. Decision support systems (DSS) are seen as crucial tools, leveraging intelligent algorithms to aid decision-making and enhance efficiency.A human-centric DSS, utilizing artificial intelligence (AI) methods, has been developed to facilitate data-driven decisions and shift responsibilities to operational staff. Special attention is given to human-DSS interaction to understand its effects on employee workflows.This work aims to conceptualize a serious game as an innovative method for evaluating the DSS. The game tests technical functionality and simulates impacts on production workflows and human-system interaction (HSI). Realistic scenarios of disturbances, errors, and tool changes will enable early error identification and problem-solving.Through this comprehensive study, improvement opportunities in HSI will be uncovered, and the relevance of the serious game in the DSS context evaluated. AI models will be validated by simulating disruption and error situations in the game, followed by detailed examination of user experience and system value.Multiple participant groups will be involved, with varying levels of DSS assistance, allowing for differentiated analysis of system impacts. This approach will provide a comprehensive understanding of the DSS's practical applicability and benefits in real production environments. Future plans involve conducting the study and reporting the results.
The increasing integration of digital technologies and high connectivity in the production environment significantly influences disruption and error man- agement in operational-tactical production areas. This transformation is the result of growing demands for flexibility, responsiveness, and resilience in
modern production environments and also has immediate consequences for the human factor in production (Kieviet, 2016).
In this context, production employees are facing increasingly complex challenges in dealing with faults and errors (Glöckner 2020; Schmitt et al., 2018). The need for quick and informed decision-making in chaotic sit- uations and under stress requires a redesign of approaches to disruption and error management. DSS are considered key factors as they enable han- dling and processing vast amounts of data through intelligent algorithms to support decision-making and increase operational efficiency (Holsapple, 2008). As a result of these developments, a human-centered DSS based on meth- ods of AI was being developed. This innovative system analyzes complex decision situations and generates rational decision proposals for operational staff. Special attention is paid to the interaction between humans and systems to understand the impacts of this DSS on employee work practices.
The main goal of this work is the comprehensive conception of a seri- ous game. Serious games are chosen as a method for evaluation due to their ability to engage participants actively, provide immediate feedback, and offer a safe environment for experiential learning. Their advantages over traditional evaluation tests include higher engagement levels, personalized learning experiences, and the simulation of complex scenarios, ultimately leading to improved retention and long-term learning outcomes. It is serving as an innovative method to evaluate the applicability of the system. This serious game aims not only to test the technical functionality of the DSS but also to simulate its effects on production processes and the interaction between humans and the system. By creating realistic scenarios of disrup- tions, manufacturing errors, and tool changes on production machines, the aim is to enable early error detection during deviations and initiate timely problem-solving processes.
This work intends not only to understand the practical applicability and benefits of the DSS in the real production environment but also to uncover improvement potentials in HSI. The system’s AI models are tested by simu- lating new types of problems and errors in a serious game. Then, the user experience and how valuable the system is are carefully studied in a safe test- ing environment after the serious game. Through the participation of multiple groups in this comprehensive study – one group with the prototype of the system, a control group without support, and groups with restricted system functions – a holistic understanding of the practical applicability and benefits of the DSS is obtained. These insights form the basis for future developments and implementations.
The paper follows a structured approach, beginning with an exploration of conceptual basics surrounding disruption and error management on the shop floor, highlighting the growing importance of data-driven decision-making and the role of shop floor management. It then discusses HSI principles and DSS integration. The subsequent sections detail the structure of the serious game, its implementation, and expected results, providing a comprehensive overview of the study’s objectives and methodologies.
Conceptualization of a Serious Game for Validation and Optimization 3
CONCEPTUAL BASICS Disruption and Error Management on the Shop Floor
In the context of disruption and error management on the shop floor, dis- ruptions refer to events such as machine breakdowns that can lead to the shutdown of individual machines or entire production areas (DIN 40041). Errors, on the other hand, concern the production process or the end product itself, for example, consistent deviations in the dimensions of manufactured units (DIN EN ISO 9000:2015).
The importance of data-driven decisions in disruption or error situations is increasing (Schmitt & Günther, 2020). However, the amount of data and information that must be considered in decision-making during disruptions and errors are already hardly comprehensible for employees today, making system states increasingly complex in disruption and error situations (Schick et al., 2017; Stocker et al., 2014). This complexity makes it difficult for employees to cope with situations within the required reaction time without supportive technologies. This makes it increasingly difficult for employees to get through such situations within the necessary reaction time without supporting technologies, which leads to overload in the form of stress and hinders effective problem solving and thus efficient process control (Holm et al., 2014).
The rather abrupt occurrence of errors and disruptions should also be seen in this context, which requires rapid and, above all, time-critical corrective measures. According to Weyer, this urgency leads to a proper “dramatization of the disruption” (Weyer 2008). This view is supported by the SME Day study 2018 by Wolf et al., which indicates that the increasing complexity of the business environment leads to decisions under stress and time pressure. Even the mere making of decisions, regardless of their nature, can cause stress among workers, leading to a decline in performance with increasing stress levels (Wolf et al., 2018). Furthermore, employees often lack the required know-how and experience to take responsibility in the production process (Crespo et al., 2009).
In this context, shop floor management, a leadership philosophy meaning “leading at the scene of action” (Leyendecker & Pötters 2020), plays a cru- cial role. At the core of the concept are daily meetings between managers and workers on the shop floor to analyze performance and problems, especially disruptions and errors, in the value-added process (Bertagnolli 2018). This management philosophy emphasizes the human factor and prevents an iso- lated view of planning and operational implementation of work steps (Suzaki 1994; Hurtz 2013). Standardized processes enable targeted communication using both the “top-down” and “bottom-up” approaches, facilitating efficient problem- solving (Lendzian & Martin-Martin 2016; Dahm & Brückner 2017). The manager acts as a coach, supporting employees in independent problem- solving, and the decentralization of decisions to shop floor workers is a central aspect of shop floor management (Lanza et al., 2018). This con- cept ensures early identification of problems or disruptions, regular review
of goal achievement, and a structured approach to problem-solving (Horváth & Partner 2013; Dombrowski et al., 2018).
Human System Interaction To meet this demanding decision-making process, the harmonious interaction between humans and technology proves to be a crucial lever. HSI serves as the interface between humans and computers or humans and systems. Through this interface, humans and systems communicate and interact with each other in the form of information exchange (input/output).
A cognitively ergonomic design of such HSI plays a significant role in the effectiveness and efficiency of the collaboration between humans and sys- tems (Chao 2009a). HSI must be designed to enable efficient interaction between a human and the computer or system (Jacko 2007). A particular challenge is the increasing variety of requirements with a growing number of users, as HSI must fulfill different subjective requirements of as many users as possible simultaneously (Chao 2009b). In companies, this means that HSI should be able to capture their context as much as possible to adapt the pre- sented information based on the user (e.g., shop floor vs. management level). Regarding HSI design and efficiency in system operation, HSI should be as easy to use as possible for the user (input by the user) while being under- standable (output by the system) at the same time. The goal of HSI is to achieve good mutual communication and thus to create a common level for communication between humans and systems (Chao 2009b).
Decision Support System The prediction of decision outcomes using statistical analyses with com- pany and production data is a common method. Ghasemaghaei et al. (2018) and Abubakar et al. (2019) demonstrate how data can be transformed into information to support decision-makers in the decision-making process. The acquisition of company data and its processing through tools from the fields of data mining and machine learning is utilized (Ghasemaghaei et al., 2018). Further processing involves enriching the data with contextual information, thereby making them information-rich foundations for decisions. Research studies, such as those by Rowley et al. (2007), show a positive correlation between this information processing and decision outcomes.
DSSs play a central role in collecting and processing company and produc- tion data, such as machine availability and location data, for users. In contrast to traditional business intelligence systems, DSS does not only evaluate pro- cessed information but also conduct scenario analyses for various courses of action and provide evaluated action suggestions. This contributes to trans- parency in operational processes by simulating the impact of decisions and allowing for better estimation. Flexible and dynamic systems already rely on algorithms from the field of AI.
Bitkom highlights the high potential of AI in decision support in a posi- tion paper (Bitkom, 2017). The BMWi (Federal Ministry for Economic Affairs and Energy of Germany) emphasizes in a document on the use of AI in Industry 4.0 that AI, particularly in decision support and providing
Conceptualization of a Serious Game for Validation and Optimization 5
troubleshooting instructions during disruptions, has great potential (BMWi 2019). It is noted that the processing of large amounts of data and the per- ception of cognitive tasks in the context of decision-making can increasingly be supported by algorithms.
Integration of HSI in Disruption and Error Management Shop floor management holds potential for flexibility, responsiveness, and resilience by shifting responsibilities to the operational-tactical areas of pro- duction. This decentralization reduces corporate control loops and enables faster responses to unforeseen events. The decision-making capabilities of shop floor employees, currently limited due to centrally bundled leadership competencies and lack of decision-making experience, can be supported dur- ing disruptions by the development of AI-based DSSs, thereby improving decision-making in critical situations (Marques et al., 2017).
In addition to functionality, the user-friendliness of the human-AI inter- face plays a crucial role in the overall acceptance of the system and efficient communication between humans and the system (Shneiderman and Plaisant, 2004). Shop floor management, through the active involvement of employees in problem identification and solution, provides the ideal framework for integrating a DSS in disruption and error management.
SERIOUS GAME FOR VALIDATION AND OPTIMIZATION OF THE DSS The aim of this paper is to develop a concept for determining the impact of the human-centered DSS in the context of disruption and error management on the shop floor. This concept includes the development of a serious game in the Modellfabrik of the Koblenz University of Applied Sciences. The Modell- fabrik is a business laboratory in the Department of Business Administration. Using serious game, business processes can be simulated based on a model truck production from toy building kits. According to Blötz (2015), a serious game is considered a scientifically recognized method and can thus be applied as an instrument to measure the influence of shop floor management on a problem-solving process. Another point that is tested with the help of the seri- ous game is the interaction between man and machine in order to understand the effects of the system on the work processes of the store floor employees. In addition to the subjective impressions of the participants, objective metrics such as lead time and output are also captured to validate the influence of the DSS.
Prior to this, potential use cases of the DSS in the area of disruption and error management on the shop floor were identified through workshops and interviews with shop floor employees and managers in small and medium- sized industrial companies. Two use cases emerged from this, based on which the serious game was developed. One is the prediction of tool changes, and the other is the identification of errors or the recognition of specification deviations on the production machines. The system is expected to make suggestions, for example, on when a tool change would be sensible before
defective parts are produced and the production process needs to be inter- rupted. Furthermore, there was a desire expressed for a central overview of the production processes for shop floor managers so that production processes can be more effectively controlled.
Structure The serious game represents a value chain of a fictitious company in flow production for the manufacturing of the miniature truck depicted in Figure 1.
Figure 1: Product “Truck”.
Production takes place on a production line. Figure 2 illustrates the struc- ture of the flow production with the individual production stations of the serious game.
Figure 2: Structure.
The components of the truck are picked up in the central warehouse and then transported to the storage locations of the production stations. Material supply to the assembly stations is thus ensured via the storage areas.
Assembly takes place at production stations one to seven, with the material flow directed from left to right, and working according to the first-in-first-out principle.
Key performance indicators are recorded by inputting data into the system, which is available to employees at each production station. The following metrics are captured to evaluate the production process:
Lead time: Measures the total production duration from step 1 to step 7.
Productivity: Each delivered truck enhances overall productivity.
Wait times: Time employees spend waiting due to delays, inquiries, or bottlenecks.
Queries: Tracks production station inquiries or clarification needs.
Error rate: Percentage of defective products in total production.
Conceptualization of a Serious Game for Validation and Optimization 7
In addition to these key figures, the subjective perceptions were recorded through surveys and feedback from the participants. This is done through questionnaires that the participants fill out at the end of the simulation to share their impressions, feelings, assessments and observations about the serious game and the use of the DSS.
Implementation To assess the impact of the DSS, four treatment groups are required in the sim- ulation game. Each treatment group requires the participation of eight people. These subjects are distributed across the various stations of the simulation game, with one person working at each production station, and the eighth person acting as the employee responsible for the entire shop floor. The sub- jects at the individual workstations simulate the machines in the simulation game and strictly adhere to the specified instructions.
In case of queries, material shortages, or errors, the subjects are not autho- rized to find independent solutions; instead, they must seek help from the shop floor manager. The task of the shop floor manager is to oversee the entire production process and ensure that everything runs as smoothly as possible at the various workstations.
The serious game focuses on the production of the truck, which consists of 22 individual components connected by sliding or pressure connections as well as adhesives.
The serious game begins at workstation one, which is dedicated to the production of the truck. At workstations one to four, the individual parts of the truck are cleverly connected to each other, so that the basic structure of the truck is completed after the fourth production step (see Figure 3).
Figure 3: Basic structure of the truck.
In the fifth production step, the loading area, consisting of a building board, is precisely cut using a template, a pencil, and a carpet knife. Addi- tionally, a hole is drilled into the board using a toothpick. In the subsequent production step six, the resulting loading area is placed on the truck and connected with the plug. In the final production step, a stamp with the logo is pressed onto a sheet of paper, and the paper is attached to the loading area using adhesive tape. The participant at the last production step then transports the finished truck to the finished goods warehouse.
Each participant at the production stations is equipped with a laptop containing the developed software to enable the capture of key metrics. In addition to capturing and displaying key metrics, the technical equipment
also serves as hardware for the DSS. To ensure the execution of the simu- lation game with four different treatment groups and to obtain meaningful results, one group is provided with access to the complete DSS. Two other groups receive the DSS with varying degrees of restricted functionality, while the fourth group receives no technical support whatsoever. Shop floor super- visors in the three groups with the DSS are each provided with a tablet to enable them to monitor key metrics and have a comprehensive overview of the production.
The treatment groups with DSSs can inform the shop floor supervisor specifically when assistance is needed during disruptions and errors by enter- ing information into the system. The system accordingly indicates when, for example, the carpet knife needs to be replaced or the stamp needs new ink. Likewise, the shop floor supervisor receives information on their tablet when a production station is about to run out of stock, allowing them to replen- ish the inventory of the storage locations in a timely manner. Thus, the shop floor supervisor gains an overview and suggestions on how to intervene in the production process before it stalls.
The serious game lasts for 30 minutes. The goal of the participants is to produce as many trucks as possible as quickly and error-free as possible.
EXPECTED RESULTS One of the overarching goals of the system is decision support regarding shop floor disturbance and error management. For this reason, the production pro- cess of the serious game was designed to encounter disruptions and errors fairly quickly, requiring the shop floor supervisor to react. Tools such as the carpet knife and stamp were chosen to need replacement after a certain num- ber of uses. Since the participants at each production station are simulating machines, they rely on instructions to solve the problem. Furthermore, the material at the production storage locations is not sufficient for the 30-minute duration of the serious game. Therefore, the individual storage locations rely on new components being brought from the warehouse.
Significant deviations are expected in the results of the serious games between the treatment groups. It can be assumed that in the groups with the system, key metrics will be better, and subjective perceptions such as stress and overload will be lower compared to those without the system. However, it remains unclear how the individual functions of the system affect the pro- cess performance and the burden on the shop floor supervisors. Therefore, the two additional treatment groups, each receiving differently restricted sys- tem functionalities, will also be tested. This allows for a detailed analysis of the effectiveness of each function of the DSS. Through the serious game and subsequent feedback, not only the functionality of the system can be exam- ined, but also how user-friendly it is designed. Attention will be paid to how well the shop floor supervisors understand and operate the user interface of the system. Additionally, an examination will be conducted on how the sys- tem presents action suggestions and how well the users understand, interpret, and implement them.
Conceptualization of a Serious Game for Validation and Optimization 9
SUMMARY AND OUTLOOK This paper focused on the conception of a serious game for the validation and optimization of a DSS in shop floor disturbance and error management. The increasing integration of digital technologies and the rising demands for flexibility, responsiveness, and resilience in modern production environments have highlighted the need for a redefinition of approaches to disturbance and error management.
A human-centered DSS was developed to analyze complex decision situ- ations and generate rational decision proposals for operational staff. Spe- cial attention was paid to the interaction between humans and systems to understand the impact of this system on the employees’ work processes.
The main goal of the study was to design a serious game serving as an innovative method for evaluating the applicability of the system. The serious game was developed not only to test the technical functionality of the system but also to simulate its effects on production processes and the interaction between humans and the system.
The serious game, conducted in the Modellfabrik of the University of Koblenz, simulates realistic scenarios of disturbances, errors in production, and tool changes at production machines. By involving various treatment groups, the effects of the DSS on process performance and the burden on shop floor supervisors were analyzed.
The results of the serious games are expected to show significant differ- ences between the groups with and without the DSS. It is anticipated that the metrics in the groups with the system will be better, and the subjective per- ception of stress and overload will be lower than in those without the system. The detailed analysis of the individual system functions will provide a com- prehensive understanding of the practical applicability and benefits of the DSS and lay the foundation for future developments and implementations.
ACKNOWLEDGMENT
The research presented in this paper has been carried out within the joint research project “HUMEUS - Humanzentriertes Entscheidungsunter- stützungssystem für das Stör- und Fehlermanagement auf dem Shop floor” (IGF 22510 N) at the Laboratory for Machine Tools and Production Engi- neering WZL of RWTH Aachen and Koblenz University of Applied Science. The authors gratefully acknowledge the support of the funding organization The German Federation of Industrial Research Associations (AiF) within the program for sponsorship by Industrial Joint Research (IGF) of the German Federal Ministry of Economic Affairs and Energy.
REFERENCES
Abubakar M, Elrehail H, Alatailat M A; Elçi A (2019): Knowledge management, decision-making style and organizational performance. In: Journal of Innovation and Knowledge, 4(2), 104–114. Bertagnolli F (2018): Lean Management: Einführung und Vertiefung in die japanis-
che Management-Philosophie, Lehrbuch, Wiesbaden: Springer Gabler.
Bitkom e. V. (2017): Entscheidungsunterstützung mit KI. Online verfüg- bar unter https://www.uni-kassel.de/fb07/fileadmin/datas/fb07/5-Institute/IWR/ Hornung/170901-KI-Gipfelpapier-online.pdf. Blötz, U (2015). Planspiele und Serious Games in der beruflichen Bildung: Auswahl, Konzepte, Lernarrangements, Erfahrungen - Aktueller Katalog für Planspiele und Serious Games 2015. BMWi (2019): Technologieszenario “Künstliche Intelligenz in der Industrie 4.0”. Online verfügbar unter https://www.plattform- i40.de/IP/Redaktion/DE/Downloads/Publikation/KI-industrie-40.html. Chao G (2009a): Human-Computer Interaction: Process and Principles of Human-
Computer Interface Design. In: ICCAE 2009. Chao G (2009b): Human-computer interaction: The usability test methods and design principles in the human-computer interface design. In: 2nd IEEE ICCSIT. Crespo I, Spychala A, Lacker T (2009): Selbstorganisiertes Arbeiten in KMU. In:
Modernisierung kleiner und mittlerer Unternehmen. Berlin: Springer, 179–185. Dahm M, Brückner A (2017): Lean Management im Unternehmensalltag: Praxisbeispiele zur Inspiration und Reflexion, FOM-Edition, Wiesbaden: Springer Gabler. DIN 40041 (1990): Zuverlässigkeit; Begriffe. Deutsches Institut für Normung. DIN EN ISO 9000:2015 (2015): Qualitätsmanagementsysteme – Grundlagen und
Begriffe. Deutsches Institut für Normung. Dombrowski U, Wullbrandt J, Jäger F, Linge S (2018): Braunschweiger Shopfloor Management Assessment. ZWF Zeitschrift für wirtschaftlichen Fabrikbetrieb. Ghasemaghaei M, Ebrahimi S, Hassanein K (2018): Data analytics competency
for improving firm decision-making performance. In: The Journal of Strategic Information Systems, 27(1), 101–113. Glöckner H (2020). Systemdynamische Modellierung des reaktiven Fehlermanagements in Produktionssystemen. Dissertation RWTH Aachen. Aachen: Apprimus- Verlag. Holm M, Garcia A C, Adamson G, Wang L (2014): Adaptive Decision Support for
Shop-floor Operators in Automotive Industry. In: Procedia CIRP 17, 440–445. Holsapple, C. (2008). Decisions and Knowledge. In: Handbook on Decision Support Systems 1. International Handbooks Information System. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-48713-5_2 Horváth & Partner (2013): Performancesteigerung in der Produktion durch Shopfloor Management, Studienbericht, Stuttgart: Horváth & Partner GmbH. Hurtz, A; Stolz, M (2013): Shop-Floor-Management: Wirksam führen vor Ort,
Göttingen: Business Village. Jacko J A (2007): Human computer interaction: Interaction Design and Usabillity.
Berlin, Springer. Kieviet, André (2016): Digitalisierung der Wertschöpfung: Auswirkung auf das Lean Management, in: Erfolgsfaktor Serie, pp. 41–59, [online] doi:10.1007/978-3-662- 49752-4_3. Lanza G, Nyhuis P, Fisel J et al. (2018): Wandlungsfähige, menschzentrierte Struk-
turen in Fabriken und Netzwerken der Industrie 4.0 (acatech Studie), München: Herbert Utz Verlage. Lendzian H, Martin-Martin R (2016): Shopfloor-Management – Nachhaltige Problemlösungen schaffen. In: Künzel (Hg.), Erfolgsfaktor Lean Management 2.0. Wettbewerbsfähige Verschlankung auf nachhaltige und kundenorientierte Weise. Erfolgsfaktor Serie. Berlin: Springer, 83–98.
Conceptualization of a Serious Game for Validation and Optimization 11
Leyendecker B, Pötters P (2020): Shopfloor Management. Führen am Ort des Geschehens. München: Hanser. Marques, Maria/Carlos Agostinho/Grégory Zacharewicz/Ricardo Jardim-Gonçalves
(2017): Decentralized decision support for intelligent manufacturing in Industry 4.0, in: Journal of Ambient Intelligence and Smart Environments, vol. 9, no. 3, pp. 299–313, [online] doi: 10.3233/ais-170436. Rowley J et al. (2007): The wisdom hierarchy: representations of the DIKW hierarchy. In: Journal of Information Science, 33(2), 163–180. Schlick J, Stephan P, Loskyll M (2017): Industrie 4.0 in der praktischen Anwendung.
Automatisierung und Logistik. In: Vogel-Heuser et al. (Hg.): Handbuch Industrie 4.0. Berlin: Springer. 57–84. Schmitt R H, Günther R. (2020): Digitalisierung des Fehlermanagements. Ein (selbst-) lernender Ansatz für kleine und mittlere Unternehmen. In: ZWF, 115(6), 391–393. Schmitt R H, Ruessmann M, Hellebrandt T S, Heine I (2018): Complaint and
failure management: Key insights from a cross-industry study. Online verfüg- bar unter https://www.mckinsey.com/business-functions/operations/our-insights/ complaint-and-failure-management-key-insights-from-a-cross-industry-study. Shneiderman, Ben/Plaisant, Catherine (2004): Designing the user interface: strategies for effective human-computer interaction. 4. Auflage. Boston, USA. Pearson/Ad- dison Wesley. Stocker A, Brandl P, Michalczuk R, Rosenberger M (2014): Menschzentrierte IKT-
Lösungen in einer Smart Factory. In: Elektrotech. Inftech, 31(7), 207–211. Suzaki K (1994): Die ungenutzten Potentiale – Neues Management im Produktionsbetrieb München: Hanser. Weyer J (2008): Techniksoziologie – Genese, Gestaltung und Steuerung sozio-
technischer Systeme. München: o. V. Wolf T, Fueglistaller U, Müller J (2018): KMU und Entscheidungen. Online verfügbar unter https://www.alexandria.unisg.ch/256294/1/KMU-Tag-Studie_2018.pdf.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Usability and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-32-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional