Lucas Paletta, Herwig Zeiner, Michael Schneeberger, Yusuf Quadri (2023). Digital Shadows and Twins for Human Experts and Data-Driven Services in a Framework of Democratic AI-based Decision Support. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 73, pp. 23–33. https://doi.org/10.54941/ahfe1003971
Current automated and hierarchical structured production processes can only insufficiently deal with the upcoming flexibilization, specifically regarding the requirements within Industry 5.0. The European project FAIRWork fosters the ‘democratization’ of decision-making in production processes, hence the participation of all involved stakeholders, by introducing a decentralized AI system. Hybrid decision-making faces the challenge first to digitally represent the relevant actors – here we propose the use of digital twins – and the interpretation of that digital twin, by a human expert or by a computer algorithm, to achieve better decisions. Research on existing sensors and data technology is required. In particular, the digital representation of human operators requires so called “Intelligent Sensor Boxes”.Method: ‘Intelligent Sensor Boxes’ are firstly determined by a dedicated group of sensors, such as, low-cost sensors, biosensors, wearables, human sensors, or even virtual sensors. Specific attention is dedicated to the development of the ‘Digital Human Sensor’ (DHS) applying AI-enabled Human Factors measurement technology. Each instantiation of a DHS provides a digital vector of Human Factors state estimates, such as, digital biomarkers on physiological strain, affective state, concentration, cognitive workload, situation awareness, fatigue. On the basis of these vectors, we determine cost function parameters associated with typical (inter-)actions in the work environment. We outline an advanced approach to represent cognitive strain by studying workload related to task switching, multitasking and interruption as well as monotony effects. Furthermore, we will investigate cognitive strain in the context of environmental parameters, such as, air quality, and combine IOT with wearable bio-sensor shirts, smartwatches with biosensors, eye tracking glasses, digital events, and spatiotemporal patterns from human-machine interaction. Cost functions for optimization algorithms can be related to well-being of the worker, this allows data processing with the goal to optimize according to several input factors using data that are derived from humans or from machines like lines and robots. Those data are described with corresponding meta data to result in a descriptive data lake. Such meta data correspond to domain specific models like the production process, the working environment model, or resources models. Data processing and optimization algorithms can then be applied on this data lake. This task complements existing data with human based sensor data and provided adapted data mining tools.Results: We present relevant methodologies for human-centered wearable or mobile measurement technologies for psychological and ecological constructs as typical instantiations of the novel framework. Furthermore, the embedding of the schema of ‘Intelligent Sensor Boxes’ into the framework of ‘Democratic AI-based Decision Support’ (DAI-DDS) is sketched and argued. An outlook on future research trajectories, in particular, in the context of the FAIRWork project, is outlined in detail, and Ethical guidelines are discussed. Experimentation Laboratories, such as, the Austrian Human Factors Lab, are not only considered and presented as a co-creation space to develop new ideas, but also as test, training and communication environment for and between all stakeholders of an innovation chain. Conclusion: The framework and development of ‘Intelligent Sensor Boxes’ with data quality control and including decision modules is described in the context of its relevance within production related environments. ‘Digital Human Sensors’ applying AI-enabled digital Human Factors measurement technology will represent key drivers in the novel Industry 5.0 era.
Current automated and hierarchical structured production processes can only insufficiently deal with the upcoming flexibilization, specifically regar- ding the requirements within Industry 5.0. The European project FAIRWork fosters the ‘democratization’ of decision-making in production processes, hence the participation of all involved stakeholders, by introducing a decen- tralized AI system. Hybrid decision-making faces the challenge first of digitally representing the relevant actors – here we propose the use of digital twins – and the interpretation of that digital twin, by a human expert or by
a computer algorithm, to achieve better decisions. Research on existing sen- sors and data technology is required. In particular, the digital representation of human operators requires so-called “Intelligent Sensor Boxes”.
The concept of ‘Intelligent Sensor Boxes’ is introduced as a framework to integrate human-centred relevant digital sensing into a larger framework of decision-making in production environments. ‘Intelligent Sensor Boxes’ are firstly determined by a dedicated group of sensors, such as low-cost sensors, biosensors, wearables, human sensors, or even virtual sensors. Specific atten- tion is dedicated to developing the ‘Digital Human Sensor’ (DHS) applying AI-enabled Human Factors measurement technology. Each instantiation of a DHS provides a digital vector of Human Factors state estimates, such as digital biomarkers on the physiological strain, affective state, concentration, cognitive workload, situation awareness, and fatigue. Based on these vectors, we determine cost function parameters associated with typical (inter-)actions in the work environment.
Through intelligent human factors computing components, we envision an advanced approach to represent cognitive strain by studying workload related to task switching, multitasking, and interruption as well as monotony effects. Furthermore, we will investigate the cognitive strain in the context of environmental parameters, such as air quality, and combine it with wearable bio-sensor shirts, smartwatches with biosensors, eye-tracking glasses, digital events, and spatiotemporal patterns from human-machine interaction.
Cost functions for optimization algorithms can be related to the well-being of the worker, this allows data processing with the goal to optimize according to several input factors using data that are derived from humans or machines like lines and robots. Those data are described with corresponding metadata to result in a descriptive data lake. Such metadata correspond to domain- specific models like the production process, the working environment model, or resources models. Data processing and optimization algorithms can then be applied to this data lake. This task complements existing data with human- based sensor data and provided adapted data mining tools.
We present relevant methodologies for human-centred wearable measu- rement technologies for psychophysiological and ecological constructs as typical instantiations of the novel framework. Furthermore, the embedding of the schema of ‘Intelligent Sensor Boxes’ into the framework of ‘Demo- cratic AI-based Decision Support’ (DAI-DDS) is sketched and argued. An outlook on future research trajectories in the context of the FAIRWork project is outlined in detail, and Ethical guidelines are discussed. Experimentation Laboratories, such as the Human Factors Lab (Figure 1; Paletta, 2020), are not only considered and presented as a co-creation space to develop new ideas, but also presented as a test, training, and communication environment for stakeholders of innovation chains.
HUMAN FACTORS IN DECISION-MAKING Studies of human decision-making have demonstrated that stress exacerba- tes risk-taking. Since all decisions involve some element of risk, stress has a critical impact on decision quality (Porcelli & Delgado, 2017). Decisions
Digital Shadows and Twins for Human Experts and Data-Driven Services 25
Figure 1: The human factors lab and its role in FAIRWork. (a) The lab embeds vari- ous workspaces. (b) Various sensor technologies and platforms are available to test human-computer interfaces and monitor psychophysiological parameters during an interaction.
improve with stress up to an optimal threshold beyond which deterioration is observed. Naturalistic decision-making research (Klein, 2008) shows that in situations with higher time pressure, higher stakes, or increased ambiguities, experts may use intuitive decision-making rather than structured approaches. They may follow a recognition-primed decision that fits their experience and arrive at a course of action without weighing alternatives.
Stress directly affects human decision-making (Galvan and Rahdar, 2013) and hence can lead to several undesirable consequences, including a restri- ction or narrowing of attention, increased distraction, increases in reaction time, and deficits in the person’s working memory (Driskell and Salas, 1996). Studies of human decision-making demonstrate that stress exacerbates risk- taking and impacts decision quality. Since most managerial decisions involve some element of stress, decision aids such as decision support systems (DSS) have been proposed to mitigate its effects. Existing research has largely atten- ded to two key stressors, i.e., time pressure and information overload. For a holistic understanding of decision-making under stress and to improve decision support, an extended set of stressors and psychological experiences underlying stressful decisions is examined.
In terms of a class of stressors called ‘Decision Stressors’ there are four Decision Stressors that affect decision quality: Information overload, time pressure, complexity, and uncertainty.
Stress and Risk-Taking: the decision-makers’ likelihood to engage in risk varies greatly based on multiple decision-inherent features inclu- ding uncertainty, framing of a decision - as a potential gain or loss - and valuations of outcome valence, magnitude, and probability of receipt. As such, decisions involving risk-taking rely in part on stress-susceptible valuation and learning processes.
Information overload is a gap between the volume of information and the tools we must assimilate. Information used in decision-making is to reduce or eliminate uncertainty. Excessive information affects pro- blem processing and tasking, which affects decision-making. Humans’ decision-making becomes inhibited because human brains can only hold a limited amount of information.
Decision fatigue is when a sizable number of decision-making leads to a decline in decision-making skills. People who make decisions over an extended period begin to lose the mental energy needed to analyse all possible solutions. Impulsive decision-making and decision avoida- nce are two possible paths that extend from decision fatigue. Impulse decisions are made more often when a person is tired of analysing situations or solutions; the solution they make is to act and not think.
Though many decisions must be made under stress and many deci- sion situations elicit stress responses themselves (Starcke and Brand, 2012), decision-makers could enhance their decision-making performance and pro- tect against potential decision failures (Whyte, 1991) by adopting certain coping strategies, such as the following:
Use the “balance sheet” exercise. Janis (1982) suggests that decision- makers can foster vigilance by using the “balance sheet” exercise. This is a pre-decision process that asks the decision-maker to confront and answer the questions about the potential risks and gains not previously contemplated.
Use checklists. Kruglanski (1986) suggests that potential decision fai- lures could be prevented by preparing a checklist for all aspects of the alternatives, and before selecting the “best” alternative, these aspects should be compared against the criteria checklist.
Heighten the fear of failure. Kruglanski (1986) advocates that inserting several reminders into the decision-making process may help to raise the fear of failure. People may be assigned to remind the decision-makers of the negative consequences of their course of action.
Analyse the sources of decision-making stress. Several authors, such as Billings et al., (1980) suggest that one of the significant sources of decision-making stress is the necessity of choosing within a fixed period. Keeping the information load at a standard pace can have implications to reduce the level of cognitive and emotional stress felt by the decision-maker.
Digital Shadows and Twins for Human Experts and Data-Driven Services 27
Use decision support systems whenever feasible. It has been shown that decision support systems could be specifically designed to mitigate the negative effects of stress on human decision-making (Phillips-Wren and Adya, 2009).
Order task priorities. Research findings suggest that providing training on how to order task priorities and use efficient search strategies is an alternative way to enhance decision quality and prevent potential decision failures while making decisions under stress. Several cognitive behavioural stress-coping training programs are effective in combatting decision-related stress (Cannon-Bowers and Salas, 1998). The main aim is to prevent panic in a stressful situation and to achieve the best possible cognitive processing and decision outcome.
FAIRWork will implement a decision support system (DSS) to mitigate the negative effects of stress by applying AI-driven decision-making that considers human factors. Via Human Factors based analytics of the inno- vative decision-making process, and considering the integration of stress coping strategies above, we will lay a scientifically valid basis to design stress reducing DSS for better, i.e., fair, and more efficient decision-making.
DIGITAL MODELS; DIGITAL SHADOWS; DIGITAL TWINS Digital models are simulations of the environment and people in an area. A digital model is, for example, a virtual three-dimensional representation of an object that can be used for simulation and analysis. A digital model might be a functional 3D representation of a production line.
Digital twins (DT) are more than pure data and consist of models of the represented object, and can also contain simulations, algorithms and services that describe or influence the properties or behaviour of the represented object or offer services about it.
Digital shadow (DS) is an extended concept of digital twin (DT) where the DS has a virtual representation of physical assets and there is a one-way data flow between the physical object and its virtual counterpart allowing the vir- tual model to adapt to the physical asset. The DS gives real-time information about the status of the physical asset, which helps to interact with other DT or DS. An example of DS refers to the measurement data we collect from humans by biosensors, based on this data we can assess how the person is doing at the moment, i.e., with little or much stress.
The DT - in addition to the DS - contains models that can predict what the human condition will be in future time slots, e.g., what if we would experie- nce certain types of stress, e.g., carrying heavy car parts for a long time. This information, if fed back to a decision-making component, could eventually prevent stress, and influence the human being by avoiding too much stress in time.
For Human Factors based analytics for decision-making, the digital sha- dow of human operators requires a so-called “Intelligent Sensor Box” (ISB). The meaning of an ISB is to represent a human operator through a set of parameters that are pivotal for any higher-level decision-making regime. The
DS of the human operator ideally should be capable to quantify variables that characterize its physical, cognitive, affective, social, and motivational behaviour, in interaction with its workplace environment. For this purpose, sensors would be attached to a body sensor network via wearables. Further sensors would apply surveillance tasks via remote measurement technology, and additional sensors would represent the workers’ context within the work- place environment via human-machine and human-environment interactions. These data would be transmitted as well to a globally accessible data lake that is related to the DSS’s knowledge base after having applied an anonymization procedure. Within this data lake, global key performance indicators (KPIs) and functions are stored to be input to the AI-based enrichment module that will provide machine learning for further insight as well as optimization methods both of which would enrich the global DAI-DSS environment.
Workspace, equipment, and expertise. The Human Factors Lab (HFL) at Joanneum Research, Austria, combines state-of-the-art human-centred mea- suring technologies with AI-enabled software for behaviour-based analytics and assessment of psychological constructs in digital systems. It analyses in the context of cognitive ergonomics human stress as well as the cognitive, affective, and motivational state using mobile and wearable technologies applied at work or home. The infrastructure is equipped to investigate psych- ologically relevant parameters in real-time including a wearable biosensor and Neuroergonomics workplace using functional near-infrared spectro- scopy (fNIRS) as well as VR/AR technologies, high precision eye tracking (stationary, mobile, wearable) and motion sensing devices, furthermore treadmills, social robots, and software libraries for gaze-based analytics. The HFL has access to various IoT software libraries for permanent data acquisition for various sensors and gateways to set up a wireless sensing network.
Targeted research. Specific attention is dedicated to the development of the ‘Digital Human Sensor’ (DHS) applying AI-enabled Human Factors measu- rement technology. Each instantiation of a DHS provides a vector of Human Factors state estimates – e.g., on stress, affective state, concentration, wor- kload, situation awareness, fatigue, etc. – with the purpose to determine cost function parameters associated with typical (inter-)actions in the work envi- ronment. The HFL will be used to test certain novel technologies, be it for the measurement of reduced stress concerning alternative decision-making pathways and interaction devices, or in the context of monotonous work that fosters fatigue and errors in the processing. We will develop a novel approach to estimating cognitive strain by studying workload related to task switching, multitasking, and interruption as well as monotony effects, using wearable bio-sensor shirts, smartwatches with biosensors, eye tracking glasses, digi- tal events, and spatiotemporal patterns from human-machine interaction, etc and concerning environmental strain, such as insufficient air quality.
The internal architecture of the DAI-DSS Intelligent Sensor Box supports the data flow from human- and workplace-mounted sensor data to higher abstractions of human factors, i.e., dominantly ergonomic, and psycho- physiological constructs that determine the mental state and behaviours of the human, and from this the sociotechnical systems within the production
Digital Shadows and Twins for Human Experts and Data-Driven Services 29
process. The integration of human-centred data into the overall DAI-DSS decision-making process needs the anonymization of the highly vulnerable psychophysiological data. However, personalized data are of interest to the individual worker or decision-maker and are available for individual insight on a security- and privacy-preserving basis.
The internal architecture of the DAI-DSS Intelligent Sensor Box as well as its dependencies on other components is presented in Figure 2.
Local Wearable Sensor Network: Non-obtrusive wearable sensors are the basis to measure human factors during the execution of tasks at the work- place. Figure 3 depicts the sensors for human factors state estimation from wearable data streams and digital data analytics: (a) biosensor shirt (QUS, Austria) for vital data monitoring, (b) eye-tracking glasses (Pupil Labs, Ger- many) for recording eye movements and derive from the digital sensor data stream the mental state analytics. The Pupil Labs Invisible (b) are very lightw- eight calibration-free eye tracking glasses with a detachable scene camera and cable connection to a mobile recording unit. The digital sensor network in FAIRWork consists of wearables that are specifically suited to measure psych- ophysiological variables, such as cognitive-emotional stress, physiological strain, and fatigue.
Figure 2: Architecture of intelligent sensor box.Figure 3: Sensors for technologies with nonobtrusive, light-weight human factors state estimation from wearable data streams and digital data analytics: (a) biosensor shirt (QUS, Austria) for vital data monitoring, (b) eye-tracking glasses (Pupil Labs, Germany) for recording eye movements and derivation of mental state analytics.
The local environment of the workplace can impact the human factors state of the individual workers. Several sensors are used to represent the state of stressors and their impact on human profiles. The Local Workplace Sensor Network might include sensors for air quality, temperature, noise, etc. The data can be transferred to the Local Data Receiver Module where the sensor data are pre-framed for storage in the local database. The Local Data Receiver Module implements the communication between sensor devices and a local server component. This module enables either data from the stationary care or to represent the DSS by another care organization. Local data management consists of a local data lake that receives data from the sensor networks and receiver module and stores the data for real-time data management. It is a Mongo DB that enables rapid access and storage. The component also initia- tes specific data managing operations (see digital health profile). The digital health profile includes biometric data and indicators about well-being and health from individual workers. It needs specific conditions on security and privacy since it contains the most vulnerable data. The local data manage- ment (above) might operate to anonymize in an early stage several data to make them available for further feature-managing computing. Based on the data of the local data management component, Human Factors Intelligent Services will operate in real-time. These services are at the core of the Intelli- gent Sensor Box and include methods that were developed in earlier projects but also newly developed estimators of psychophysiological variables, such as stress, concentration, fatigue, and motivation. In addition, environmental variables - e.g., air quality depending on the percentage of oxygen, opera- ting temperature at the workplace, etc. – will be considered. Furthermore, optimization methods will be available to construct novel measures that par- ticularly refer to the requirements of socio-technological systems. A typical example would be the computation of individual duration of time periods or extents of time shifts that would enable work with high motivation and the highest safety prediction score. This component generates the dynamic
Digital Shadows and Twins for Human Experts and Data-Driven Services 31
human profiles that can be used, after anonymization, for further processing in the DAI-DSS Knowledge Base.
The component of the local decision support system (DSS) will predict cer- tain risk levels with relevance for the socio-technical system that arises from the sensor networks. The physiological strain index (PSI; Moran, 2000) will provide risk levels for cardiovascular cause-based risk of dropout. Cognitive- emotional risk stratification (Paletta et al., 2022) will provide safety-oriented risk levels due to estimated scores for mental fatigue. Other risk levels will be discussed and applied according to the requirements of the DAI-DSS arch- itecture and the use cases. The User Interfaces Providing Individual Human Factors State can display individually relevant information to the worker depending on their interest. These data are particularly personalized and need specific concern on security and privacy encapsulation. However, this service might be relevant from an ethical point of view to integrate the motivation of the worker into the DAI-DSS framework. Before transferring the resulting risk level data to the DAI-DSS Knowledge Base, these must be anonymized. Several algorithms are under consideration, with the common objective to render the vulnerable data without reference to the individual but at the same time enable further integration of the data into the DAI-DSS framework. The main individual subcomponents can be described as follows:
The Intelligent Sensor Box provides services for supporting device mana- gement on the edge. With the registry, a list of devices and services can be added. For devices and services, the functions “add”, “modify”, and “delete” are supported.
Data storage – especially the saving of time series – should be suppor- ted. Therefore, a suitable edge storage database is used. Data can be processed in different ways (e.g., filtered, aggregated, summarized) and the diverse algorithm applied to the personal profiles.
RESTful APIs are used to integrate the user interface. Sending data to the box is supported in the form of a streaming interface or a dedicated RESTful API.
The resource manager handles basic resources such as hard disk, space in the data storage, etc. The resource manager is used to estimate whether sufficient resources are still available for basic functions. If the resource manager runs out of resources, it can report this event to a neighbouring edge analytics box, or another connected software component.
Furthermore, a monitoring service is supported.
The DAI-DSS Intelligent Sensor Box (ISB) is a very important part for pro- viding the data for the digital twin. The DAI-DSS Knowledge Base will receive biosensor data as well as risk level-based metadata. These data are mandatory to be able to construct measures based on concrete experience in the HFL or use case-specific labs, and finally, to develop meaningful persona constructs. In ISB we apply local intelligent services to comply with privacy concerns. In the AI Enrichment component, we develop the persona models. Further opti- mization algorithms will be implemented by using the more dynamic human profiles (human factors service results). As described in the development focus area, the development of persona will be further generated in the
DAI-DS AI Enrichment Module. Risk levels for working groups or typical worker’s representatives are designed. Ultimately, the Digital Twin of Digital Human Sensor (DHS) will be operated there.
There will be the focus on three main development areas, as follows,
Mapping environment and machine-relevant data to digital Human Factors estimators (services mapping the local context)
Development of local decision support components and services from validated digital human factors estimators (HF-DSS) that are based on biosensor data (Digital Shadow). Development of specific risk levels for the well-being of the worker that will be transferred to the knowledge base.
Research and development on specific optimization variants by using human profiles and other use case-relevant resources.
The development of persona profiles (see description above).
CONCLUSION
The framework and development of ‘Intelligent Sensor Boxes’ with data qua- lity control and including decision modules are described in the context of its relevance within production-related environments. ‘Digital Human Sen- sors’ applying AI-enabled digital Human Factors measurement technology will represent key drivers in the novel Industry 5.0 era.
ACKNOWLEDGMENT
This work was partially funded by project FAIRWork (grant No 101069499 of the European Commission).
REFERENCES
Billings, R. S., Milburn T. W. & Schaalman M. L. (1980). A model of crisis perception: A theoretical and empirical analysis, Administrative Science Quarterly, 25(2), 300–316. Driskell, James & Salas, Eduardo & Johnston, Joan. (1999). Does Stress Lead to a
Loss of Team Perspective? Group Dynamics: Theory, Research, and Practice. 3. 291–302. 10.1037/1089-2699.3.4.291. Galván, A., Rahdar, A. (2013). The neurobiological effects of stress on adolescent decision making, Neuroscience, Volume 249, 2013, Pages 223-231, ISSN 0306- 4522, https://doi.org/10.1016/j.neuroscience.2012.09.074 Janis, I. L. (1982). Groupthink, (2nd ed.), Houghton Mifflin, Boston. Klein, G. (2008). Naturalistic Decision Making. Human Factors, 50(3), 456–460.
https://doi.org/10.1518/001872008X288385 Kruglanski, A. W. (1986). Freeze-think and the Challenger, Psychology Today, August, 48–49. Moran D. S. (2000). Stress evaluation by the physiological strain index (PSI). Journal
of basic and clinical physiology and pharmacology, 11(4), 403–423. https://doi.or g/10.1515/jbcpp.2000.11.4.403 Paletta, Lucas (2020) Human Factors Lab, DIGITAL – Institute for Digital Technologies, Joanneum Research Forschungsgesellschaft mbH, Graz, Austria. Website: https://www.joanneum.at/en/digital/infrastructure/human-factors-laboratory.
Digital Shadows and Twins for Human Experts and Data-Driven Services 33
Paletta, L., Pszeida, M., Schneeberger, M., Dini, A., Reim, L., Kallus, K. W. (2022).
Cognitive-emotional Stress and Risk Stratification of Situational Awareness in Immersive First Responder Training, 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), Ioannina, Greece, 2022, pp. 1–4, doi: 10.1109/BHI56158.2022.9926805. Phillips-Wren, G. & Adya, M. (2009). Risky Decisions and Decision Support - Does Stress Make a Difference? Proceedings of JAIS Theory Development Workshop Sprouts: Working Papers on Information Systems, 9(55). https://sprouts.aisnet.o rg/9-55 Porcelli, A. J., & Delgado, M. R. (2017). Stress and Decision Making: Effects on
Valuation, Learning, and Risk-taking. Current opinion in behavioral sciences, 14, 33–39. https://doi.org/10.1016/j.cobeha.2016.11.015 Starcke, K., Brand, M. (2012). Decision making under stress: A selective review, Neuroscience & Biobehavioral Reviews, Volume 36, Issue 4, 2012, Pages 1228- 1248, ISSN 0149-7634, https://doi.org/10.1016/j.neubiorev.2012.02.003 Whyte, G. (1991). Decision failures: why they occur and how to prevent them. The
Executive, 5, 23–31.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-49-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional