Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Cognitive training for adults with developmental deficits led by a socially intelligent robot

1 Faculty of Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia ↗
2 Varstveno Delovni Center, Toncke Hocevar Ljubljana, Slovenia ↗
3 Faculty of Arts, University of Ljubljana, Ljubljana, Slovenia ↗
Publication year
Digital Object Identifier10.54941/ahfe1008123

Abstract

This work presents a socially intelligent robotic system for cognitive training of adults with developmental deficits in a therapist-half-supervised setting. The core idea is to move beyond one-way robot prompting toward a structured, ethically grounded, data-aware intervention framework in which the robot, end user, and therapist form a coordinated triad. The proposed system combines robot behaviour management, session orchestration, user monitoring, and therapist oversight within a unified architecture suitable for iterative real-world deployment.The benefit of this approach lies in the effective integration of artificial intelligence components, human-robot interaction, psychology, and the extensive experience of human experts. Interaction in a real-world environment and the labeling of test datasets occur simultaneously. The contribution is fourfold. First, we define a computer-robot-therapist-end-user architecture that explicitly models information flow, therapist control, and adaptive interaction. Second, we provide an experimental procedure for cognitive training sessions that includes training content, session organization, and a user-experience evaluation framework. Third, we introduce a data protection and ethics plan tailored to vulnerable populations, emphasizing informed participation, controlled data access, and responsible system adaptation. Fourth, we outline an approach for objective estimation of the end user’s cognitive state, with the longer-term goal of enabling robot behaviour that is responsive to fatigue, attention, overload, or engagement. Preliminary results indicate that socially guided robot-based training is feasible and acceptable within the proposed framework. A central ongoing challenge is to identify meaningful robot-user interaction events in the relation to their contexts (situations) and to map them to suitable user and robot actions. We therefore treat interaction-event modelling not as a secondary implementation detail but as a foundational design layer for adaptive socially assistive robotics. The longer-term significance of this work lies in connecting clinical and social needs with computational interaction design, making robot-led training more personalized, transparent, and operationally safe for adults with developmental deficits.

INTRODUCTION

Demographic changes (United Nations, 2024) and the situation of vulnerable groups (WHO, 2023) present society with the challenge of providing dignified and effective care for vulnerable groups, such as older people and people with developmental disabilities.

This work presents a socially intelligent robotic system for cognitive training of adults with developmental deficits in a therapist-half-supervised setting. The core idea is to move beyond one-way robot prompting toward a structured, ethically grounded, data-aware intervention framework in which the robot, end user, and therapist form a coordinated triad. The proposed system combines robot behaviour management, session orchestration, user monitoring, and therapist oversight within a unified architecture suitable for iterative real-world deployment. The benefit of this approach lies in the effective integration of artificial intelligence components, human-robot interaction, psychology, and the extensive experience of human experts. Interaction in a real-world environment and the labelling of test datasets occur simultaneously.

The goal of this research is to pave the way for a socially intelligent, situation-aware robotic system capable of independently leading cognitive training for adults with developmental deficits. In this regard, designing realistic usage scenarios that allow us to collect data in real-world settings is crucial. We propose a socially intelligent robotic system for cognitive training of adults with developmental deficits. The system integrates therapist oversight, adaptive robot behavior, and user monitoring within a unified architecture. We define interaction procedures, data protection measures, and an evaluation framework focused on user experience and cognitive outcomes. Preliminary findings suggest feasibility and acceptance of robot- led training.

The contribution is threefold. First, we define a computer-robot-therapist- end-user architecture that explicitly models information flow, therapist control, and adaptive interaction. Second, we provide an experimental procedure for cognitive training sessions that includes training content, session organisation, and a user-experience evaluation framework. Third, we introduce a data protection and ethics plan tailored to vulnerable populations, emphasising informed participation, controlled data access, and responsible system adaptation.

STATE OF THE ART Assistive Technologies

According to (WHO and UNICEF, 2022), Assistive technology (AT) is the application of organised knowledge and skills related to assistive products, including the systems and services required to provide them. Its purpose is to maintain or improve an individual’s functioning, independence, and well-being.

Social Robotics Social robotics is a branch of robotics concerned with the design, development, and study of robots that interact with humans through socially appropriate behaviours and communication. Social robots are designed such that social interaction is a fundamental component of their operation rather than merely an interface for control or task execution, see (Fong et al., 2003).

Cognitive Training for Adults With Developmental Deficits 485 Situation Awareness and Edge AI

Situation awareness (SA) refers to an individual’s or system’s ability to perceive relevant environmental elements, comprehend their significance in relation to current goals, and anticipate their future states. SA is considered a critical prerequisite for effective decision-making in dynamic and complex environments, see (Endsley, 1995a)(Endsley, 1995b).

Edge Artificial Intelligence (Edge AI) is a computing paradigm that integrates artificial intelligence with edge computing by performing data processing, inference, and decision-making directly on edge devices or nearby edge servers rather than in centralized cloud infrastructures, see (Deng et al., 2020)(Xu et al., 2022). This approach enables real-time responsiveness, reduced communication latency, and enhanced privacy (Gill et al., 2024).

Wizard of Oz Approach The Wizard of Oz (WoO) methodology is an experimental prototyping technique in which participants interact with a system they believe to be autonomous, while some or all of the system’s functionality is performed by a hidden human operator (the “wizard”). The method allows researchers to evaluate user interaction, system design, and behavioural responses before implementing the underlying intelligent algorithms (Kelley, 1984)(Riek, 2012)(Steinfeld et al., 2009). In this research, the WoO technique is also utilised to annotate interaction data in real time in real environment.

Ethics and Data Protection The proposed robotic system interacts with vulnerable populations of individuals with disabilities; ethical considerations extend beyond technical performance to include respect for human autonomy, dignity, privacy, transparency, and informed consent. Researchers emphasise that social robots should support users without fostering excessive dependence, deception, or loss of personal agency. (Sharkey and Sharkey, 2012)(Dautenhahn, 2007). Data protection is particularly important because social robots often collect sensitive information through cameras, microphones, physiological sensors, and behavioral monitoring systems.

MATERIAL AND METHODS User Group Gefinition

Target group consists of adults with moderate intellectual disabilities, a neurodevelopmental condition characterized by significantly reduced intellectual functioning (IQ approximately 35–49) and limitations in

adaptive behavior, including conceptual, social, and practical skills. These individuals usually require ongoing support in everyday life, communication, decision-making, and social participation, but many are able to develop stable routines, form meaningful relationships, and participate in structured work and community activities.

PROPOSED SOLUTION In this section, the proposed robot system architecture, cognitive training and user usability measurement procedure are presented.

System Architecture Architecture Aequirements

The system architecture requirements are based on the main robot task, which is to conduct cognitive training sessions independently of a human operator. These requirements include data protection, ethics-related considerations, and, of course, a cost-effective solution. Requirements are:

1. Simple usage in a real environment: the system will be used in a real environment by users not trained in technology use, as well as by human operators (therapists) who are not focused on technology. 2. Viable system in real life: the overall cost of the system should be low enough and is estimated not to exceed €8,000. 3. The robot is capable of: general conversation in English and Slovenian, artificial social intelligence (recognition of users’ stress, fatigue and other social signals, expression of social signals such as emotion via graphical communication), and situation awareness (following users’ actions, reacting to calendar events, etc.). 4. Language capabilities: communication between the robot and the user is bidirectional, and the system is capable of content recognition and user intention recognition. 5. Wizard of Oz approach for human supervision and real-environment data annotation is supported. This means the robot executes instructions from a human operator in real time, and these instructions are recorded. 6. Real-time data processing: data such as video and physiological signals are processed in real time to generate context, including triggers for the robot. 7. Data storage, protection and ethics: sensitive user data (video, physiological signals) is processed locally (using Edge AI principles). User events, system events and therapist instructions are time-stamped and stored locally. 8. Robot embodiment: the appearance and feel of the robot must be sufficiently acceptable for end users during data collection and in real environment usage.

Cognitive Training for Adults With Developmental Deficits 487 Proposed Architecture

The architecture is shown in Fig. 1. It supports experimental data collection (including the Wizard of Oz approach) and includes the Edge AI components required for real-time context generation, which is a key element of situation awareness in real environments. Further details are provided in the caption of Fig. 1. The main approach to data protection is: 1. Local data storage, and 2. Local data processing using the Edge AI approach.

Data Collection and Annotation Procedure User, contextual, and system data collection is automated by the robot app and the WoO app; As already indicated, to protect sensitive user data, the data is stored locally and not transmitted over any network. Data annotation is implemented using the WoO app, which is also used by the therapist to trigger robot events (such as what it says, what it displays on the graphical interface, and what physical movement, like raising the arm, it performs). Therefore, real-time data annotation is integrated into the monitoring process by the therapist.

Edge AI and Data Storage To protect sensitive user data, this information is never sent outside the collection site. Consequently, the Edgy AI approach is used for data processing, and data storage is available locally.

Data processing involves: 1. Real-time video processing (user’s face detection, emotion and stress recognition, skeletal detection to identify physical movements such as lifting a hand, and specific event detection such as ceasing to collaborate); 2. Processing of the user’s physiological signals (electrodermal activity, heart rate, and others) to enhance social signal recognition; 3. Processing of smart home sensor data (temperature, illumination, noise, etc.) to improve real-time context generation. All these

data are used to make the robot system more independent from human operators.

Cognitive Training Procedure The goals of cognitive training are to: 1. Stimulate users’ cognitive activity; 2. Activate their physical activity; and 3. Allow automatic estimation of selected cognitive characteristics of users based on technological measurements while the cognitive training is conducted by the robotic system. The framework for the cognitive training was chosen to be a children’s game called ”Dan noˇc” in Slovenian. To automatically and objectively estimate selected aspects of the current cognitive abilities of end users, the procedure should stimulate complex responses, including oral responses, as these are measurable and rich in information relevant to the user’s current status, such as emotion recognition and fatigue.

Cognitive Training Requirements To achieve these goals, the requirements for the cognitive training procedure are as follows:

1. The user must follow the time sequence of instructions and perform actions; 2. The procedure must be feasible and acceptable for the designated end user group; 3. The user’s actions must be observable (by humans and the robotic system); 4. The user’s successes and failures in responding to instructions must be measurable; 5. The user’s complex responses (physical movement, oral responses) are required and encouraged; 6. To keep users properly motivated, the training should be neither too easy nor too difficult. 7. The procedure must be gamified and enjoyable for end users; 8. The procedure must be feasible for both the human operator and the robotic system and compliant with requirements posed.

As users’ performance improves over time, the training procedure must adapt accordingly, continuously taking their performance into account and gradually increasing—or, when necessary, decreasing—the level of difficulty.

Proposed Cognitive Training Procedure The cognitive training procedure can be led by a human, a robotic system, or a combination of both using the Wizard of Oz approach, with the leader referred to as such here.

Cognitive Training for Adults With Developmental Deficits 489 The task is based on the Day–Night paradigm, which targets inhibitory control and attention. Participants begin with their hands in a resting position. When the experimenter says “Day”, the participant raises both hands. When the experimenter says “Night”, the participant lowers both hands. For participants with intellectual disabilities, visual cues are strongly recommended. For example, a large sun on a light background can represent “Day,” while a moon and stars on a dark background can represent “Night.” Before the test phase, participants complete 4–6 practice trials at a slow pace to ensure that the rules are understood. During practice, errors are corrected immediately (e.g., “Remember: Night means hands down.”). The test phase then proceeds without feedback. During the test phase, the experimenter presents a sequence of “Day” and “Night” cues at a steady pace (e.g., one cue every two seconds). Several blocks of 10–20 trials are administered.

Usability Measurement Acceptable usability of the proposed systems is crucial for their application in real environments. To assess this, we need to measure each user group separately, namely end users (those who receive cognitive training), therapists (those who monitor and support the process and, if necessary, annotate data), and technical support. Only the first two user groups are covered here. We list candidate measurement instruments for each group. For end users candidates include The Rapid Assessment for Developmental Disabilities (Hom et al., 2018) and Leiter-3: Nonverbal Cognitive and Neuropsychological Assessment (Roid and Koch, 2017); for human operators: classic usability instruments such as SUS (Brooke, 1996), UMUX (Finstad, 2010), and UEQ (Laugwitz et al., 2008).

PRELIMINARY FINDINGS, CONCLUSSION AND FURTHER WORK Preliminary findings are based on three recorded sessions involving three end users and one human operator, a psychologist by profession, conducted at the “Varstveno delovni center” where our users reside regularly. The therapists led the training and made all decisions. The findings are as follows:

1. The minimal technical requirements of the system are met. This means data was recorded and annotated successfully. 2. Data protection was ensured by local storage only. 3. The robot’s appearance was acceptable and well received by the end users. 4. The general user experience perceived by the end users was good. 5. The general user experience perceived by the human operator was good. 6. Real-time data annotation using the WoO app was feasible when up to eight commands were used.

Preliminary results indicate that socially guided robot-based training is feasible and acceptable within the proposed framework. A central ongoing challenge is to identify meaningful robot-user interaction events in relation to their contexts (situations) and to map them to suitable user and robot actions. We therefore treat interaction-event modelling not as a secondary

implementation detail but as a foundational design layer for adaptive socially assistive robotics. The longer-term significance of this work lies in connecting clinical and social needs with computational interaction design, making robot-led training more personalized, transparent, and operationally safe for adults with developmental deficits.

ACKNOWLEDGMENT

This research was supported by project P2-0246 ICT4QoL - Information and Communications Technologies for Quality of Life.

REFERENCES

Aaron Steinfeld, Terrence Fong, David Kaber, Michael Lewis, Jean Scholtz, Alan Schultz, and Michael Goodrich. (2009) Common metrics for human-robot interaction. Proceedings of the ACM/IEEE International Conference on Human- Robot Interaction, pages 33–40, 2009. Aimee van Wynsberghe. (2021) Healthcare robots: Ethics, design and implementation.

Science and Engineering Ethics, 27(1):1–20, 2021. Alan Lindsay, Andres Ramirez-Duque, Ronald P. A. Petrick, and Mary Ellen Foster.

(2022) A socially assistive robot using automated planning in a paediatric clinical setting, AI-HRI Symposium at AAAI Fall Symposium Series (FSS). Amanda Sharkey and Noel Sharkey. (2012) Granny and the robots: Ethical issues in robot care for the elderly. Ethics and Information Technology, 14(1):27–40, 2012. Bettina Laugwitz, Theo Held, and Martin Schrepp. (2008) Construction and

evaluation of a user experience questionnaire. In HCI and Usability for Education and Work, volume 5298 of Lecture Notes in Computer Science, pages 63–76. Springer, 2008. Christina Hom, David M. Walsh, Eric W. Doran, and Ira T. Lott. (2018) The rapid assessment for developmental disabilities. 2018. Dianlei Xu, Tong Li, Yong Li, Xiang Su, Sasu Tarkoma, Tao Jiang, Jon Crowcroft,

and Pan Hui. (2020) Edge intelligence: Architectures, challenges, and applications. Frontiers of Computer Science, 16, 2020. Gale H. Roid and Christopher Koch. (2017) Leiter-3: Nonverbal cognitive and neuropsychological assessment. In R. Steve McCallum, editor, Handbook of Nonverbal Assessment, chapter 8, pages 127–150. Springer International Publishing, Cham, Switzerland, 2017. John Brooke. (1996) Sus: A “quick and dirty” usability scale. In Patrick W. Jordan,

Bruce Thomas, Bernard A. Weerdmeester, and Ian L. McClelland, editors, Usability Evaluation in Industry, pages 189–194. Taylor and Francis, London, 1996. John F. Kelley. (1984) An iterative design methodology for user-friendly natural language office information applications. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pages 26–41. ACM, 1984. Kraig Finstad. (2010) The usability metric for user experience. Interacting with

Computers, 22(5):323–327, 2010. Laurel D. Riek. (2012) Wizard of oz studies in hri: A systematic review and new reporting guidelines. In Journal of Human-Robot Interaction, volume 1, pages 119–136, 2012. Mica R. Endsley. (1995) Measurement of situation awareness in dynamic systems.

Human Factors, 37(1): 65–84, 1995a.

Cognitive Training for Adults With Developmental Deficits 491 Mica R. Endsley. (1995) Toward a theory of situation awareness in dynamic systems.

Human Factors, 37(1): 32–64, 1995b. Shuiguang Deng, Hailiang Zhao, Weijia Fang, Jianwei Yin, Schahram Dustdar, and Albert Y. Zomaya. (2020) Edge intelligence: The confluence of edge computing and artificial intelligence. IEEE Internet of Things Journal, 7(8):7457–7469, 2020. Sukhpal Singh Gill, Muhammed Golec, Jianmin Hu, Minxian Xu, Junhui Du,

Huaming Wu, et al. (2024) Edge ai: A taxonomy, systematic review and future directions. ACM Computing Surveys, 2024. Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. (2003) A survey of socially interactive robots: Concepts, design, and applications. Robotics and Autonomous Systems, 42(3–4):143–166, 2003. United Nations. (2024) World Population Prospects 2024. United Nations, New

York, 2024. WHO. (2023) Disability and health. https://www.who.int/news-room/fact-sheets/ detail/ disability-and-health, 2023. WHO and UNICEF. (2022) Global Report on Assistive Technology. WHO, Geneva,

Switzerland, 2022.

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET 2026)

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET 2026)

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 979-8-950676-12-3

Explore this ISBN in the AHFE Bookstore Search the bookstore catalogue for this ISBN; a listing is not guaranteed · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →