Luo Tongyu (2026). Design and Experience Improvement of Multi-sensory AI Products for Human-Computer Interaction with Autistic Children. In: Universal Design and Design for Inclusion. AHFE Open Access, vol. 209, pp. 150–159. https://doi.org/10.54941/ahfe1008029
Many intelligent products for children fail to meet the needs of those with autism, often due to single interaction modes and a lack of personalized feedback. To address this, we designed a multi-sensory AI system specifically for children with autism, aged 4–6 years. Our approach integrates human factors engineering with special education principles. The system uses a hybrid interface combining a smart terminal with physical building blocks that provide tactile, visual, and auditory feedback. An AI engine personalizes tasks and regulates sensory input. We evaluated the system with 40 autistic children in a controlled experiment. Results indicate that compared to a standard touch-screen game, our system significantly improved task completion rates and reduced completion time. It also increased effective interaction duration and reduced the frequency of sensory rejection behaviors. Children using our system showed a higher proportion of positive emotional responses. This work offers a practical framework for designing more accessible, adaptive AI tools for special education.
In the 2020s, artificial intelligence has driven rapid development in children’s intelligent education hardware, transforming interaction from passive instruction execution to natural, personalized human-computer communication. However, children aged 2–7 show concrete thinking, sensory dependence, and limited attention according to Piaget’s cognitive development theory, and simply childifying adult product design leads to usability problems such as interface confusion, low speech recognition satisfaction, and insufficient physical tactile experience. Multi-sensory interaction is highly consistent with children’s natural cognition and can improve processing efficiency while reducing cognitive load, and AI further enables dynamic and personalized feedback by perceiving children’s real- time status. Existing research has verified the value of multi-sensory interaction in digital intervention products for autistic children supported by AR and somatosensory technology, physical sensory training tools, and occupational therapy and educational paradigms; scholars including Yin et al. (2020), Jan (2024), and Sun (2025) have explored its application
Design and Experience Improvement of Multi-Sensory AI Products 151 in social skill training, sensory regulation, and rehabilitation for children with autism spectrum disorder and other developmental difficulties, and enterprises such as Southpaw Enterprises have also launched related sensory products. Nevertheless, current research still has obvious gaps: insufficient integration of AI and multi-sensory interaction, lack of a systematic design framework for autistic children, insufficient quantitative empirical evidence, and few targeted multi-sensory feedback design guidelines. Therefore, this study combines multi-sensory fusion and AI technology to construct a systematic design framework, develop a product prototype for 4–6-year-old autistic children, and verify its effectiveness through experiments, so as to improve interaction experience and promote the intelligent and personalized development of special education and rehabilitation.
AI-ENABLED MULTISENSORY FUSION INTERACTION DESIGN FRAMEWORK Design Philosophy of the Framework
The framework starts from a simple premise: the child, not the technology, should be the center of the design process. This means three things in practice. First, the interaction rhythm must match the child’s information processing speed, not the other way around. Second, the system should tolerate and even encourage trial and error, rather than forcing children onto a single “correct” path. Third, individual differences matter. A design that works for one autistic child may overwhelm another. These considerations led us to prioritize adaptability over fixed interaction rules.
Multi-sensory integration in this framework goes beyond simply adding more channels of input or output. We focused on coordination across three dimensions: temporal (feedback from different channels must arrive at the same time), semantic (different channels should convey complementary, not conflicting, information), and intensity (no single channel should dominate or over-stimulate). The main innovation is embedding AI into the interaction loop so that the system can adjust its multi-sensory feedback based on real- time observations of the child’s state. This moves beyond simple rule-based responses toward genuine adaptation.
Framework Structure and Core Elements The framework consists of three layers arranged in a closed loop.Input layer: This layer handles multi-modal sensing, including touch, voice, vision, and environmental context. Data from different sources are fused to build a more accurate picture of the child’s behavior and state. We made specific design choices to accommodate the sensory sensitivities common among autistic children, such as using lower microphone gain and softer visual contrast where appropriate.
Processing layer: This is the decision-making core. It takes fused data from the input layer and performs three functions: intent recognition (what is the child trying to do?), emotion classification (how is the child feeling?), and personalization (what does this child need right now?). The personalization component tracks each child’s age, skill level, and interaction history to
152 Tongyu generate adaptive feedback strategies. We note that this layer faces inherent trade-offs. For example, more accurate emotion recognition requires more data and longer processing time, which can delay feedback. Our current implementation prioritizes speed over precision, based on the observation that autistic children are particularly sensitive to response delays.
Output layer: This layer generates coordinated visual, auditory, and tactile feedback. Temporal synchronization is critical here. If a visual cue appears noticeably before or after a corresponding vibration, the child may perceive two separate events rather than one integrated experience. We implemented a central timing controller to keep all channels aligned within 50 milliseconds.
The three layers form a continuous perception-decision-feedback loop. As the system collects more interaction data from a given child, the personalization module gradually improves its predictions. This allows the product to evolve with the child’s development rather than becoming obsolete after a few months.
Application Value of the Framework For designers and researchers, this framework provides a way to decompose a complex interaction system into analyzable modules. The input layer raises questions about what sensors are needed and how to handle noisy or missing data. The processing layer forces explicit decisions about which child states to recognize and how to respond. The output layer requires careful engineering of feedback timing and intensity.
However, several practical limitations should be noted. First, the framework assumes reliable real-time sensing, which is difficult to achieve with young children who move unpredictably. Second, the processing layer’s personalization algorithms require substantial amounts of user data to work well, creating a cold-start problem for new users. Third, the closed-loop design increases system complexity and cost, which may not be feasible for all product contexts.
DESIGN PRACTICE: DESIGN OF MULTISENSORY FUSION AI GAME SYSTEM FOR AUTISTIC CHILDREN Design Background and Target User Positioning
The global prevalence of Autism Spectrum Disorder (ASD) is about 0.76%, and the prevalence in China ranges from 0.7% to 1%, showing an upward trend year by year. At present, there are more than 2 million autistic children aged 0 to 14 in China, with an annual increase of about 200,000. Due to the lack of effective therapeutic drugs, ASD has a high disability rate, bringing heavy economic and psychological pressure to patients’ families. In this context, exploring effective intervention support for autistic children through human-computer interaction technology is of great social significance.This design practice focuses on the core obstacle of autistic children, namely language and social barriers. For this group, the game system design must fully consider their special behavioral characteristics: they are prone to show extraordinary investment and emotional attachment to specific themes, which can be transformed into design advantages; they are likely to be in a daze for a long time or hyperactive during the game and need assistance
Design and Experience Improvement of Multi-Sensory AI Products 153 to complete actions; they are prone to emotional breakdown leading to task interruption; they are likely to lack a sense of role identity and find it difficult to truly engage in role-playing. The main reasons for the lack of role identity include cognitive disconnection, narrow interests, environmental interference, etc., and their strong interest in brightly colored images can provide a clear direction for visual design.
Theoretical Intervention Framework and Design Integration This system integrates a variety of empirically supported autism intervention theories. Applied Behavior Analysis (ABA) is the core methodology, which decomposes target tasks into a series of small steps and uses appropriate reinforcement methods for gradual training until children master all steps and can apply them independently. Game-based teaching provides a low-pressure learning environment where trial and error have no negative consequences, fun maintains participation motivation, and structured characteristics reduce anxiety. The concept of art therapy promotes emotional regulation through creative expression and sensory experience, providing a non-verbal expression channel for children with limited language skills. Based on the above framework, the role-playing game design is divided into four categories: cognitive education, life situation, career experience and emotional socialization.
System Architecture and Interaction Process Design To verify the interaction design framework proposed in Chapter 2, this study designed and developed a prototype of the “Star Island” AI interaction system for autistic children. The target users are autistic children aged 4 to 6, and the product is positioned as an intelligent rehabilitation auxiliary system integrating ABA intervention, gamified learning and emotional companionship, adopting a “smart terminal + physical building block sensing” hybrid interaction mode. The hardware configuration includes a 7-inch touch screen smart base, NFC sensing building block modules, dual microphone array, front camera and stereo speakers. The design goals are carried out from three levels of user experience: at the visceral level, it attracts children to contact through the space island theme, rounded shape and high-saturation colors; at the behavioral level, it decomposes complex skills into micro-steps through task decomposition strategies and reinforces correct behaviors with multisensory feedback; at the reflective level, it helps children establish a sense of role identity by recording growth trajectories and integrating them into game plots.
The overall interaction process of the system starts with user positioning. After startup, the screen displays two options: parent-therapist mode and child mode. The parent mode provides professional functions such as intervention goal setting and task difficulty adjustment, while the child mode directly enters the main game interface. AI personalized game content generation is the core innovation of this system. The AI engine generates a highly personalized game world and task sequence based on the intervention goals set by parents, children’s historical behavior data, life experience records and ability development evaluation. Based on ABA, each social skill is decomposed
154 Tongyu into several micro-steps and presented gradually from easy to difficult. For example, “greeting others” is decomposed into five steps: looking at the other person, recognizing the greeting, imitating the response, taking the initiative to greet, and greeting in combination with the context. The system automatically records progress and advances the difficulty in a timely manner.
Figure 1: Diagram of the smart interactive experience.
Tactile Interaction Design Combining Virtual and Real “Star Island” deeply couples physical NFC building blocks with the on-screen virtual world, which is consistent with the cognitive characteristics of autistic children. The whole system is equipped with more than 20 types of NFC building blocks of different shapes, colors and materials, and the physical form of the building blocks forms a natural mapping with their functional meanings: the water drop shape is used for watering and hand washing, the heart shape is used for expressing emotions, and the star shape is used for rewarding achievements. When a child picks up a building block and puts it into the sensing slot, the LED light strip at the edge of the slot flashes softly to provide visual guidance, the slot makes a slight click to provide auditory confirmation, the linear vibration motor generates pulse vibration to deliver tactile confirmation, and the screen changes visually synchronously. AI dynamically responds to building block interaction, adjusting the form and intensity of feedback according to the child’s mastery level. When learning a new skill, clear voice prompts and highlight guidance are provided, and the support is gradually withdrawn as the child masters the skill, realizing the “prompt fading” principle in ABA.
Figure 2: Illustration of a smart interactive product.
Design and Experience Improvement of Multi-Sensory AI Products 155 Progressive Design for Role Identity Construction
To address the problem of lack of role identity, the system designs a progressive construction path. The first stage is role perception, where children form an initial impression through brightly colored visual images and gentle voice introductions. The second stage is role understanding, where children are helped to understand the behavioral patterns of roles through decomposed task sequences. For example, “seeing a doctor” is decomposed into steps such as greeting, using a stethoscope, asking about the condition, prescribing medicine, and accepting thanks. The third stage is role identification, where children are guided from “operating a role” to “becoming a role” through first-person perspective task design, allowing children to decide what to do next in the doctor game. The fourth stage is role transfer, which connects game roles with children’s real-life experiences. For example, special tasks are generated according to the hospital experiences uploaded by parents, making children feel that they can also become “little doctors” in reality.
In terms of emotional intervention, the system designs a dynamic response mechanism based on multimodal emotion recognition, integrating data such as facial expressions, operation intensity and voice characteristics to evaluate children’s emotional states. When an impending emotional breakdown is detected, the sensory load is first reduced through environmental adjustment; if it is not alleviated, the task is simplified to help children rebuild confidence; if necessary, comfort intervention is activated to make the game character suspend the task and switch to the companion mode; in extreme cases, parents are notified to intervene. The parent-therapist terminal equipped with the system provides functions such as data tracking, intervention suggestions, life experience records and family extension activity library, helping parents transfer the intervention effect from the game scenario to daily life.
Figure 3: Game flow and logic diagram.
EXPERIMENTAL VERIFICATION AND RESULT ANALYSIS Experimental Design
To scientifically evaluate the interaction efficiency and user experience of the “Star Island” system compared with traditional single-sensory children’s products, this study designed and implemented a controlled experiment. The experiment was carried out in Changsha, Hunan Province, and a total of 40 autistic children aged 4–6 were recruited. The inclusion criteria included a confirmed diagnosis of ASD by a Grade A tertiary hospital, basic grasping and touch operation abilities, and no severe hearing or visual impairment. A paired design was adopted. After matching according to children’s age and CARS score, they were randomly divided into an experimental group and a control group, with 20 people in each group. The experimental group used the “Star Island” multisensory AI system designed in this study, while the control group used a common single- touch screen children’s tablet game on the market.
All subjects were required to complete two standardized tasks. Task A was a basic operation, which required putting building blocks of corresponding colors and shapes into designated areas according to on-screen prompts, while the control group dragged virtual patterns to corresponding areas on the screen. Task B was a simple social imitation, which required following the system’s guidance to complete a short social story including gazing, greeting and responding.
Experimental Results and Analysis All experimental data were analyzed using independent-sample t-tests in SPSS 26.0. Prior to analysis, all dependent variables were tested for normality using the Shapiro-Wilk test (p > .05 for all measures), and Levene’s test confirmed homogeneity of variance. Table 1 presents the comparison of operational performance indicators between the two groups.
Table 1: Comparison of operational performance indicators between experimental group and control group.
Table available in the original PDF paper.
Indicators Experimental Group Control Group Improvement Operation Completion Rate 95.2% 78.0% +22.1%
Average Task Completion Time 42.5 seconds 68.3 seconds –37.8% Average Error Operations 1.8 times 5.3 times –66.0%
As shown in Table 1, the experimental group achieved a completion rate of 95.2%, while the control group completed 78.0% of the operations. The difference was statistically significant (t(38) = 4.21, p < .01). Regarding task completion time, the experimental group finished tasks in 42.5 seconds on average, compared to 68.3 seconds for the control group (t(38) = –5.63, p < .001). For error operations, the experimental group averaged 1.8 errors, whereas the control group averaged 5.3 errors (t(38) = –6.12, p < .001).
Table 2: Shows the results for interaction engagement and emotional experience.
Table available in the original PDF paper.
Indicators Experimental Group Control Group Improvement 286.5 seconds 170.2 seconds +68.3% Average Effective Interaction Duration
31.5% 18.2% +73.1%
Proportion of Positive Emotion Duration
Reduced by over 65% Baseline –65.0%
Frequency of Sensory Rejection Behaviors In Task B, children in the experimental group maintained effective interaction for an average of 286.5 seconds, while the control group averaged 170.2 seconds (t(38) = 5.89, p < .001). Facial expression analysis revealed that the experimental group spent 31.5% of interaction time showing positive emotions, compared to 18.2% in the control group (t(38) = 4.73, p < .01). Observational records alsoindicated that sensory rejection behaviors occurred substantially less frequently in the experimental group compared to the control group, with an estimated reduction of approximately 65%.
Discussion of Results The quantitative results support several conclusions about the effectiveness of the proposed system. First, at the operational level, the combination of physical tactile feedback with visual and auditory cues appeared to reduce cognitive load for autistic children. This may explain the higher completion rate and fewer errors observed in the experimental group. Second, at the engagement level, the dynamic and coordinated multi-sensory feedback seemed to sustain children’s attention for longer periods. The substantial difference in effective interaction duration (286.5 seconds vs. 170.2 seconds) suggests that the physical building block interaction was more engaging than touch-screen only interaction. Third, at the emotional level, the AI-driven emotion recognition and regulation mechanism likely contributed to the higher proportion of positive emotions and lower frequency of sensory rejection behaviors. These findings are consistent with existing research on multi-sensory intervention for children with autism spectrum disorder. Overall, the experimental results provide empirical support for the AI-enabled multi-sensory fusion interaction design framework proposed in this study. However, it is important to note that these results are based on a single session with a relatively small sample size. Longer-term effects and generalization to home or classroom settings remain to be tested.
CONCLUSION
This study set out to address a practical problem: how to design AI-enabled multi-sensory products that better serve the needs of autistic children. We proposed a child-centered, four-layer design framework consisting of input, processing, and output layers that form a dynamic “perception-decision- expression” loop. Applying this framework, we developed the “Star Island” system, which uses virtual-real NFC building block interaction, step-by-step
158 Tongyu task decomposition, and a progressive role identity construction mechanism. A controlled experiment with 40 autistic children showed that our system outperformed a standard touch-screen game on multiple measures, including task completion rate, completion time, error rate, effective interaction duration, and positive emotional expression. These findings suggest that the integration of physical tactile feedback with AI-driven personalization is a promising direction for autism intervention tools.
Several limitations of this study should be acknowledged. First, the sample size was modest (N = 40) and all participants were recruited from a single city in China. This limits the generalizability of the findings to broader populations or different cultural contexts. Second, the experiment measured only immediate performance and emotional responses. We do not know whether the observed benefits would persist over weeks or months of use. Third, the controlled laboratory setting does not fully replicate the complexity of real-world environments such as homes or classrooms. Fourth, we did not include a long-term follow-up assessment, so the durability of the intervention effects remains unknown.
Future research should address these limitations by conducting longitudinal studies with larger and more diverse samples, ideally in naturalistic settings. Cross-cultural comparisons would help determine whether the design principles generalize beyond the Chinese context. Additionally, future work could explore the neural mechanisms underlying the observed behavioral improvements, for example using EEG or fMRI to measure changes in sensory processing and social attention. Ethical considerations around data privacy and the potential for over-reliance on AI systems should also be carefully examined as this technology moves toward practical deployment.
REFERENCES
ASU London. (2025, July 31). Living Labs: Designing sensory toys for neurodivergent children. ASU London News. https://asu-london.ac.uk/news/living-labs-designing- sensory-toys-for-neurodivergent-children/ Enabling Devices. (2025, August 28). 3 Places to Buy Sensory Toys Online for Special
Needs Students [Sponsored content]. TEACH Magazine. https://teachmag.com/3- places-to-buy-sensory-toys-online-for-special-needs-students/ Makarci. (2025). Stepping Stones for Kids 12pcs Obstacle Courses Play Indoor Outdoor [Product listing]. Woot. https://home.woot.com/offers/stepping-stones- for-kids Piller, A., Pfeiffer, B., Hebert, D., Kuhaneck, H., & Brown, L. (2025). Systematic review
of sensory-based interventions for children and youth (2015–2024). Frontiers in Pediatrics, *13*, 1720179. https://doi.org/10.3389/fped.2025.1720179 Southpaw Enterprises Inc. (n.d.). Company Profile. Dun & Bradstreet. Retrieved March 19, 2026, from https://www.dandb.com/businessdirectory/southpawenterprisesinc.- moraine-oh-15285302.html Sun, L. (2025). Children’s sensory interaction music enlightenment system based on
sensory interaction and game elements. Entertainment Computing, *52*, 100873. https://doi.org/10.1016/j.entcom.2024.100873 Yang, G. (2025, August 26). Gamified Exercise Program for Children with Sensory Processing Challenges (Clinical Trial Registration). Northeast Normal University. https://ichgcp.net/clinical-trials-registry/550758-design-of-a-sensory-integration- intervention-program
Design and Experience Improvement of Multi-Sensory AI Products 159 Yin, X., Hou, S., & Hu, H. (2020). Research on Interactive Design of Social Interaction
Training APP for Children with Autistic Spectrum Disorder (ASD) Based on Multi-modal Interaction. 2020 International Conference on Innovation Design and Digital Technology (ICIDDT), 1-6. (Note: This is an inferred conference publication based on the title and year; specific page numbers may vary.)
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Universal Design and Design for Inclusion
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-85-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional