Minglei Chen (2026). Design and Evaluation of a Multimodal Feedback VR Exergame for Individuals with Low Vision. In: Metaverse, Virtual Environments and Game Design. AHFE Open Access, vol. 217, pp. 153–163. https://doi.org/10.54941/ahfe1007662
People with partial vision loss often face limitations in psychological healing and physical exercise due to insufficient visual ability and inadequate interaction design. Existing solutions largely focus on superficial accessibility improvements, while most VR systems remain visually dominant, limiting usability and experience for this group. To address this gap, we present Light of Sound Healing, a VR exergame designed for partially sighted users. Guided by multisensory substitution interaction, the system employs spatial audio as the primary medium for navigation and interaction, translating auditory cues into directional and distance signals. Players locate audio targets and perform physical actions within an immersive environment. The design integrates inclusive design, serious games, and art therapy, with progressive challenges introduced through variations in target size, quantity, and movement. Findings indicate that a multimodal feedback system combining spatial audio, visual cues, and haptic vibration enhances perception and immersion. This approach highlights the potential of auditory-centric interaction for accessible VR and digital rehabilitation. Overall, this study proposes an auditory-centered VR interaction framework that improves engagement and movement experience for visually impaired users, while offering design insights for inclusive immersive systems.
Globally, many individuals with partial visual impairment face challenges in daily life, social participation, and recreation, while current rehabilitation mainly relies on medical treatments and assistive technologies like screen readers and magnifiers (Lewis, 2021). These methods mainly focus on compensatory skills, often neglecting the combined benefits of psychological support and physical activity, which research shows are crucial for reducing stress, boosting confidence, promoting social engagement, and improving overall quality of life (Saxena et al., 2013). However, interventions integrating psychological healing with physical activity for partially visually impaired individuals remain limited (Shinohara & Wobbrock, 2016). Most existing approaches often lack sustained engagement and immersive experience, thus there is a growing need for innovative digital solutions that integrate psychological and physical rehabilitation (Slater & Sanchez-Vives, 2016).
154 Minglei Such approaches have the potential to provide more engaging experiences, representing the direction for future research (Deterding et al., n.d.).
Serious games and VR are increasingly applied in rehabilitation, combining engaging gameplay with therapeutic goals (Deterding et al., 2011) and offering immersive, controlled environments that support relaxation, sensory stimulation, and exercise for visually impaired users (Emmelkamp & Meyerbröker, 2021). However, existing VR accessibility solutions still rely heavily on visual information or focus on isolated functions, with spatial audio often treated as a secondary cue rather than a primary interaction modality (Jain, Junuzovic, Ofek, Sinclair, R. Porter, et al., 2021). Consequently, the potential of sound for guiding movement, fostering emotional engagement, and supporting embodied interaction remains underexplored (Chang et al., 2024; India et al., 2021), highlighting the need to further investigate audio- centered interaction approaches in VR to create more inclusive and engaging experiences for visually impaired users (Yamagami et al., 2022).
Auditory interaction provides visually impaired users with non-visual digital access, with research showing that spatial audio and sound-based interfaces effectively convey direction, distance, rhythm, and context without relying on vision (Siu et al., 2020). Beyond information delivery, sound also evokes strong emotional responses, enhancing immersive and therapeutic experiences (Dang & Lee, 2024). In VR, spatial audio enables 3D soundscapes for perceiving position and movement, supporting sound- based navigation and interaction (Jain, Junuzovic, Ofek, Sinclair, Porter, et al., 2021). This enables sound-based navigation and interaction, expanding the possibilities for non-visual engagement (Jain, Junuzovic, Ofek, Sinclair, Porter, et al., 2021). Such characteristics open new avenues for designing VR experiences that integrate physical movement, rhythmic perception, and emotional expression (Xia et al., 2024). By making auditory interaction the primary channel, VR games can create inclusive spaces for visually impaired users, supporting both physical activity and emotional immersion (Dang & Lee, 2024). This design approach offers a promising direction at the intersection of auditory perception and rehabilitation-oriented interaction (Chang et al., 2024).
RELATED WORK
Providing Accessible Virtual Reality for Visually Impaired Users Accessibility in VR is an important topic in HCI, especially for users with visual impairments (Siu et al., 2020). Traditional solutions such as screen readers and high-contrast interfaces are designed for desktop and mobile systems and do not translate well to immersive VR, where spatial perception and visual feedback are central (Li et al., 2021). As a result, visually impaired users often face significant barriers when engaging with immersive experiences. To address this, researchers have explored non-visual interactions, including auditory cues, haptic feedback, and multimodal approaches for navigation and spatial understanding (Swaminathan et al., 2021). While their results show improvements in spatial understanding, the
Design and Evaluation of a Multimodal Feedback 155 interaction remains relatively simple and lacks task-driven engagement. In addition, Brock et al. proposed a multimodal VR interaction system that integrates spatial audio and haptic vibration to assist users in recognizing virtual objects and environmental structures. Although multimodal feedback significantly enhances environmental comprehension, the system mainly emphasizes object recognition rather than immersive interaction (Brock et al., 2015a). Walker et al. (2006) also developed an audio-based navigation assistance system, demonstrating that dynamic auditory cues can improve directional performance in VR, however, it is primarily limited to navigation tasks. Overall, while existing work establishes a foundation for accessible VR, it emphasizes functional assistance over immersive and engaging experiences. Integrating auditory interaction, multimodal feedback, and gamification to support physical participation and engagement remains a key direction for future research.
Serious Games and Virtual Reality Used for Rehabilitation Recently, serious games combined with VR have been widely applied in rehabilitation and health promotion, using immersive environments and game mechanics to support physical training while enhancing engagement and long-term adherence (Deterding et al., 2011). For instance, VR-based task- oriented training has been shown to support motor recovery and improve engagement in stroke rehabilitation, though such systems still rely heavily on visual feedback, limiting non-visual interaction (Laver et al., 2017). Similarly, Lohse et al. (2014) reviewed the effectiveness of virtual reality–based rehabilitation systems, showing that such interventions can improve balance, coordination, and user motivation, but still relying primarily on visual feedback for interaction. Powers et al. (2008) demonstrated that immersive virtual environments can reduce anxiety and stress through therapeutic interventions, although these approaches remain largely centered on visual experiences. From an accessibility perspective, simplified interaction design can improve participation for older adults and users with limited abilities, and multimodal feedback can support spatial understanding for visually impaired users, though primarily in navigation contexts (Gerling et al., 2012).
Audio-Based Interaction and Spatial Audio Navigation Auditory interaction, as a key non-visual interface, has gained increasing attention in accessibility and immersive VR research (Gerling et al., 2012). Spatial audio technologies simulate how sound propagates in three-dimensional space, enabling users to perceive the direction, distance, and movement of virtual objects through auditory cues (Jain, Junuzovic, Ofek, Sinclair, Porter, et al., 2021). This allows users to perform navigation and orientation tasks even in the absence of visual information (Siu et al., 2020). Prior studies show that directional audio effectively supports spatial understanding and decision- making, making it widely used in navigation and assistive systems. For example, For example, Walker et al. (2006) used sound beacons in a Virtual Auditory Display to guide navigation, improving efficiency and orientation, though interaction was limited to pathfinding. Similarly, Lokki et al. (2005)
156 Minglei developed a spatial audio interface for 3D navigation, enhancing localization but focusing mainly on information delivery rather than interactive experiences. Brewster et al. (1993) demonstrated that variations in pitch and rhythm can guide user actions and improve efficiency, though primarily in interface-level interactions. Other work combines spatial audio with haptics to support spatial understanding, but remains centered on environment exploration rather than task-driven interaction (Swaminathan et al., 2021).
DESIGN
Figure 1: Main image.
Design Rationale Light of Sound Healing is a VR-based exergame for partially visually impaired users, grounded in multisensory substitution interaction. It challenges vision- dominant paradigms by positioning sound as the primary perceptual channel, creating an immersive auditory-centered environment that supports physical activity and emotional regulation. Unlike conventional VR rehabilitation systems, the design employs spatial audio and rhythm-based guidance for navigation, interaction, and feedback, integrating artistic therapy with exergaming to enhance engagement and sustained participation. The system comprises two components: an artistic therapy module that uses sound and immersion to promote emotional release and self-awareness, and a physical training module informed by sensory compensation theory, featuring abstract visuals with reduced visual load and directional audio targets that guide movement. This auditory-driven approach not only accommodates the perceptual needs of partially visually impaired users but also offers a novel multimodal exercise experience for sighted users.
Design and Evaluation of a Multimodal Feedback 157 Interaction Workflow
The proposed VR exercise game adopts a structured interaction workflow supported by multimodal feedback to ensure intuitive navigation and accessibility for partially visually impaired users. As shown in Figure 2, the process includes menu navigation, level progression, gameplay interaction, performance feedback, and system control. Upon entering the game, players access a main interface with options such as New Game, Load Game, and Volume Settings, navigating via controller input with real-time voice guidance announcing selections to support low-vision users. The game is organized into three levels, each introducing a single variable to gradually increase difficulty: target size (Level 1), number of targets (Level 2), and target movement (Level 3), with multiple stages in each level requiring sequential completion. During gameplay, players enter a virtual environment where they locate auditory targets using spatial audio cues, with increasing sound intensity indicating proximity. Players move toward sound sources and trigger actions via the controller to eliminate targets. Each stage is completed by eliminating a required number of targets within a time limit, after which the system provides multimodal feedback, including visual and auditory prompts, allowing players to proceed or return. If unsuccessful, players receive retry prompts and can restart or exit. Throughout gameplay, system controls such as pause and exit are available, with progress automatically saved and restored upon re-entry, ensuring continuity and usability.
Figure 2: Interaction workflow.
Technical Implementation As illustrated in Figure 3, the system is developed in Unity and integrates spatial audio, motion tracking, and multimodal feedback, delivered via a PICO VR headset. Spatial audio serves as the core component, simulating three-dimensional sound propagation to convey direction and distance. Parameters such as sound source position, volume, and frequency are carefully calibrated to enhance localization accuracy and create clear spatial distinctions. For interaction, the system uses the headset and controllers to capture real-time movement data, including body position, orientation, and hand gestures, which are mapped into the virtual environment to detect proximity to sound targets and alignment with rhythmic cues. To enhance
158 Minglei immersion, a multimodal feedback mechanism provides synchronized auditory, visual, and haptic responses when users perform correct actions, reinforcing interaction awareness and engagement. From a system optimization perspective, particular attention is given to sound localization accuracy and interaction latency. By refining audio processing and motion detection, the system minimizes delay between user actions and feedback, ensuring smooth, natural, and reliable interaction.
Figure 3: Technology roadmap.
Design Details As shown in Figure 4, the visual design is inspired by cellular division processes, including splitting, recombination, and fusion. These biological dynamics are abstracted into flowing light and organic color patterns. Within the VR environment, this micro-level visual metaphor creates a pure and vibrant energy field, guiding users into a focused and calm therapeutic experience through minimalistic and inclusive visual expression. As shown in Figure 5, the gameplay system combines task-driven interaction with rhythm-based engagement. Players are required to locate and strike sound targets within a limited time. As levels progress, the size, number, and movement patterns of auditory targets increase in complexity. At the same time, players are required to perform more intensive physical movements, thereby supporting
Design and Evaluation of a Multimodal Feedback 159 exercise and motor training objectives. As shown in Figure 6, The system employs a layered multimodal feedback mechanism, integrating auditory, visual, and haptic cues. Auditory feedback provides directional guidance and rhythm cues, visual effects enhance environmental atmosphere and signal success, and haptic vibration delivers clear tactile confirmation during key interactions. This combination not only improves usability but also enhances emotional engagement and immersion. As shown in Figure 6, The volume settings interface allows users to independently adjust two audio components: auditory target signals and impact sound effects. The adjustment process is accompanied by a visual animation inspired by cellular division, providing intuitive feedback that reflects changes in volume levels.
Figure 4: Interface UI element design.Figure 5: Different level maps.Figure 6: Haptic feedback design diagram.Figure 7: Custom volume design diagram.
DISCUSSION
Accessible VR for Psychological Well-being and Physical Training in Partially Visually Impaired Users This study proposes an accessible virtual reality (VR) system that integrates psychological well-being and physical training for partially visually impaired users. Existing research in VR accessibility and rehabilitation primarily focuses on functional support, such as spatial navigation training or basic environmental exploration, often neglecting the combination of emotional engagement and embodied physical activity (Jain, Junuzovic, Ofek, Sinclair, Porter, et al., 2021). While these approaches demonstrate the effectiveness of virtual environments in improving spatial cognition, they provide limited support for holistic rehabilitation that includes both mental health and physical exercise. To address these gaps, the proposed design combines auditory-guided interaction with immersive environments to support both emotional regulation and bodily movement. By incorporating relaxation- oriented scenarios alongside movement-based tasks, the system enables users to engage in therapeutic experiences that promote stress reduction, confidence building, and physical activity simultaneously. This integrated approach responds to the limitations identified in prior work by extending VR applications beyond task-oriented training toward a more comprehensive and user-centered rehabilitation framework.
Integrating Serious Games and Gamified Interaction into VR Design This study incorporates serious game principles and gamified interaction mechanisms into virtual reality (VR) design to enhance user engagement and participation. Prior research in VR-based rehabilitation demonstrates that immersive environments can support motor training and recovery; however, many systems rely on repetitive, task-oriented exercises or visually guided interactions, which may reduce long-term motivation and limit accessibility for visually impaired users (Laver et al., 2017). Although these approaches improve functional outcomes, they often lack engaging interaction structures that sustain user interest over time. To address these limitations, the proposed design embeds game mechanics such as task progression, reward systems, and interactive challenges into the VR experience. By transforming rehabilitation tasks into goal-oriented and enjoyable activities, the system promotes intrinsic motivation and continuous participation. This approach extends beyond traditional training models by emphasizing user experience and engagement, while also reducing reliance on visual feedback. In doing so, it offers a more inclusive and sustainable framework for VR-based rehabilitation, aligning therapeutic objectives with engaging and accessible interaction design.
Audio-Centered Interaction and Spatial Audio Navigation in VR This study introduces an audio-centered interaction paradigm that leverages spatial audio as the primary modality for interaction in virtual reality. Existing research on auditory interfaces and VR accessibility shows that sound can effectively convey spatial information such as direction and distance,
Design and Evaluation of a Multimodal Feedback 161 supporting navigation and environmental awareness for visually impaired users (Lokki & Grohn, 2005; Walker & Lindsay, 2006). However, in most prior systems, audio is treated as a supplementary cue, mainly used for navigation or simple feedback, with limited support for complex interaction or embodied activities. To address these limitations, the proposed design employs spatial audio not only for navigation but also as a core mechanism to guide movement, structure tasks, and enhance emotional engagement. By integrating rhythm, dynamic sound cues, and multimodal feedback, the system enables users to interact with the virtual environment through hearing and bodily movement rather than vision. This approach expands the role of auditory interaction from passive information delivery to active experience design, providing a more intuitive and immersive interaction model. It contributes a novel framework for accessible VR systems by combining spatial audio, gameplay mechanics, and embodied interaction.
CONCLUSION
This study addresses the need for accessible VR experiences for partially visually impaired users, as existing approaches rely on visual interaction or single functions, limiting engagement. We propose an audio-centered VR design that integrates psychological healing and physical training through gamified interaction, using spatial audio as the primary modality to guide movement and evoke emotional engagement without vision. This approach expands auditory interaction in VR and offers a novel framework for inclusive, immersive rehabilitation, highlighting a promising direction for accessibility-focused HCI research.
REFERENCES
Brewster, S. A., Wright, P. C., & Edwards, A. D. N. (1993). An evaluation of earcons for use in auditory human-computer interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’93, 222–227. https://doi.org/10.1145/169059.169179 Brock, A. M., Truillet, P., Oriola, B., Picard, D., & Jouffrais, C. (2015). Interactivity
improves usability of geographic maps for visually impaired people. Human– Computer Interaction, 30(2), 156–194. https://doi.org/10.1080/07370024.2014 .924412 Chang, R.-C., Hung, C.-S., Chen, B.-Y., Jain, D., & Guo, A. (2024).
SoundShift: Exploring sound manipulations for accessible mixed-reality awareness. Designing Interactive Systems Conference, 116–132. https://doi. org/10.1145/3643834.3661556 Dang, K., & Lee, S. (2024). Musical performances in virtual reality with spatial and view-dependent audio descriptions for blind and low-vision users. The 26th International ACM SIGACCESS Conference on Computers and Accessibility, 1–5. https://doi.org/10.1145/3663548.3688492 Emmelkamp, P. M. G., & Meyerbröker, K. (2021). Virtual reality therapy in mental
health. Annual Review of Clinical Psychology, 17(1), 495–519. https://doi. org/10.1146/annurev-clinpsy-081219-115923
motion-based game interaction for older adults. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1873–1882. https://doi. org/10.1145/2207676.2208324 India, G., Jain, M., Karya, P., Diwakar, N., & Swaminathan, M. (2021). VStroll: An audio-based virtual exploration to encourage walking among people with vision impairments. Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, 1–13. https://doi.org/10.1145/3441852.3471206 Jain, D., Junuzovic, S., Ofek, E., Sinclair, M., Porter, J., Yoon, C., Machanavajhala,
S., & Ringel Morris, M. (2021). A taxonomy of sounds in virtual reality. Designing Interactive Systems Conference 2021, 160–170. https://doi. org/10.1145/3461778.3462106 Jain, D., Junuzovic, S., Ofek, E., Sinclair, M., R. Porter, J., Yoon, C., Machanavajhala, S., & Ringel Morris, M. (2021). Towards sound accessibility in virtual reality. Proceedings of the 2021 International Conference on Multimodal Interaction, 80–91. https://doi.org/10.1145/3462244.3479946 Laver, K. E., Lange, B., George, S., Deutsch, J. E., Saposnik, G., & Crotty, M.
(2017). Virtual reality for stroke rehabilitation. Cochrane Database of Systematic Reviews, 2018(1), 1–3. https://doi.org/10.1002/14651858.CD008349.pub4 Lewis, S. (2021). As the American foundation for the blind celebrates its centennial, the Journal of Visual Impairment & Blindness embarks on its 115th year of continuous publication, with thanks to those who are making it possible. Journal of Visual Impairment & Blindness, 115(1), 3–4. https://doi. org/10.1177/0145482X20987706 Li, J., W. Tigwell, G., & Shinohara, K. (2021). Accessibility of high-fidelity prototyping
tools. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–17. https://doi.org/10.1145/3411764.3445520 Lohse, K. R., Hilderman, C. G. E., Cheung, K. L., Tatla, S., & Van Der Loos, H. F.
M. (2014). Virtual reality therapy for adults post-stroke: A systematic review and meta-analysis exploring virtual environments and commercial games in therapy. PLOS One, 9(3), e93318. https://doi.org/10.1371/journal.pone.0093318 Lokki, T., & Grohn, M. (2005). Navigation with auditory cues in a virtual environment. IEEE Multimedia. https://doi.org/10.1109/mmul.2005.33 Powers, M. B., & Emmelkamp, P. M. G. (2008). Virtual reality exposure therapy
for anxiety disorders: A meta-analysis. Journal of Anxiety Disorders. https://doi. org/10.1016/j.janxdis.2007.04.006 Saxena, S., Funk, M., & Chisholm, D. (2013). World health assembly adopts comprehensive mental health action plan 2013–2020. Lancet, 381(9882), 1970– 1971. https://doi.org/10.1016/S0140-6736(13)61139-3 Shinohara, K., & Wobbrock, J. O. (2016). Self-conscious or self-confident? A diary study
conceptualizing the social accessibility of assistive technology. ACM Transactions on Accessible Computing, 8(2), 1–31. https://doi.org/10.1145/2827857 Siu, A. F., Sinclair, M., Kovacs, R., Ofek, E., Holz, C., & Cutrell, E. (2020). Virtual reality without vision: A haptic and auditory white cane to navigate complex virtual worlds. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3313831.3376353 Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive
virtual reality. Frontiers in Robotics and AI, 3, 1–47. https://doi.org/10.3389/ frobt.2016.00074
Design and Evaluation of a Multimodal Feedback 163 Swaminathan, S., Yim, Y., Hudson, S. E., Bennett, C. L., & Carrington, P. (2021).
From tactile to NavTile: Opportunities and challenges with multi-modal feedback for guiding surfaces during non-visual navigation. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi. org/10.1145/3411764.3445716 Walker, B. N., & Lindsay, J. (2006). Navigation performance with a virtual auditory display: Effects of beacon sound, capture radius, and practice. Human Factors: The Journal of the Human Factors and Ergonomics Society, 48(2), 265–278. https://doi.org/10.1518/001872006777724507 Xia, C., Min, T., & Sugiura, Y. (2024). AudioMove: Applying the spatial audio to
multi-directional limb exercise guidance. Proceedings of the ACM on Human- Computer Interaction, 8(MHCI), 1–26. https://doi.org/10.1145/3676489 Yamagami, M., Junuzovic, S., Gonzalez-Franco, M., Ofek, E., Cutrell, E., Porter, J. R., Wilson, A. D., & Mott, M. E. (2022). Two-In-one: A design space for mapping unimanual input into bimanual interactions in VR for users with limited movement. ACM Transactions on Accessible Computing, 15(3), 1–25. https://doi. org/10.1145/3510463
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Metaverse, Virtual Environments and Game Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-93-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional