Zitong Zhou, Wei Gong (2025). Multi-Sensory Integration and Emotional Responses: The Impact of Materials on Perception and Emotion. In: Accessibility, Assistive Technology and Digital Environments. AHFE Open Access, vol. 162, pp. 168–178. https://doi.org/10.54941/ahfe1006024
With the rapid advancement of virtual reality (VR) technology, multi-sensory integration has become a significant area of focus in the emotional design of virtual environments. Materials play an important role in shaping both perception and emotion, while tactile stimuli are particularly influential in cognitive and emotional responses. Additionally, research indicates that there is an integration between vision, hearing, and touch. Studies have demonstrated that combining various sensory inputs, particularly tactile, visual, and auditory stimuli, can enhance cross-sensory integration. While research has examined the individual effects of tactile, visual, and auditory stimuli on perception, less attention has been paid to how these senses integrate and combine, especially within virtual reality. The study explores the impact of materials on cognition and emotion, as well as the integration between tactile, visual, and auditory inputs. An experiment was conducted in the virtual reality world "The Library of Solitude by the Sea," with thirty-two participants interacting with three materials. The research focused on the integration of tactile, visual, and auditory stimuli by stimulating all three senses simultaneously. Emotional responses were measured through questionnaires and skin conductance responses (GSR). The findings revealed that different materials influenced participants’ emotions and perceptions, with the multi-sensory group reporting greater levels of pleasure and disgust than the single-sensory group. This suggests that various materials evoke different emotional responses and that the combination of tactile, visual, and auditory stimuli leads to a more intense emotional response. The use of multiple sensory stimuli is essential for enhancing immersion and emotional consistency in virtual environments. The study fills a gap in the literature on how multi-sensory integration influences emotional responses. It demonstrates that combining tactile, visual, and auditory stimuli produces a stronger emotional response than the use of a single sensory input. The study also highlights how different materials affect emotional responses. The results indicate that multi-sensory integration enhances emotional engagement and depth. This research offers valuable insights into the emotional design of virtual environments, particularly in the development of immersive experiences through multi-sensory integration. Furthermore, it provides a foundation for future research on multi-sensory design and contributes to the continued evolution of emotionally immersive virtual worlds.
Multi-Sensory Integration and Emotional Responses: The Impact of Materials 169
INTRODUCTION
The sense of touch plays an important role in perception and cognition, and research has shown that the tactile sensation of different materials has a significant effect on emotions: soft, smooth surfaces often trigger pleasure, whereas rough, hard touches may trigger discomfort or aversion (Iosifyan & Korolkova, 2019). Two hypotheses are proposed in this study: first, does tactile sensation significantly influence emotional responses? For example, soft velvet may trigger pleasure, whereas rough sandpaper may trigger disgust; second, how do visual, auditory, and tactile sensations synergize in the emotional experience? Research has shown that visual and auditory information can trigger stronger emotional responses when they are aligned (Pan et al., 2019).
This study combines VR technology with real tactile stimuli to create controlled multisensory situations where participants receive visual, auditory, and tactile stimuli at the same time, breaking away from previous studies where only real environments or virtual haptic simulations were used. Through multimodal data collection such as subjective questionnaires, galvanic skin response (GSR), and interviews, this study aims to reveal how tactile sensation synergizes with other senses to influence emotions and provide guidance for enhancing users’ emotional experience in VR environments.
LITERATURE REVIEW
Research has shown that different physical properties of materials (e.g., soft, rough, smooth, etc.) trigger different emotional experiences: soft materials (e.g., fur and silk) are typically associated with happiness, whereas rough materials (e.g., sandpaper and wire sponges) are associated with negative emotions (fear, anger, etc.) (Iosifyan & Korolkova, 2019). In addition, people tend to prefer familiar stimuli and experience more pleasure in passive exploration (Etzi, Spence, & Gallace, 2014). Physiological data also suggests that pupil dilation is significantly greater when touching pleasant or aversive materials than neutral materials, showing that emotional intensity is proportional to physiological arousal (Bertheaux et al., 2020). This study will explore the role of material haptics in more complex VR multisensory environments and their interaction with other sensory information.
Multi-sensory cross-modal interaction investigates how different sensory channels integrate to influence perception and emotion. In audiovisual integration, visual and auditory information coherence enhances emotional experience (Fujisaki et al., 2014). In the absence of tactile devices, Lécuyer et al. found that visual stimuli triggered tactile perception (Lécuyer, 2009). Auditory-tactile interactions have also shown that each channel effectively integrates without inhibiting the other (Porcu et al., 2014) and that cross-sensory integration helps to form a consistent sense of presence in virtual environments, especially in visual and tactile interactions, where the phenomenon of synaesthesia plays a key role (Biocca et al., 2001). In this study, we will present visual, auditory, and tactile sensations simultaneously
170 Zhou and Gong
in VR to explore the effects of multichannel congruence and incongruence on emotion.
The effects of multisensory integration on emotion and cognition can be supported from both neuroscience and psychological perspectives. The perceptual multichannel model suggests that the brain processes external stimuli in such a way that information from each sense is not independent, but rather interacts across channels (Martino & Marks, 2000). Tactile stimuli activate visual areas, supporting the integration of multisensory information (Lacey & Sathian, 2015). Emotional responses, cognitive appraisals, and behaviors are enhanced when sensory stimuli are congruent (Schreuder et al., 2016), whereas incongruent stimuli may diminish the emotional experience (Schreuder et al., 2016). Alshaer (2025) found that the combination of visual and haptic sensations significantly elevated the “trust” emotion, with higher emotional intensity than a single sensory condition (Alshaer, 2025). Combining other sensory stimuli can activate different cognitive functions and enhance immersion, providing a theoretical basis for designing multimodal interactions (Gori et al., 2011). In summary, theoretical and empirical studies have shown that multisensory integration affects not only the perceived content, but also the intensity, nature, and duration of emotions.
Although there have been studies on haptics, emotions, and multisensory interactions, most of them have been conducted in the laboratory and failed to restore real situations. This study provides visual, auditory and tactile contexts through VR technology to construct a more relevant multisensory experience. Previous studies have mostly focused on two sensory combinations and some emotional dimensions and lacked the simultaneous measurement of subjective experience and physiological responses under triple stimuli. This study will comprehensively depict multisensory emotional response curves through questionnaires and GSRs, and explore the role of tactile sensation in emotional amplification in multisensory contexts. The research results will provide a valuable basis for VR therapy, game design, and product design.
EXPERIMENTAL DESIGN AND METHODS This experiment combines virtual reality (VR) scenarios and real-object touch to investigate the effects of multisensory stimuli on emotional responses. Participants sequentially experienced three materials, glass, wood, and fur, in a VR environment to assess the effects of tactile, visual, and auditory stimuli on emotional responses. The tactile properties of each object were synchronized with the visual and auditory effects in the virtual environment to explore how multisensory stimuli affect emotional fluctuations through different materials and to test whether multisensory stimuli elicit more significant emotional responses than single visual stimuli.
Experimental Environment Design The experiment was conducted in a virtual environment, the “Seaside Library”, whose architectural prototype is based on the Sanlian Bookstore
Multi-Sensory Integration and Emotional Responses: The Impact of Materials 171
Seaside Public Library designed by architect Dong Gong in Qinhuangdao. The bookstore is warmly decorated, surrounded by abundant bookshelves and chairs, with a view of the vast beach outside the window, accompanied by background sound effects of waves and wind. On the one hand, the calmness and serenity of the environment itself serve as an emotional baseline that stabilizes the participants when they enter the experiment and enhances the sense of immersion; on the other hand, it is hoped that the natural landscape reduces stress and enhances positive emotions.
Figure 1: Seaside library VR environment.
The three tactile objects in the experiment - a glass window, a rough wooden table, and a fur cushion - were placed in a virtual bookstore, and participants would circumnavigate the virtual environment and correspond in turn to touching these objects and experiencing them in reality.
Figure 2: Tactile objects in a seaside library.
Haptic Task Design & Visual and Auditory Design The haptic task was a central part of the experiment, and we selected three significantly different materials, glass, wood, and fur, with the aim of covering a broad haptic spectrum from hard and smooth to soft and rough, and reducing the cognitive learning cost of the user experience through common materials of life. The significant differences in texture and temperature of these three materials allowed us to examine how different tactile properties affect emotional responses and to compare the role of multisensory stimuli in different material contexts. The haptic task sequence was fixed, and participants would touch the glass window, wooden table, and fur cushion in turn, with the freedom to explore the surface of the items, touching, rubbing,
172 Zhou and Gong
or tapping to perceive their texture, for two minutes for each material. The realistic size of each object was 30cm*40cm to ensure that participants could fully perceive the texture of the objects.
Figure 3: Realistic counterparts of tactile objects: glass, wood, artificial fur in that order.
This experiment does not use additional haptic enhancement devices but relies solely on direct contact between participants and real objects. Research has shown that realistic haptic feedback enhances presence in virtual environments and makes virtual objects more realistic and believable (Gallace & Spence, 2014). In the experiment, haptic tasks will be performed in a specific VR virtual environment and synchronized with auditory stimuli to create a multisensory experience. Arrows on the floor guide the experimenter forward, and the participant sequentially performs the haptic experience at three stopping points to ensure spatial and temporal correspondence between vision and touch. For example, participants touched the glass in reality only when they were facing the virtual glass window in VR to avoid a mismatch between visual and tactile information.
In terms of auditory design, auditory feedback in the experiment will be synchronized with haptic and visual feedback to enhance the overall experience. The auditory feedback is further enhanced by capturing the sound of participants interacting with objects through microphones, creating a complete multi-sensory perception. In addition, background sound effects (e.g., the sound of waves, wind, etc.) will be played continuously to enhance the natural atmosphere and maintain immersion. The auditory context is kept consistent in the experimental and control groups to ensure no biasing effects on the two groups.
Experimental Process The experimental procedure was as follows: Participants entered the virtual environment of the “Seaside Library” wearing a VR device, and filled out an emotional assessment questionnaire to record their initial emotional state (e.g., happy, sad, angry, etc.). Afterward, the participants contacted the three materials in turn, each time touching for about two minutes, and filled out a short emotional assessment questionnaire to record their reactions. Meanwhile, the galvanic skin response (GSR) device continuously recorded physiological responses. Upon task completion, participants completed a detailed PAD questionnaire to review emotional changes and a short
Multi-Sensory Integration and Emotional Responses: The Impact of Materials 173
semi-structured interview to obtain subjective feedback and qualitative information.
Experimental Design This experiment used a between-groups design in which 32 voluntarily enrolled subjects were randomly assigned to an experimental group (multisensory stimulation group, n = 16) and a control group (uni-sensory vision group, n = 16). All participants signed an informed consent form and received a brief training on the operation of the VR before the experiment. Prior to the experiment, participants were given a half-hour meditation period to recover emotionally. The experimental group received a synergistic combination of visual, tactile, and auditory sensory stimuli, touching real objects and receiving corresponding visual and acoustic feedback; the control group received only visual stimuli in VR, viewing virtual scenes and listening to ambient background sounds, with no tactile feedback or object interaction sounds. The two groups were equally distributed in other conditions (e.g., age, gender, VR experience, etc.) to minimize the influence of individual differences on the results.
Figure 4: Subjects in the experimental group performed the experiment.
Data Collection and Analysis This experiment used multiple methods of data collection to comprehensively assess the effects of tactile, visual, and auditory synergies on affective responses. Data collection consisted of an affective assessment questionnaire, galvanic skin response (GSR), and semi-structured interviews to obtain quantitative and qualitative affective data from multiple perspectives. The emotion assessment questionnaire used the Ekman emotion model and a Likert scale to assess participants’ subjective emotional states, covering six
174 Zhou and Gong
basic emotions: happiness, sadness, anger, fear, surprise, and disgust. At the end of the experiment, the overall emotional experience was assessed using the PAD affective measure, including the three dimensions of pleasure, arousal, and dominance. In addition, galvanic skin response (GSR) data were continuously recorded to reflect the participants’ emotional arousal levels and provide an objective quantification of emotional intensity. At the end of the experiment, semi-structured interviews were conducted to collect qualitative data to gain insights into the participants’ emotional changes, immersion, and salient experiences in different haptic tasks, which provided support and explanation for the quantitative questionnaire and physiological data.
RESULTS
Effects of Different Materials on Emotions The experimental results showed that tactile stimuli of different materials significantly affected emotional responses. Statistical analyses showed a significant main effect of material type on emotional indicators. For example, the feeling of happiness after touching fur was significantly higher than that after touching glass (mean about 4.1 vs. 2.4, p < 0.01), while the feeling of fear was highest when touching glass (mean about 3.6), which was significantly higher than that of wood (2.3) and fur (2.1) (p < 0.01). These results suggest that different materials trigger different emotional responses. For physiological data, the electrical skin responses (SCL and SCR) also showed significant differences. Glass material triggered higher peaks of galvanic skin response, showing stronger emotional arousal; whereas wood showed flatter galvanic skin changes. Overall, both subjective mood scores and physiological indicators support the conclusion that “material type influences mood”.
- Glass: Negative emotions rose significantly when touching glass. Fear scores rose from 2.8 to 3.6 (p < 0.01), and sadness also increased, while feelings of happiness decreased. During interviews, participants mentioned that the cold touch of glass made them uneasy, consistent with elevated skin conductance levels, suggesting that it triggered stronger emotions of tension and alertness. The galvanic skin response (GSR) showed a higher number and magnitude of SCRs in the glass condition, indicating greater mood swings. Overall, glass triggered negative emotions such as fear and anxiety.
- Wood: Emotions tended to be positive and relaxed when touching wood. Happiness scores increased from 3.0 to 3.8 and fear decreased significantly (from 2.8 to 2.3). Although sadness increased slightly (p > 0.05), this may be related to individual associations. During the interviews, participants described wood as providing a “warm, relaxing” feeling, consistent with lower physiological arousal - the frequency and magnitude of SCRs were significantly lower in the wood condition than in the glass. Wood triggered calmer emotions, increased feelings of pleasure, and reduced negative emotions such as fear and disgust.
- Fur: Touching fur resulted in a 35% increase in pleasure (from 3.0 to 4.1, p < 0.01) and a significant increase in surprise (from 2.7 to 3.9, p<0.01).
Multi-Sensory Integration and Emotional Responses: The Impact of Materials 175
Negative emotions decreased significantly, especially disgust (from 3.1 to 1.8, p < 0.001). Interview feedback indicated that fur touch “surprised and delighted” the participants and provided a sense of comfort, intimacy and security. Skin electrical responses were frequent (SCR), but more derived from pleasurable arousal, and baseline levels decreased, indicating less tension and aversion.
In summary, the different materials each triggered different mood changes - glass was biased towards triggering negative emotions, wood brought moderately positive and calming emotions, and fur strongly elicited positive emotions and surprise.
Comparison of Mood Swings Between Experimental and Control Groups Comparison of the experimental and control groups revealed that the experimental group, which received real tactile stimuli, had greater mood swings than the control group, which received only audiovisual stimuli. On the subjective questionnaire, the experimental group’s happiness scores increased by 1.1 points (from 3.0 to 4.1) in the fur condition, while the control group’s increased by only 0.5 points (from 2.6 to 3.1). In the glass condition, fear increased by 0.8 points in the experimental group and 0.3 points in the control group. The galvanic skin data showed that the number and magnitude of SCRs were generally higher in the experimental group than in the control group, especially during dramatic mood changes, such as pleasure from fur and tension from glass. The control group had a smoother GSR response and limited mood fluctuations due to the lack of tactile input. The PAD mood measure showed that the experimental group had a minimum value of 1.46 (corresponding to a mild mood) with a positive p-value, and an A-value that was higher than that of the control group of 1.10 (corresponding to a boring mood) with a negative p-value, indicating that the experimental group had a higher state of pleasantness, concentration, and a more intense experience. The results of the interviews also indicated that the experimental group’s tactile stimuli enhanced situational realism and emotional responses. Participants in the control group mentioned that their emotional reactions were “not as strong as expected” or “a little out of character” because they “could only see” and could not touch.
DISCUSSION
Tactile and Multisensory Synergistic Amplification of Emotional Experience The present study demonstrated that tactile stimuli can significantly amplify the intensity of emotional experiences. When visual and auditory stimuli were combined with tactile input, participants exhibited stronger physiological and emotional responses. This amplification effect stems from the sense of immersion and reality provided by haptics, which makes emotional responses more realistic and intense. Haptics allowed participants to not only “see” the situation, but to “feel” a part of the situation, thus
176 Zhou and Gong
enhancing emotional responses. In addition, research has emphasized the significant impact of multisensory immersion on emotions. Multi-sensory integration creates a more realistic experience and triggers stronger and richer emotional responses. In the experiment, single-sensory stimulation triggered lower emotional intensity, while contexts that combined tactile sensations significantly increased physiological arousal and mood swings. Multi-sensory stimuli not only amplify emotional intensity, but may also change the nature of emotional responses, especially when sensory cues are congruent, positive touch (e.g., soft fur) reinforces pleasurable emotions, whereas when sensory cues are incongruent, complex emotional responses may be generated. This suggests that sensory synergies need to be considered when designing emotional experiences.
Comparison of Results With Existing Studies Overall, our results are consistent with findings from existing studies, while also providing new detailed perspectives. First, in terms of the effect of touch on emotions, it has been shown that touching different materials can be associated with different emotions, and our experiments validate and extend this. For example, participants’ touch of soft fur produced significant pleasure and surprise, whereas touch of smooth and cold materials produced tension and disgust, supporting the idea that different materials can induce different emotions. In terms of multisensory integration, the literature also points to the effects of touch working in conjunction with other senses, such as real touch making negative emotions more intense. On the other hand, our experiments corroborate that adding tactile sensations (e.g., the warm touch of fur) in pleasant situations does amplify positive emotional responses. It is important to note that our study is one of the attempts to compare the effects of material-specific haptics in immersive environments, and we provide more nuanced evidence of how differences in material texture shape emotions than previous studies focusing on the “general presence or absence of haptics” or the “overall sense of presence”. We provide more detailed evidence of how differences in material texture shape mood. This not only reinforces existing findings, but also provides empirical support for subsequent studies on the association between tactile material selection and mood.
Practical Applications & Future Research Directions This study demonstrates that haptics can significantly enhance emotional responses and has important practical applications. In virtual reality and game design, haptic feedback can enhance immersion and emotional engagement; in affective therapy and psychological interventions, multisensory stimulation can help regulate emotions; in product and environmental design, designers can create appropriate emotional atmospheres based on material texture. Considering the limited sample size and material types in this study, future research should expand the sample, enrich the materials (e.g., roughness, smoothness, temperature, vibration, etc.), and incorporate senses such as smell and taste to explore the synergistic effects of multi-sensory combinations on emotions. In addition,
Multi-Sensory Integration and Emotional Responses: The Impact of Materials 177
the moderating effects of individual differences (e.g., personality traits, cultural background) on tactile-emotional responses and the long-term effects of multisensory stimuli should be investigated. These studies will advance the field of emotion and multisensory integration.
CONCLUSION
This study explored the effects of tactile, visual and auditory multisensory stimuli on emotional responses. It showed that tactile stimuli of different materials significantly altered emotions, and multisensory stimuli enhanced emotional experiences more than single stimuli: fur enhanced positive emotions (e.g., pleasure, surprise), while glass induced negative emotions (e.g., fear, sadness). The experimental group had higher galvanic skin responses and subjective scores at the peak of emotion than the control group due to realistic tactile feedback, validating the importance of multisensory integration in emotional design. This finding provides theoretical support for virtual reality and emotional design, suggesting that combining multisensory stimuli can create more immersive, emotionally rich experiences. Future research could further explore the effects of more tactile elements (e.g., temperature, vibration) and individual differences on multisensory experiences.
REFERENCES
Alshaer, A., 2025. The role of sensory experiences in evoking emotional responses in virtual reality. Journal of Umm Al-Qura University for Engineering and Architecture, pp. 1–13. Bertheaux, C., Toscano, R., Fortunier, R., Roux, J. C., Charier, D. and Borg, C.,
2020. Emotion measurements through the touch of materials surfaces. Frontiers in Human Neuroscience, 13, p. 455. Biocca, F., Kim, J. and Choi, Y., 2001. Visual touch in virtual environments: An exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence: Teleoperators & Virtual Environments, 10(3), pp. 247–265. Etzi, R., Spence, C. and Gallace, A., 2014. Textures that we like to touch: An
experimental study of aesthetic preferences for tactile stimuli. Consciousness and cognition, 29, pp. 178–188. Fujisaki, W., Goda, N., Motoyoshi, I., Komatsu, H. and Nishida, S. Y., 2014. Audiovisual integration in the human perception of materials. Journal of Vision, 14(4), pp. 12–12. Gallace, A. and Spence, C., 2014. In touch with the future: The sense of touch from cognitive neuroscience to virtual reality. OUP Oxford. Gori, M., Mazzilli, G., Sandini, G. and Burr, D., 2011. Cross-sensory facilitation
reveals neural interactions between visual and tactile motion in humans. Frontiers in psychology, 2, p. 55. Iosifyan, M. and Korolkova, O., 2019. Emotions associated with different textures during touch. Consciousness and cognition, 71, pp. 79–85. Lacey, S. and Sathian, K., 2015. Crossmodal and multisensory interactions between
vision and touch. In Scholarpedia of touch (pp. 301–315). Paris: Atlantis Press. Lécuyer, A., 2009. Simulating haptic feedback using vision: A survey of research and applications of pseudo-haptic feedback. Presence: Teleoperators and Virtual Environments, 18(1), pp. 39–53.
178 Zhou and Gong
Martino, G. and Marks, L. E., 2000. Cross-modal interaction between vision and touch: The role of synesthetic correspondence. Perception, 29(6), pp. 745–754. Pan, F., Zhang, L., Ou, Y. and Zhang, X., 2019. The audio-visual integration
effect on music emotion: Behavioral and physiological evidence. PloS one, 14(5), p. e0217040. Porcu, E., Keitel, C. and Müller, M. M., 2014. Visual, auditory and tactile stimuli compete for early sensory processing capacities within but not between senses. Neuroimage, 97, pp. 224–235. Schreuder, E., Van Erp, J., Toet, A. and Kallen, V. L., 2016. Emotional responses
to multisensory environmental stimuli: A conceptual framework and literature review. Sage Open, 6(1), p. 2158244016630591. Woods, B., O’Philbin, L., Farrell, E. M., Spector, A. E. and Orrell, M., 2018.
Reminiscence therapy for dementia. Cochrane database of systematic reviews, (3).
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Accessibility, Assistive Technology and Digital Environments
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-38-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional