Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Experience And Interaction - Application Of Audiovisual Synesthesia In Interactive Devices

School of Design, Hunan University, Changsha City, Yuelu district, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1001939

Abstract

Background: As material culture improves, people's need for spiritual culture becomes more and more urgent. Art exhibitions are an important way for the public to participate and absorb cultural and spiritual nutrients, and the interactive installation works in the exhibition are a favorable form of creation that can bring the audience closer to the works. However, the diversity of audiences and the varying degrees of professional inculcation have led to some audiences being turned away. This situation extent an obstacle to the dissemination and development of the arts.Aims: The interactive installation removes the distance between the audience and the artworks, enhances the interactivity and experience between the audience and the artworks, and diversifies the exhibition format and enriches the visual language.Method:Through the variety of exhibition displays, new forms of artistic expression are discussed. Specifically, the phenomenon of audiovisual association and the correlation characteristics that exist between audiovisual factors are studied, key influencing factors are extracted and applied to the creation of interactive installation artworks, opening up the way of perception for the audience to recognize the works through multiple channels.Consequence:Through the creation of the experimental works, it was found that there is a cross-correlation between the sound and the visual presentation in the interactive installation; secondly, the audience is transformed from a single spectator to a participant. Compared to static installations or sculptures, interactive installations with audio-visual associations have unparalleled advantages in terms of creative dimension and cognitive engagement with the work.Keywords: audiovisual synesthesia、 interactive artwork、 sound consciousness、experience、exhibition

INTRODUCTION

The diversity of media languages has given creators more ways to express themselves; at the same time, interactive installation art has become more popular and accepted by audiences in recent years. It is not only the linguistic richness of the installation itself but also the interactive relationship between the space and the viewer in the installation that makes this art form accessible to a growing number of audiences.

As the times have changed, audiences and art creators have changed the way art is presented and the language of expression. This change is linked to the knowledge and reception of information by the group in which it is located in the context of the times and era. In this paper, we investigate and

© 2022. Published by AHFE Open Access. All rights reserved. 214

Experience and interaction - Application of Audiovisual Synesthesia 215

explore the methodology of ’audiovisual association’ in the theory of asso- ciation and apply it to the creation of interactive installation art to discover more possibilities and the richness of the language of artistic expression. Ulti- mately, through the creation of two sets of interactive installations, this paper further examines how to better use ’audiovisual association’ in the creation of interactive installations, and at the same time, examines and investigates the interaction between the audience and such artworks.

METHODOLOGY

Synesthesia perception is an innate, unlearned human ability. The term syna- esthesia first originated in Greece and means ’connected sensation’, referring to the mental processes caused by one sensory stimulus to the intuition of other senses that are not directly related. The inducing stimuli that cause synaesthesia are called the inducer, and the sensations induced by the inducing stimuli are called the concurrent. Common inducing stimuli include external stimuli perceived by the five sensory organs: sight, sound, smell, taste and touch. Commonly associated experiences are taste, visual, etc. The SACHS first described the phenomenon of synaesthesia in 1812, but it was not until the rise of cognitive neuroscience that the study of synaesthesia was taken seriously.

Human perception of the world is the result of the combined acquisition and processing of information from different sensory channels such as sight, hearing, touch, taste, and smell. The combined perceptual results of multi- ple sensory channels help us communicate with the outside world, and there are very significant differences between the many perceptual pathways in how we recognize and communicate with things. Experimental psycholo- gist Treicher’s research shows that the human brain receives 83% of external information through the senses each day, with the visual sense accounting for 83% and the auditory sense for 11%. The combined action of the dif- ferent senses helps us understand things more comprehensively and helps to reduce dependence on one pathway or another (4). For example, the auditory sense is second only to the visual sense in terms of the information a person obtains from exposure to external stimuli and is a very effective way of sup- plementing the visual channel when it is not available or when information is more difficult to recognize visually. For human vision, the central recess of the retina (the sharpest part) has only a 1° - 2° field of view, whereas hea- ring provides 360° of information without focusing on the output, making it relatively more flexible and scalable than vision.

Humans can receive large amounts of information in everyday life and process complex tasks without being overloaded. This is mainly because humans have five different perceptual channels, each of which is coordinated to form an organic, unified whole, with multiple perceptual systems working together to channel information internally. Furthermore, different perceptual channels automatically change how people perceive the external world as the mechanism of action and how information is processed varies, instead of imposing a substantial cognitive burden independently on a particular perceptual channel.

216 Zhou

Thus, a clear approach to information and a better way of perceiving what is being received provides more access and means of resolving the relationship between the art creator (the output side of the information) and the audience (the receiver of the information).

Secondly, psychological studies have shown that the exchange of informa- tion through multisensory channels can effectively improve the quality of a person’s performance of a specific task. Of the five human sensory channels, hearing is second only to vision in the amount of information it obtains from the outside world and can be an excellent complement to vision, and sound can increase the number of information people obtain or reduce their reliance on the visual channel.

Artistic creation is highly subjective and the final presentation of artwork is influenced by various factors, such as the creator’s access to information, the environment in which the creator lives, the education and knowledge system the creator has received, the creator’s habits and preferences, etc. All of these presentations affect every step of the art creator’s thinking and actions in the creative process. With all of these factors, the challenge becomes relatively unstable for the viewer because as each unique individual, the reception and transformation of information develop an understanding that is relevant or familiar to them. However, audiovisual synaesthesia in the interactive instal- lation undoubtedly gives the audience more channels to receive information and provides an effective way of perception and information feedback.

DEVELOPMENT Practice 1: ’Blank Space’ Installation, an Interactive Art Installation With an Audio-Visual Focus

“Blank Space” is a collection of works inspired by the smell of memory. On a walk in the countryside after dinner, the author and his mother were wal- king in the countryside, as usual, when a sudden breeze came in, mixed with the unique smell of the earth and the fields, which for a moment brought the author’s thoughts back to his childhood, to his ignorant and curious memories.

In this solo exhibition, the author attempts to create an art installation using the methodology of “audiovisual synaesthesia” to appreciate the work better, read the information, and realize the interactive relationship with the artist or the work while entering the space.

Thus, before creating and conceiving the work, the author thinks that the work needs to trigger the sensory stimulation of the viewer through some aspects in a specific context to achieve the creator’s motivation and thus the interaction between the work and the viewer.

In the past, the works were hung after the walls had been treated (sanded, holes filled, painted and hung). In addition, the atmosphere of the previous exhibition was relatively lively, but in the author’s opinion, this lively atmo- sphere created a certain contradiction in the space, how to use and resolve these problems will also be one of the problems to be solved in this exhibition.

Experience and interaction - Application of Audiovisual Synesthesia 217

Thus, addressing the relationship between the presentation of the works and the many viewers and the space is the issue that the works in this exhibi- tion need to address. The author’s work is required to have a relationship with the space, which becomes part of the work, and therefore tries to recreate a space within the space, but does not want such an act to seem so deliberate, so the author, after intentionally avoiding the structure that the space itself pos- sesses, recreates the wall around the original wall foundation, and an empty space comes out.

Secondly, the need to consider how to ’calm’ the viewer at the opening of a lively exhibition, or the psychological preconceptions of entering the gallery from its surroundings, is also an issue that needs to be addressed in work.

“Blank Space’ is a work that evokes body language through sound and then visual imagery. The sound is presented through headphones, which increases the viewer’s privacy and creates an invisible wall between reality and the work. The headphone cable is threaded directly through the wall, while the furthest distance between the cable and the wall is controlled at 30cm. The primary purpose of doing so is that, with the help of all acts, the audience can quickly sink into the space and have an artistic experience in a relatively quiet and clean environment. This inspiration, drawing on meditation and facing the wall, both of which help the viewer to concentrate and mentally presuppose the environment for the perceptual triggers that follow, serves to increase the efficiency of attention and allows for a more effective allocation of attentional resources.The sound in the headphones is created by being outdoors and collecting natural sounds: wind, aquatic, frogs, rain, and the sound produced by branches and leaves colliding with each other. All the viewer needs to do is to quietly follow the viewing patterns and interaction methods set out in work, make visual and auditory associations, and realize what they want to see in their minds.

(The audience picks up the headphones to listen and achieve an interactive

relationship with the work) 2018.

218 Zhou

Practice 2: Analysis of the Creation of “Walk into the Jungle” - “Secret of the Jungle” “I have come through the wilderness and the jungle”

“You carried the author in a boat into a sea of flowers The woods are lush, the butterflies are fluttering The birds sing, the fireflies take shape It’s like falling into a dream of the Wizard of Oz” The pace of life in modern society is getting faster and faster, and the pressure between work, family and interpersonal interactions is frequently flashing. As people grow older and are inundated with all kinds of desires, their dreams are no longer light, and their hearts have long since failed. In order to heal people’s anxiety, irritability, depression and other mood swings caused by unhealthy emotions, the author attempt to create a forest healing space as the creative origin of the installation “Jungle Secret,” trying to give the audience a relaxing experience of mind and body in a dream-like space, and a wonderful visual interaction experience with a differentiated visual experience.

Before conceiving the whole work as a whole, the author intentionally seeks to create materials from elements related to it and tries to achieve a psychological link and fit between the various materials, especially in the case of fragmented information, by thinking and designing to integrate the different materials and ultimately to serve the thematic idea to be conveyed. From the outset, it was conceived that the whole piece needed to be visually differentiated, surrounded by sound, fun and interactive at the same time.

The choice of materials needs to reflect the feeling of the vocabulary that corresponds to it, such as: distance, broken walls, metallic texture, softness, time, air, glass, transparency, reflection, childishness, touch. There is also the consideration of the future transport and maintenance of the piece. Based on the influence of these ideas, humans chose the more common materials such as: acrylic, stainless steel, aluminum-plastic ban, led strip sets, interactive controllers, sound players, and ocean balls.

The entire installation, which will consist of four different sets of boxes, will be presented as a whole on a stand with a diameter of ten meters. The attempt creates a unique space distinct from its natural surroundings to achieve a totally immersive viewing experience.

In the process, humans began by making a transparent box with an acrylic border, perforated and broken in appropriate places, to echo the ’sense of distance, broken walls and transparency’ that the work itself is intended to convey and to prepare the base for the work. The primary material for this forest is stainless steel of different thicknesses and sizes, scanned and cut, welded together to mimic the landscape of a particular part of the forest, and then each leaf and trunk is polished and painted after the welding and modeling are complete. (The production process as shown at the top the next page).

Once all the original parts and materials have been prepared, humans assemble the single finished plant or branch into a clear box of acrylic. The final visual presentation is made by placing four different shaped plants on an

Experience and interaction - Application of Audiovisual Synesthesia 219

(Stainless steel blade assembly, painting, welding process).

(Cutting shapes, assembly welding, polishing, painting of stainless steel materials).

aluminum panel with a robust reflective effect, which on the one hand allows for a spatial breakthrough and, on the other hand, provides a quantitative enrichment and a sense of hierarchy through the mirror reflection.

Once the piece was assembled, the visible part of the work was almost complete, but considering that humans wanted to restore a space related to nature, time and interaction in a somewhat healing way, humans added a sound part to work. The sounds in work are collected from raw nature, with

220 Zhou

(the process of install the leaf).

(assembly complete).

insects chirping, birds chirping, noisy afternoon robins, frogs, streams, rain- drops and wind. Secondly, humans then incorporate light into the installation to realize the visual impact of the changes in lighting on the installation at different times of the day. Humans will differentiate between four phases in terms of time cues: early morning, midday, late afternoon and late night, and correspond to the sound systems produced at different times, forming a com- plete natural ecosphere. The lighting itself changes automatically in sequence with the four main time phases, and when the audience enters the 10m diame- ter stand, the lighting changes to become brighter, achieving a body language dialogue with the audience and the work through the changes in lighting. Finally, the final touch of the ’ocean ball,’ a childhood favorite, is intended to echo the humans’ inner child and to create more associations as they move between the pieces.

At this point, the entire creation of the interactive installation through audiovisual synaesthesia has been finished and presented.

Experience and interaction - Application of Audiovisual Synesthesia 221

(An ocean ball frozen on a leaf, like a water drop and symbolising

memories of childhood).

CONCLUSION

Experience and interaction - the application of audiovisual synaesthesia in interactive installations not only brings the artist a wealth of means of cre- ation and presentation, but also allows the individual information reserves of the artist to be transmitted and transformed through a variety of methods and media, thus better serving the expression of the artist’s ideas in creation.

In addition, the use of audiovisual synaesthesia in the interactive instal- lation brings a different artistic experience and visual enjoyment, which is an effective way for the general audience, who lack a rich knowledge base, to obtain information and perceive and feel the intentions of the art creator from multiple perspectives, and also has a more positive effect on the audi- ence’s better understanding and interaction with the work. Furthermore, the use of audiovisual synaesthesia in the interactive installation brings the audi- ence closer to the artwork and breaks down the information barriers in the artwork.

The following points can also be drawn from the experimental research and the practice of the work, through the application of experiential and interactive-visual synaesthesia in the interactive installation work.

I: Space and Mental Preparation. (Entry Phase) The use of audiovisual synaesthesia in interactive installations can give the viewer a more immersive way of experiencing art, in such a way that requires a certain amount of space for the presentation of the work. The work can remove unnecessary distracting information while giving the viewer some space and psychological preparation in the ensuing environment, thus pre- conditioning the access and exposure to the information that follows.

II: Visual and Auditory Links. (Experiential Phase) The main discussion here is the combination and use of the visual plus the auditory in the installation. At the early stage of creation, the visual and auditory design should be considered complementary or interrelated. More

222 Zhou

positive or informative links can be created, giving the audience more room for imagination in such an environment. The work can engage the viewer’s multiple sensory systems and the connection between the installation and the art. This is something that humans perceive well in the installations Blank Space and Jungle Secrets.

(Audiovisual synaesthesia - a thought sequence diagram for use in instal- lation artworks)

REFERENCES

Brewster S A. Using Non-speech Sound to Overcome Information Overload. 1997, 17(3): 179–189. Brewster S A, Wright P C, Edwards A N D. An Evaluation of Earcons for Use in

Auditory Human-computer Interfaces. Amsterdam: ACM/IFIP INTERCHI, 1993 Rayner, K.& Pollatsek, A. (1989). The Psychology of Reading, pp. 9–10. Englewood Cliffs, New Jersey: Prentice-Hall International, Inc Walker V A, Brewster S A. Spatial Audio in Small Screen Device Displays. Personal

Technologies 2000 4(2): 144–154. Lemmens P M C, Bussemakers M P, Haan A D. The Effect of Earcons on Reaction Times and Error-rates in a Dual-task VS a Single-task Experiment. Atlanta, GA: ICAD, 2000 Mondloch C J,Maurer D. Do Small White Balls Squeak? Pitch-object Corresponde-

nces in Young Children. Cognitive, Affective & Behavioral Neuroscience, 2004, 4(2): 133–136. Walker P, Bremner J, Mason U. et al. Preverbal infants5 Sensitivity to Synesthetic Cross-modal Correspondences. Psychological Science, 2010, 21(1): 21–25. Brownbb. Recognition of aspects of consciousness through association with EEG

alpha activity represented by a light signal [J]. Psychophysiology, 1970, 6(4): 442–452. He QI; Wang Haiying. (2020).The effects of meditation on attention. China Academic Journal Electronic Publishing House. Vol. 28, NO. 2, 284–293. Fletcher L B, Schoendorff B, Hayes S C. Searching for mindfulness in the brain: A

process-oriented approach to examining the neural correlates of mindfulness [J]. Mindfulness, 2010, 1(1): 41–63.

Cover of the proceedings volume Ergonomics In Design

Explore the complete digital volume

Ergonomics In Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-23-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →