Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Psychogeography with Jack B. Yeats Art Sounding Gallery: Augmented Reality Locative Experience for Blind People

1 Technological University Dublin, Dublin, Ireland ↗
2 Trinity College Dublin, University of Dublin Dublin, Ireland ↗
Publication year
Digital Object Identifier10.54941/ahfe1001639

Abstract

Sensory Substitution Devices (SSDs) are a relatively novel concept, based on the idea of the multisensory brain. Research on synaesthesia and sensory pairings has revealed that sensory modalities of the brain are interconnected. Nature has demonstrated examples of people who have lost one sense, which has then been substituted by increased ability in another, for example the case of Daniel Kish who navigates like a bat (Burgess, 2021) by clicking of the tongue (echolocation principle). To find a methodology for translating information from one sense to another, or substitute one sense with another is the principle for all SSDs. A number of approaches to assistive technologies for different impairments have been developed, for example for blind people, such as the vOICe “seeing with sound” and EyeMusic Apps, which convert visual images and colours into sound. While most SSDs are focused on functionality to offer life assistance, such as for navigation, little or no work has been done to include the blind into the emotional world of Visual Arts, despite the fact that there are 45 million blind people in the world (Amir Amedi). In this paper, we present an audio GPS based walking app that presents a translation of the visual expression of artworks by sound/music to deliver the emotional content of the paintings to blind people. The music is composed for six artworks of Irish artist symbolist Jack B. Yeats (1871-1957), specifically reflecting on shapes, colours and emotional content of painting by composer experienced in audio-visual synchronisation via synaesthesia (Rudenko and Córdoba Serrano, 2017). The project is centred around the development of a new methodology for multisensory design (MSD) through the design, implementation, and evaluation of a locative art experience with Augmented Reality (AR), hosted by Haunted Planet Studios (director Mads Haahr).

SYNAESTHESIA AND THE BRAIN Daphne and Charles Maurer stated “that all human infants start life with a form of synaesthesia in which all the senses are intermingled, and in which vision may be triggered by hearing as well as sight” (Ward, Jamie, 2008, p. 37). In Infants Synaesthesia, “sights have sounds, feelings have tastes, and smells can make (him) dizzy” (Maurer and Mondloch, 2006, p. XXI). Many scientists use synaesthesia model for SSDs (Hamilton-Fletcher et al., 2016),

© 2022. Published by AHFE Open Access. All rights reserved. 30

Psychogeography with Jack B. Yeats Art Sounding Gallery 31

and research has been carried out on adult synaesthetes to get insight into how sensory pairings work. Richard Cytowic writes: The science of synesthesia now spans several levels of magnitude – from DNA at the molecular level, to early cognition, brain imaging, all the way up to whole-organism behaviour that includes art and creativity. Brain organization is now seen as multiplex rather than modular, a formulation popular in the ’80 that insisted the senses travelled along isolated channels and could not interact. We now know that senses are highly intertwined and that the cerebrum is full of recurrent feedback and feedforward loops (Cytowic, 2018, p. XV).

Synaesthesia is a benign condition that is characterised by cross-modal perce- ptions. In addition to perceiving one stimulus, synaesthetes may perceive an additional quality to the experience – a colour or smell or feeling. Resea- rch has shown that people with synaesthesia are more aware of such sensory pairings due to stronger connections in their neural system. “Synaesthesia is a window into perception, thought and language” (Ramachandran and Hubbard, 2001) possibly solving the mystery of “how all of our brains create sensory experiences”(Ward, Jamie, 2008, p. 2) to perceive the world and rea- lity. Charles Spence observes that “a growing body of empirical research on the topic of multisensory perception now shows that even non-synaesthetic individuals experience cross-modal correspondences, that is, apparently arbi- trary compatibility effects between stimuli in different sensory modalities” (Parise and Spence, 2012). The brain is constructed in this way to survive by rewiring, reconstructing and substituting one sense with another in case of a loss.

SENSORY SUBSTITUTION DEVICES Chromesthesia is a type of synaesthesia characterized by a pitch to colour experience (composer Messiaen had chromesthesia). Amir Amedi and his associates at the Hebrew University of Jerusalem’s ELSC suggested to use this principle in reverse, to sound the colour for his SSDs App. (The artist Kandinsky had this type of synaesthesia.) Amedi’s App EyeMusic intro- duces a “visual” colorful experience for the blind using auditory sensory substitution, where “musical notes on a pentatonic scale generated by natu- ral instruments (to) convey the visual information in a pleasant manner” (Abboud et al., 2014). The colours are represented by timbres of instruments, even choir.

Peter Meijer also works with visual-to-auditory SSDs. His team uses “com- puterized sampling of the system output and subsequent calculation of the approximate inverse (sound-to-image) mapping provided the first convi- ncing experimental evidence for the preservation of visual information in sound representations of complicated images” (Meijer, 1992). Their softw- are (vOICe) works by converting camera images into sounds. Built as an Android app with over 100,000 downloads, it is “enabling people around the world to build mental images of the objects around them” (Meijer, 1992).

32 Rudenko and Haahr

The sound is non-musical but follows a precise shape and with future deve- lopment of retina prostheses and brain implants, the approach has a very promising future.

David Eagleman, Stanford University, Lecturer on Neuroscience and co- founder of the SSD start-up Neosensory says that “Synesthesia made him wonder if our senses were more pliable than assumed. Plus, there had been 50 years of research showing the brain is capable of sensory substitution—taking in information via one sense but experiencing it with another”(Kotler, 2019). Eagleman’s company produces a wristband watch, which vibrates sound for deaf people, letting them perceive sound information in tactile sense.

Jamie Ward, Giles Hamilton-Fletcher and Thomas D. Wright at the Uni- versity of Sussex designed a study that “introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings betw- een different sensory dimensions)” (Hamilton-Fletcher et al., 2015). Their study revealed that SSD applications, even if not fully providing sensory substitution, increase general mental abilities of users: “Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition” (Hamilton-Fletcher et al., 2015).

Another body of research has been carried at University of Sussex on “artificially induced synaesthesia,” which stated that after nine weeks, the participants passed the test: “results of an IQ improvement therefore pro- visionally indicate that cognitive training including synesthetic associations may in the future be a promising new tool for vulnerable clinical groups to enhance general mental ability” (Bor et al., 2014).

DESIGN PROCESS To apply SSD principles to perceive visual with audio (in our case, perceive Art with Music) we center our project around the development of a new methodology for multisensory design (MSD) through the design, implemen- tation, and evaluation of a locative art experience with Augmented Reality (AR) and a synaesthesia model for composing. Our Jack B. Yeats Art Soun- ding Gallery AR app1 showcases six paintings by Jack B. Yeats (1871–1957), an important symbolist painter not only on Irish scene but in the world. His father is a writer John Butler Yeats and his brother is the famous poet William Butler Yeats.

In this app, the composer Svetlana Rudenko follows closely the principle of “translation” of art details such as shapes, colours, perception of characters into music composition, which is then combined with intellectual research into the life and beliefs of Jack B. Yeats through fragments of interviews, art critics’ articles (Foley, Declan J., 2009a) and Irish folklore. The compo- nents of soundscape episodes are: a) Piano improvisation reflecting closely on 1Download: https://rebrand.ly/psychogeography-with-jack-b-yeats-ar-download.

Psychogeography with Jack B. Yeats Art Sounding Gallery 33

Figure 1: Screen shot from concept video Psychogeography with Jack B. Yeats: AR set of exact location of the painting’s spectators.
Original PDF Figure 1:

the emotional colour palette of the art image with the key/tonality, selection of genre, time-signature and type of musical texture; b) Violin and Piano arrangement of Irish songs (arrangement by Kenneth Rice) inclusion sym- bolises reflection on Jack B. Yeats’ background and close thematics to the painting to add Irish flavour of location; c) voice narration (Mads Haahr) consisting of extracts from critics on art, interviews, letters reflecting on inner world of the artist. Six episodes were produced, varying in texture to stand out individually as a fragment of sound vision.

The first AR episode is The Liffey Swim (1923) describing the annual river swim down the River Liffey. We can see O’Connell bridge and “the spectators are shown standing on Bachelor’s Walk… the viewpoint is out towards the open sea and sky” (Kennedy, 2009, p. 153). As the painting depicts the cheer- ful atmosphere of the event and there is no-conflict emotions, the music is set in G-major key, 6/8 time-signature of Irish song arrangement “Oh Dublin, you’re my city.” The modern AR app user can experience the psychogeo- graphy of the real location depicted by the painting with its characteristic ambient sound of seagulls and the river smell (see figure 1).

In opposition to the mono mood of The Liffey Swim, the second episode Flower Girl (1926) is full of contrasts of characters and emotions. Jack B. Yeats reflected on the differences of social status of Dubliners: “The woman, dressed in a red cloche hat and smart evening coat, is obviously of a different social class. Both women are of the same height and they look directly at each other, but the tilt of the woman’s head and her rigid expression make clear that she regards the flower girl as her inferior” (Kennedy, 2009, p. 156). The Piano improvisation here reflects on the brightness of the day and flower girl playfulness, the music is set in F-major and 6/8 time signature with simple melody/accompaniment texture and ornaments typical to Irish folk songs. The form of the episode here is ABA, where B is the middle section, which is Violin and Piano arrangement of the famous song Molly Malone in 3/4 time- signature. The original song is in major key, and by folk tradition is sung in a rather joyful manner, even the story of the girl is tragic (she died of a fever),

34 Rudenko and Haahr

as are stories of many girls who were selling products of poor families’ small businesses.” The narration describes the painting and return to the A music section to the “sweet” flower girl.

The third episode is Grief (1951). The Piano improvisation is in D-minor key and 4/4 time-signature with triplets in accompaniment and suspended time melody, giving the effect of a sigh. According to the National Gallery of Ireland: “Grief is perhaps Yeats’ most powerful expression of his abhor- rence of war. It appears to originate from a sketch entitled ‘let there be no more war,’ which the artist made in his last workbook. At the centre of the painting an apocalyptic figure, on horseback amidst and angry mob, gestu- res aggressively. In the foreground, an elderly man looks at the blood that drips onto his hands, while a mother places her arm protectively around a small child” (NGI, 2021). The Irish traditional song is Port na bPúcaí, arranged for Piano and Violin by Kenneth Rice. The song is a lament as in saying goodbye to the living world “And I will not be in this world” (lyrics of the song).

The fourth episode is The Singing Horseman (1949). Music and horses are common subject of Jack B. Yeats paintings. Here, “one of Yeats’s most jubiliant paintings, a young man on a large, restless horse, clasps his hands together, casts his head back and sings aloud to the heavens” (NGI, 2021), full of life. The narration to this episode is Jack B. Yeats’ own words: “I dislike the word ‘art’ as to painting. There is only one art and that is the art of living. Painting is an occupation that’s in that art and that occupation is the freest of all the occupations of living” (Foley, Declan J., 2009b, p. 30). The music here is Piano improvisation in B flat minor, shifting between 4/4 and 6/4 time signatures, and reflecting on nostalgic moods of temporality of the world, transitioning into brightening B flat major Piano and Violin song in a hymn choral chords manner.

The fifth episode is Old Walls (1945). For Jack B. Yeats, the world of memories is very important. Narration to this episode soundscape is: “In old age, Yeats painted several works of this kind, in which a solitary, elderly figure appears to wander in familiar territory. It seems reasonable to imagine that such figures, aged and reflective, represent the artist himself, revisiting places or past experiences and pondering his current circumsta- nces.” (NGI, 2021). The Piano improvisation is in D-minor/Dmajor, 3/4 time-signature, with the rhythm reminiscent of Debussy’s Footsteps in the Snow, where a slow pace of steps symbolises old age. There are bright interventions in D-major which reflect on innocent and happy memories of childhood. “Sligo was a bond for all of the Yeatses. It provided the lore and landscape for WBY’s earliest work and for ‘The Man and Echo’ and ‘The Black Tower’ … For Jack, too, the subjects to which he returned throughout his life in his writing and his paintings were the landscape and human cavalcade of those he met in his boyhood: sailors, ballad singers, circus performers, jockeys and, always, horses” (Murphy, 2009, p. 135). The second component of the music is the Irish song My Sligo Home, Pia- no/Violin arrangement in G major/G minor. The artist and his father John Butler Yeats had many philosophical and literature discussions. The narration

Psychogeography with Jack B. Yeats Art Sounding Gallery 35

from this episode is from a 1920 letter from father to son (Foley, Declan J., 2009c, p. 159). The sixth episode is Dinner Hour at the Docks (1928). NGI describes this as follows: “Yeats was drawn throughout the 1920ths to everyday urban activities and fleeting moments … In this instance, a docker eats in silence the meal brought to him by his wife, who sits by his side and leans attentively towards him. It is a moment of quiet and respite, but also of human conne- ction” (NGI, 2021). The chosen song for this episode is an English-language Irish folk song called The Wild Rover, E-major key, which is a popular song often performed live by trad musicians in Irish pubs.

THE LOCATIVE AR EXPERIENCE With Jack B. Yeats Art Sounding Gallery for blind people, we are offering an Augmented Reality (AR) Audio GPS based walking app to experience art with music. The app translates the visual expression of artworks by the music and narration to deliver the emotional content of the paintings to blind people. The App provides two modes: Dublin settings and Random location. In the first instance, the experience is intended to be located in the original settings of paintings, when possible. For example, the painting “Liffey Swim” shows a scene near O’Connell bridge in Dublin, and by situ- ating the corresponding AR encounter with the art, music and narration, the audience has a psychogeographical experience of the art. Concept video: https://rebrand.ly/jack-yeats-ar-prototype In Random mode, anywhere in the world, preferably in a park, the user can create himself/herself a type of psych- ogeographical experience of Dublin, augmenting their locations with our AR Art soundscapes. While this paper considers the experience for blind people primarily, it could also be used as a tour for sighted people, to experience the gallery together. For safety reasons (to avoid obstacles and difficult terrain) while using the app, the blind person should be accompanied by a sighted companion. For the sighted, the app contains a radar and a map to find the paintings, which are pinned to GPS locations and compass directions. Once found, the audience experiences the music and sound and has the psychoge- ographical experience after which they. can take a photo of the AR painting. For blind users, the random mode can be ajusted so that the person can scan images around 360° to experience the gallery.

CONCLUSION

In designing this app we used the main principle of SSDs for the blind – seeing with ears, providing translation of Jack B. Yeats’s Art experience with/th- rough music and narration. The composition process followed a synaesthesia model of cross-modal perception closely. The AR frame provides a psychoge- ographical effect: to experience the paintings’ locations in Dublin, or by the user augmenting his/her local location with Art soundscapes and mentally bringing the experience to Ireland.

The app could be used by the sighted and/or together with blind people as encouragement of outdoor activities and social inclusion into the visual world of the sighted. We hope that the app not only encourages associative

36 Rudenko and Haahr

thinking but also gives enjoyment to its users and we are planning further studies of user experience in the future.

REFERENCES

Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., Amedi, A., 2014. Eye- Music: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restor. Neurol. Neurosci. 32, 247–257. Bor, D., Rothen, N., Schwartzman, D., Clayton, S., Seth, A., 2014. Adults Can Be

Trained to Acquire Synesthetic Experiences. Sci. Rep. 4. Burgess, K., 2021. Click of the tongue helps blind people to navigate like bats. Times UK. Cytowic, R.E., 2018. Synesthesia. MIT Press, Cambridge, Massachusetts; London,

England. Foley, Declan J. (Ed.), 2009a. The Only Art of Jack. B. Yeats, Letters & Essays. The Lilliput Press, Dublin. Foley, Declan J. (Ed.), 2009b. Eamon Andrews interviews Jack B. Yeats on Radio

Éireann, in: The Only Art of Jack Yeats: Letters & Essays. The Lilliput Press, Dublin, pp. 29–31. Foley, Declan J. (Ed.), 2009c. Letters of John Butler Yeats to his son Jack B. Yeats: 1920-2, in: The Only Art of Jack Yeats: Letters & Essays. The Lilliput Press, Dublin, pp. 159–176. Hamilton-Fletcher, G., Mengucci, M., Medeiros, F., 2016. Synaestheatre: Sonifica-

tion of Coloured Objects in Space. Presented at the International Conference on Live Interfaces. Hamilton-Fletcher, G., Wright, T.D., Ward, J., 2015. Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device. Multisensory Res. 29 337–363. Kennedy, R., 2009. Jack Yeats and Dublin, in: Foley, Declan J. (Ed.), The Only Art

of Jack Yeats: Letters & Essays. The Lilliput Press, Dublin, pp. 150–158. Kotler, S., 2019. The Wristband that Gives You Superpowers: David Eagleman.

Neo.Life. Maurer, D., Mondloch, C.J., 2006. The infant as synesthete? Atten. Perform.

449–471. Meijer, P., 1992. An experimental system for auditory image representations. Biomed.

Eng. IEEE Trans. 39, 112–121. Meijer, P., n.d. The vOICe vision: seeing with sound. Murphy, M.M., 2009. “Of loyal nature and of noble mind”: Jack B. Yeats and his siblings, in: The Only Art of Jack Yeats: Letters & Essays. The Lilliput Press, Dublin, pp. 134–143. Parise, C., Spence, C., 2012. Audiovisual crossmodal correspondences and sound

symbolism: A study using the implicit association test. Experimental Brain Research 220(3-4):319-33. Proulx, M., Ptito, M., Amedi, A., 2014. . Neurosci. Biobehav. Rev. 41, 1–2. Ramachandran, V.S., Hubbard, E.M., 2001. Synaesthesia- A Window Into Perception, thought and Language. J. Conscious. Stud. 8, 3–34. Rudenko, S., Córdoba Serrano, M.J., 2017. Musical-Space Synaesthesia: Visualisa-

tion of Musical Texture. Brill Online Books J. Smith, D., 2014. Blind Sight: The Next Generation of Sensory Substitution Technology. URL http://blogs.discovermagazine.com/crux/2014/04/28/blind-sight-the- next-generation-of-sensory-substitution-technology/#.WQ9G7LzyuuV Ward, J., 2020. Synaesthesia as a Model System for Understanding Variation in the

Human Mind and Brain. Sussex University. Ward, Jamie, 2008. The frog who croaked blue : synesthesia and the mixing of the senses. Routledge, London and New York. Ward, J., Meijer, P., 2010. . Conscious. Cogn. 19, 492–500.

Cover of the proceedings volume Human Factors in Accessibility and Assistive Technology

Explore the complete digital volume

Human Factors in Accessibility and Assistive Technology

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-13-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →