Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Synesthetic Design and Intangible Heritage: Mixed Reality as a strategy for preserving and promoting endangered traditions

Publication year
Digital Object Identifier10.54941/ahfe1006715

Abstract

Intangible heritage is the name given to practices, expressions, different knowledge, and skills that several communities recognize as being part of their cultural identity. This includes aspects such as traditions, oral histories, rituals, performing arts, and craftsmanship passed down through generations. Intangible heritage, unlike its tangible counterpart, cannot be materialized. However, it plays a unique role in enhancing and deepening cultural diversity. More importantly, it fosters a sense of belonging, underlining the significance of preserving these living traditions. Intangible heritage now faces threats such as lifestyle changes, urbanization, and the decreasing interest of new generations in traditional cultural manifestations due to digital advancement and the influence of globalized companies.Intangible heritage, unlike its tangible counterpart, cannot be materialized. However, it plays a unique role in enhancing and deepening cultural diversity. More importantly, it fosters a sense of belonging, underlining the significance of preserving these living traditions.Studies state that virtual museums and platforms are effective in promoting intangible heritage and that technologies such as virtual and augmented reality provide effective experiences for its dissemination and preservation. However, limitations persist due to the predominant focus on audiovisual senses, neglecting others such as smell, taste, touch, proprioception, kinetic, vestibular, and thermal, limiting the user experience.Virtual and augmented reality are technologies that make it possible to experience a virtual world and add layers of information to the real one. Virtual reality (also known as VR) makes it possible to create an immersive digital environment where users can interact using audiovisuals such as VR goggles. Augmented reality (AR) allows digital elements to be added and superimposed to the real-world using devices such as smartphones or special glasses, providing an enriched perception of the natural environment. While VR offers users an entirely new experience, AR complements existing reality by integrating information and virtual elements in an interactive and visually appealing way, and both technologies have applications in gaming, education, training, and entertainment. However, being focused on the audiovisual senses, they leave other senses unstimulated, neglecting the ones such as smell, taste, touch, proprioception, kinetic, vestibular, and thermal, limiting the user experience.This study explores how, through synesthetic design, a multi-sensory approach can be conceived for virtual and augmented reality experiences. It uses robotics to stimulate different senses and analyse its impact on immersion and acquiring knowledge related to intangible heritage. It also aims to investigate how robotics can be used for sensory stimulation, what senses can stimulate, how they can be integrated into the immersive experience, and how this practice can be used to preserve intangible heritage.The results obtained will help to define synesthetic design practices, integrate the analyzed technologies into the preservation and dissemination of Portugal's intangible heritage and serve as a reference for future projects related to both synesthetic design and sensory stimulation practices.

© 2025. Published by AHFE Open Access. All rights reserved. 220

Synesthetic Design and Intangible Heritage: Mixed Reality as a Strategy 221

INTRODUCTION

The preservation of cultural heritage has long been a global concern addressed by international organizations such as UNESCO (United Nations Educational, Scientific and Cultural Organization), the leading institution for cataloguing and protecting heritage in its various forms: Cultural Heritage, Natural Heritage, Intangible Heritage, Underwater Cultural Heritage, Cultural Landscapes, and the Memory of the World. These categories have been officially recognized since 1972, following the launch of the World Heritage Convention, which aimed to safeguard cultural and natural assets such as monuments, historical ensembles, cultural sites, natural monuments, and landscapes.

Among these, Intangible Cultural Heritage (ICH)—which includes oral traditions and expressions, performing arts, social practices, celebrations, rituals, knowledge related to nature, and skills linked to traditional craftsmanship—was formally recognized through the Convention for the Safeguarding of the Intangible Cultural Heritage of 2003 (UNESCO, 2025). The main objectives of this convention are the safeguarding of intangible cultural heritage, respect for the cultural expressions of the communities, groups, and individuals concerned, raising awareness at local, national, and international levels of its importance, and promoting international cooperation and assistance (Marques, 2018, p. 53).

Portugal joined this global movement in 2011, with the inclusion of Fado, a popular urban song, in UNESCO’s Representative List of the Intangible Cultural Heritage of Humanity. Since then, several other elements have received recognition: the Mediterranean Diet in 2013 (alongside Cyprus, Croatia, Greece, Spain, Italy, and Morocco), with Tavira representing Portugal (Nutrição, 2025); Cante Alentejano in 2014; Falconry in 2016; the Clay Figures of Estremoz in 2017; the Podence Carnival in 2019; and the Campo Maior Festivities in 2021. Additionally, two practices are listed as in need of urgent safeguarding: the Manufacture of Cowbells (2015) and the Black Pottery Manufacturing Process of Bisalhães (2016) (Estrangeiros, 2025). In total, Portugal now has seven intangible cultural heritage elements recognized by UNESCO and two more under urgent safeguarding, within a global listing of 788 elements across 150 countries (UNESCO, Browse the Lists of Intangible Cultural Heritage and the Register of good safeguarding practices, 2025).

Despite international recognition, intangible cultural heritage continues to face serious threats such as urbanization, lifestyle changes, and declining interest from younger generations. New forms of cultural engagement and transmission are needed to keep these practices alive and relevant in a rapidly evolving digital world.

Recent studies have shown that virtual museums and digital platforms can play an important role in promoting intangible heritage, and that virtual and augmented reality technologies are particularly effective in creating engaging, immersive experiences. However, their predominant focus on audiovisual senses limits the depth and authenticity of these experiences, neglecting other

key senses such as smell, taste, touch, proprioception, kinetic, vestibular, and thermal inputs.

In this context, the present study explores how, through synesthetic design and robotic sensory stimulation, it is possible to broaden immersive VR/AR experiences beyond audiovisual channels. Using Portuguese folk legends as a case study, this research investigates how multi-sensory design can enhance the preservation and transmission of intangible cultural heritage in meaningful and innovative ways.

THEORETICAL BASIS The establishment of protection mechanisms reflects the existence of numerous direct and indirect threats that endanger the preservation, dissemination, and even the very survival of diverse cultural heritages. Among these threats, identified both in Portugal and globally, are the perception among younger generations that folklore and its narratives are outdated (Nurhuda & Abidin, 2021), and the declining number of traditional storytellers, largely due to significant lifestyle changes within local communities (Ogawa, Kobayashi, & Hoshino, 2021).

Regarding tourism, the economic valorization of intangible heritage often exerts considerable pressure on its original sources, which can lead to the erosion of local identity and the emergence of forced or stereotyped representations (Chasqueira, 2021).

In this context, the design of immersive virtual reality (VR) experiences has demonstrated significant potential across multiple fields, particularly with the growing development of the metaverse (Dincelli & Yayla, 2022). Research indicates that VR can be an effective tool for the safeguarding, dissemination, and exploration of various forms of intangible heritage (Silva & Teixeira, 2021). However, a notable gap remains in the stimulation of senses beyond the audiovisual and vestibular systems. While robotics has been employed for multisensory stimulation (Spence, 2022), the integration of these technologies into virtual and mixed reality applications is still emerging, and the design of such experiences plays a key role in developing market-ready solutions (Han, Bergs & Moorhou, 2022).

Studies suggest that the effective use of technology in the dissemination and preservation of intangible heritage must be grounded in principles of resource formation, to expand its potential applications (Povroznik, 2018). It is also essential to recognize that the sensory and cognitive connection to intangible heritage involves different types of experiential engagement, as it is intrinsically linked to cultural heritage (Lee, 2022).

This is the central question addressed by the present project. Grounded in design thinking principles and the development of serious games, this research seeks to explore how design can create a multisensory experience and investigate how the stimulation of various senses — not only the five traditional senses, but also others such as the vestibular, proprioceptive, kinesthetic, and thermal senses — can contribute to the creation of meaningful experiences aimed at promoting intangible heritage, using

Synesthetic Design and Intangible Heritage: Mixed Reality as a Strategy 223

Portuguese folk tales as a case study. Previous study made clear that most of them depend on a main sense for the narrative to happen, like the hero that touches, hear or see something (Costa, 2023).

By employing gamification strategies such as virtual environments, interactivity, graphical interfaces, metaphors, and feedback mechanisms (Salomão, Nassar, Vieira, & Gonçalves, 2021), the proposed technological approach aims to enhance engagement with traditional folk narratives among younger audiences (Ruslan, Hidayat, Muzdalipah, & Ginanjar, 2022). Moreover, several studies have demonstrated the effectiveness of virtual and augmented reality in stimulating cognitive and neural activity (Zanier, Zoerle, & Ler, 2018), and there are existing interface design processes with synesthetic characteristics that have been tested with real users (Silva, 2015).

There is a growing need for new design principles to support the development of experiences that incorporate entertainment robotics (Prattico & Lamberti, 2020), as well as further research, documentation, and content creation in this field (Cordeiro, Sousa, & Carvalho, 2021). Experiments with olfactory synesthesia in virtual reality have already been conducted, pointing to the need for further development, new approaches, and extensive testing (Barros, 2021).

Table 1: List of Portuguese tales and their dependence on a particular sense, as well as their usual classification in linguistic studies (Costa, 2023, p. 72).

Table available in the original PDF paper.

Sense Classic Category Tale of the Golden Calf Touch Supematural Tale of the Bicha- Moura of Saint Bartholomew Vision Supermatural Tale of the Moorish Woman Cave Vision Etiological Tale of the Moorish Woman of the Chaves Bridge Hearing Supermatural Tale of the Moorish Woman of Outeiro da Cruz Vision Supematural Tale of the Moorish Salúquia Vision Etiological Tale of Stone Soup Palate Etiological Tale of the Conquest of Aljezur Vision Historical Tale of the Screaming Zorra Hearing Supermatural Tale of the Almond Blossoms Vision Etiological Tale of Pedro and Inês Touch Historical Tale of the Comet of 1910 Vision Miscellaneous Tale of the Rooster of Barcelos Hearing Supematural Tale of the Black Cat Vision Urban Tale of the Holy Rosary Olafato Religious Tale of the Washerwoman’s Figs Hearing Supematural Tale of Saint Joan Princess Touch Religious

Accordingly, this project seeks to address some of the key challenges identified in the literature, contributing to the dissemination and preservation of intangible heritage through innovative, multisensory design approaches, like the ones on development on IPCA (Instituto Politécnico do Cávado e do Ave), in Portugal.

Figure 1: Ongoing experiments with synesthesia developed on IPCA, mixing VR with external stimulus through robotics.
Original PDF Figure 1:

INTANGIBLE CULTURAL HERITAGE AND LEGAL FRAMEWORK FOR THE SAFEGUARDING OF INTANGIBLE CULTURAL HERITAGE IN PORTUGAL The concept of Intangible Cultural Heritage pertains not to physical artifacts like buildings or objects, but rather to non-material aspects that convey the creative and cultural richness of diverse communities. Numerous initiatives have been undertaken in various disciplines—some officially safeguarded, while others stem from personal dedication, such as the compilation of folktales and oral histories by figures like Francisco Xavier D’Athaide Oliveira. As far back as 1898, Oliveira expressed the urgency of this work: “It is not fitting for an Algarve native who values and loves his province to stand idly by as our oral traditions crumble” (Oliveira, 2009, p. 9). Likewise, cookbooks such as Maria de Lourdes Modesto’s Cozinha Tradicional Portuguesa serve as significant contributions. In her introduction, she reflects on a “...thrilling survey, revealing centuries-old wisdom, customs, and traditions... that make us feel the pulse of life, of a family, of a region, of a country” (Modesto, 1982, p. 1).

To maintain and pass on Intangible Cultural Heritage, documentation and dissemination are regarded as key strategies—through processes of classification and transmission (Chasqueira, 2021). Alongside private efforts from scholars and cultural custodians, governmental frameworks have been set up to support the cataloging, recognition, and promotion of intangible traditions and knowledge.

In Portugal, the safeguarding of this heritage has been the responsibility of the Direção-Geral do Património Cultural since 2009, following the enforcement of Decree-Law No. 139/2009, dated June 15. This legislation “established the legal framework for the safeguarding of intangible cultural heritage, creating a legal protection system known as the ‘National Inventory of Intangible Cultural Heritage”’ (Ministros, Diário da República, 2015).

Synesthetic Design and Intangible Heritage: Mixed Reality as a Strategy 225

Thanks to this legal foundation, mechanisms are in place to enable the official registration, assessment, and protection of diverse forms of cultural expression across multiple domains.

DECREE-LAW No. 149/2015 of August 4 After the enactment of Decree-Law No. 139/2009, several amendments were introduced through subsequent laws in 2011 (Decree-Law 126-A), 2012 (Decree-Law 115 and 205), 2013 (Decree-Law 132), 2014, and 2015 (Decree-Law 167-A and 149), all aligned with Article 91(1) of Law No. 107/2001, which asserts: “For the purposes of this law, cultural heritage includes realities that, whether or not based on tangible or intangible assets, represent ethnographic or anthropological testimonies with civilizational or cultural value, significant for collective identity and memory” (Portugal, 2001). In accordance with this, Decree-Law No. 149/2015 outlines the process for including cultural manifestations in the protected list under the criteria established by the 2003 UNESCO Convention.

To be considered for inclusion, one must register the intangible cultural manifestation in the National Inventory via the Cultural Heritage website. It is then assessed by the Section for Museums, Conservation and Restoration, and Intangible Heritage.

Submissions must provide information regarding their cultural relevance, historical origins, current societal dynamics, means of transmission, potential risks, safeguarding proposals, compatibility with human rights, and sustainability alignment.

Two paths are offered: a regular classification for non-threatened heritage, or an urgent safeguarding category for at-risk traditions due to imminent threats.

The Directorate-General for Cultural Heritage oversees the review, initiating a public consultation via the Diário da República and sharing it with local authorities.

Once a final decision is made, it is published and made accessible online through the Directorate’s platforms.

The classification process upholds key principles, including early identification, community involvement, public access, and educational promotion. If approved, safeguarding policies will include measures for promoting and protecting traditions, establishing standards, offering technical support, funding research and projects, enhancing educational initiatives, and fostering cooperation with municipalities, academic institutions, and research bodies.

National classification—either regular or urgent—is a prerequisite for nomination to the UNESCO Representative List of the Intangible Cultural Heritage of Humanity (Ministros, Diário da República, 2015).

CONSULTATION AND CLASSIFICATION IN PORTUGAL The DGPC website (www.patrimoniocultural.gov.pt) offers tools to browse heritage items by category, such as Intangible, Architectural, Archaeological,

and Built Heritage. Users can find both classified and pending entries (Portuguesa, 2025).

For Intangible Heritage, proposals can be made in areas like agriculture, craft traditions, ceremonial practices, cuisine, artistic and musical expressions, performance arts, and more.

As of January 2025, 57 intangible heritage elements are registered in the DGPC system—16 fully classified and 41 still under evaluation.

METHODOLOGY

This qualitative, exploratory study combines Design Thinking and Serious Games to explore how synesthetic design, and robotic sensory stimulation can enhance immersive VR/AR experiences for the preservation of Intangible Cultural Heritage (ICH). Portuguese folk tales serve as a culturally grounded case study.

A five-stage Design Thinking process (empathize, define, ideate, prototype, test) ensures user-centered development. The Serious Games methodology integrates gamification elements—such as interactivity, feedback, and narrative progression—to enhance user engagement, particularly among younger audiences.

Multisensory stimulation—including haptic, thermal, olfactory, and proprioceptive cues—is delivered via robotic interfaces, expanding beyond traditional audiovisual inputs to deepen immersion and emotional resonance.

Participants will include students and young adults, recruited through purposive sampling, with snowball sampling used in traditional communities to collect cultural content. Data will be gathered through prototype testing, semi-structured interviews, and system interaction logs.

To evaluate the user experience, the study will employ standardized instruments such as the System Usability Scale (SUS) to assess usability, and the Igroup Presence Questionnaire (IPQ) to measure perceived presence and immersion. Additional custom questionnaires will evaluate emotional engagement and cultural resonance. Qualitative data will be analyzed thematically, while quantitative data, including interaction frequency, task completion time, and sensory engagement ratings will be examined using descriptive and inferential statistics to explore correlations between sensory inputs and user responses.

This approach builds on the literature supporting multisensory and embodied strategies in cultural education (Spence, 2022; Silva, 2015), aiming to contribute novel tools and frameworks for immersive heritage dissemination.

RESULTS AND DISCUSSION

This study presents the development of a VR/AR-based multisensory system grounded in Portuguese folk narratives, aimed at enhancing cultural transmission through immersive, interactive experiences supported by robotics.

Synesthetic Design and Intangible Heritage: Mixed Reality as a Strategy 227

The system integrates tactile, thermal, and olfactory stimuli to foster deeper emotional and cognitive engagement compared to conventional audiovisual formats. Gamification strategies are employed to increase relevance for younger, digitally native audiences by incorporating symbolic tasks and interactive metaphors derived from the stories.

Robotic elements and ambient actuators will be used to deliver targeted sensory feedback—such as motion, temperature shifts, and scents—at key narrative moments, thereby enhancing realism and multimodal immersion. To ensure cultural accuracy and ethical representation, the design process will involve collaboration with local communities and cultural experts.

The project also addresses anticipated challenges related to technical feasibility, cost, and sensory integration. Future stages will emphasize iterative user testing and optimization, with applications envisioned in educational and museum settings.

CONCLUSION

Under the framework of Portuguese legislation on the preservation of intangible cultural heritage, this study highlights the critical role of synesthetic design in enhancing the dissemination and safeguarding of cultural traditions through immersive technologies such as virtual and augmented reality. While VR and AR have already demonstrated their potential in creating engaging experiences, their predominant focus on audiovisual stimulation often limits the depth of user engagement and cultural immersion.

To address this gap, we propose an exploratory, multisensory approach that incorporates robotics to stimulate additional senses such as touch, proprioception, and thermal perception. By engaging a broader spectrum of sensory modalities, this research aims to foster more profound and emotionally resonant interactions with intangible heritage—particularly for younger, digitally native audiences.

Our findings emphasize the necessity of developing innovative methodologies that bridge traditional narratives with emerging technologies, contributing to more inclusive and sustainable forms of cultural transmission. Although still in the development phase, this project seeks to lay the groundwork for a new paradigm in cultural preservation—one that combines design thinking, sensory interaction, and embodied storytelling to enrich the ways in which heritage is experienced and understood.

Future research should continue to explore how synesthetic and robotic design can be standardized and effectively integrated into virtual heritage platforms, ensuring accessibility, cultural relevance, and long-term impact. Ultimately, this multisensory approach holds the potential to create more meaningful and engaging ways of preserving endangered traditions—not only in Portugal but across diverse cultural contexts worldwide.

ACKNOWLEDGMENT

This work was supported by UNIDCOM/IADE through the Fundação para Ciência e Tecnologia (FCT) attributed to UNIDCOM/IADE–Unidade de

Investigação em Design e Comunicação, Lisbon, Portugal, under Grant UIDP/00711/2020.

REFERENCES

Barros, J. A. (2021). Sinestesia Expandida: Experiencia Multissensorial Aromatica em Realidade Virtual. Chasqueira, Â. L. (2021). O Futuro do Passado - Conservação do Património

Imaterial Português. Faro: Techn&Art - Centro de Tecnologia, Restauro e Valorização das Artes - Universidade do Algarve. Cordeiro, P. N., Sousa, J. P., & Carvalho, A. (2021). Digitization and Gamification in Cultural Heritage: The portuguese context in the framework of national and international policies and some practical examples. 16th Iberian Conference on Information Systems and Technologies (CISTI). Costa, P. R. (2023). Lendas de Portugal - Da Tradição Oral à Representação Digital.

Lisboa: Faculdade de Belas Artes da Universidade de Lisboa. Dincelli, E., & Yayla, A. (2022). Immersive virtual reality in the age of the Metaverse: A hybrid-narrative review based on the technology affordance perspective. Estrangeiros, M. d. (2025). Delegação Permanente de Portugal Junto da Unesco.

Retrieved from Património Cultural Imaterial: https://unesco.missaoportugal. mne.gov.pt/pt/portugal-e-a-unesco/cultura/patrimonio-cultural-imaterial. Han, D.-I. D., Bergs, Y., & Moorhou, N. (2022). Virtual reality consumer experience escapes: preparing for the metaverse. Lee, H.-J. (2022). Phenomenological approach on the role of sonic heritage as an

intangible heritage. Journal of Social Sciences Research, Portugal, v. 3, n. 1, pp. 111–133. Marques, L. (2018). Património Cultural Imateial - O Olhar Antropológico. Porto: Afrontamento. Ministros, P. d. (2015). Diário da República. Retrieved from Diário da República:

https://diariodarepublica.pt/dr/detalhe/decreto-lei/149-2015-69935162. Modesto, M. d. (1982). Cozinha Treadicional Portuguesa. Lisboa: Verbo. Nurhuda, T., & Abidin, Y. (2021). Mixed Reality (MR) in Folklore Learning.

Proceedings of the 3rd International Conference on Educational Science and Teacher Profession, pp. 62–69. Nutrição, A. P. (2025). Dieta Mediterrânica. Retrieved from Associação Portuguesa de Nutrição: https://www.apn.org.pt/v0D0O0F/dieta-mediterranica. Ogawa, T., Kobayashi, K., & Hoshino, J. (2021). The First-Person VR System

Augmenting Folklore Experiences. 2021 Nicograph International (NicoInt), pp. 82–85. Oliveira, F. X. (2009). As Mouras Encantadas e os Encantamentos no Algarve.

Lisboa: Arquimedes Livros. Portugal, M. P. (2001). Procuradoria Geral Regional de Lisboa. Retrieved from Procuradoria Geral Regional de Lisboa: https://www.pgdlisboa.pt/leis/ lei_mostra_articulado.php?nid=844&tabela=leis. Portuguesa, R. (2025). Património Cultural. Retrieved from Património Cultural: https://www.patrimoniocultural.gov.pt/. Povroznik, N. (2018). Virtual Museums and Cultural Heritage: Challenges and

Solutions. The Digital Humanities in the Nordic Countries 3rd. Prattico, F. G., & Lamberti, F. (2020). Mixed-Reality Robotic Games: Design Guidelines for Effective Entertainment With Consumer Robots. IEEE Consumer Electronics Magazine.

Synesthetic Design and Intangible Heritage: Mixed Reality as a Strategy 229

Ruslan, T. S., Hidayat, E. W., Muzdalipah, I., & Ginanjar, A. A. (2022).

Transformation of Folklore Texts into Interactive Multimedia Digital Forms as Blended Learning Teaching Material. Proceedings of the Sixth International Conference on Language, Literature, Culture, and Education. Salomão, A., Nassar, V., Vieira, M., & Gonçalves, B. S. (2021). Critérios para análise de interfaces de Serious Games em realidade virtual. Silva, G. d. (2015). Sinestesia em Interfaces de Usuário: Modelando o Design da

Interação através dos Sentidos Humanos. Silva, M., & Teixeira, L. (2021). “eXtended Reality (XR) Experiences in Museums for Cultural heritage: A systematic review”. Spence, C. (2022). From the fairground sensorium to the digitalization of bodily

entertainment: Commercializing multisensory entertainments involving the bodily senses. The Senses and Society, pp. 153–169. UNESCO. (2025). Browse the Lists of Intangible Cultural Heritage and the Register of good safeguarding practices. Retrieved from Browse the Lists of Intangible Cultural Heritage and the Register of good safeguarding practices: https:// ich.unesco.org/en/lists. Zanier, E., Zoerle, T., & Ler, D. (2018). Virtual Reality for Traumatic Brain Injury.

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET 2025)

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET 2025)

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-73-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →