Liting Huang (2026). Integrating Three Modalities into One Experience: A Case Study of 2024 DigiWave—DdDd. In: Human Interaction and Emerging Technologies (IHIET 2026). AHFE Open Access, vol. 236, pp. 28–38. https://doi.org/10.54941/ahfe1008079
Exhibition design today is moving towards multimodal experience and towards deepening the visitor's participation. Visitors meet physical interfaces in the built space, mobile interfaces on their own devices, and embodied experiences that come from how the body moves, attends and acts. Whether an exhibition succeeds depends less on any single modality than on how well the three are joined, so that the visitor follows one coherent journey instead of three parallel ones. This paper analyses 2024 DigiWave—DdDd, an exhibition on the cultural evolution of sound and digital media presented under Taiwan’s Ministry of Digital Affairs and recognised in Shopping Design’s Taiwan Design Best 100 awards, as a case in which three modalities were integrated by design. It examines how perforated melamine wall structures (physical interface), a mobile puzzle that ends in a personal digital instrument (mobile interface), and a final interactive sound installation in which visitors collectively shape music and projection (embodied experience) were designed to act as one experiential arc. From this case the paper derives four design principles: each modality should express the exhibition’s core metaphor in its own register; transitions between modalities should be staged as discoveries rather than procedural handoffs; the mobile interface should add agency rather than draw attention away from the physical space; and the closing experience should set the device aside and return the body to the centre. The paper closes with implications for designers working across these modalities in cultural and brand exhibitions.
A visitor moving through a contemporary exhibition is seldom engaged by only one kind of interface at a time. Within a single room there are physical interfaces built into the space, such as walls, surfaces, objects, light and architecture; mobile interfaces on the personal device in the visitor’s hand; and embodied experiences that arise from how the body moves, where it attends and what it does. Each has its own design lineage. Physical and tangible interaction draws on a tradition of materiality and craft. The mobile layer inherits the conventions of apps, audio guides and screens. The
Integrating Three Modalities into One Experience 29 embodied dimension belongs to a longer history of how galleries choreograph movement, attention and social conduct. What is distinctive today is that all three are present together, and usually by intention.
The quality of such an exhibition rarely follows from the strength of one modality alone. A striking physical set, a clever application and an impressive interactive finale can each be excellent on their own and still add up to a fragmented visit. The decisive question is one of integration: are the three modalities joined into a single journey, or merely placed on the same floor plan? A common failure illustrates this. When the mobile layer is poorly integrated, visitors look down at their screens and stop attending to the room around them. This is the familiar “head-down” problem, in which the device displaces the object it was meant to interpret (vom Lehn and Heath, 2003).
This paper examines one exhibition that treats integration as its central design problem. 2024 DigiWave—DdDd was the 2024 edition of Taiwan’s annual DigiWave festival, devoted to the cultural evolution of sound from the analogue to the digital era. It was presented under the guidance of Taiwan’s Ministry of Digital Affairs, curated by the studio ULTRACOMBOS, and recognised in Shopping Design’s Taiwan Design Best 100 awards. The exhibition is read here as a case in which physical, mobile and embodied modalities were deliberately composed into one experiential arc.
The paper makes two contributions. It first offers an analytical lens, three modalities and one experience, for reading and designing multi-modal exhibitions. It then derives four transferable design principles from the case, each grounded in a specific design decision and then generalised. The sections that follow review related work, set out the case-study method, present the analysis and the resulting principles, and close with implications for cultural and brand exhibition design.
MULTIMODAL INTERACTION IN EXHIBITION DESIGN Four bodies of work inform the analysis: embodied and tangible interaction; user experience and multisensory design; the museum visit as a designed experience; and the integration of material and digital media in exhibitions. A fifth strand, on personal mobile devices and the mobile-mediated visit, is taken up separately because the visitor’s own phone is central to this case.
Embodied and Tangible Interaction Dourish (2001) framed embodied interaction as the idea that meaning arises through practical, bodily engagement with the world rather than through abstract symbol manipulation. This provides a foundation for treating the body itself as an interface. Hornecker and Buur (2006) developed the related notion of tangible interaction, proposing a framework that links physical artefacts, the layout of physical space and the social interaction that space affords. Ciolfi’s (2015) work on embodiment and place adds that physical form and spatial arrangement are not neutral containers; they actively carry experience.
An exhibition can also be read as a designed user experience (UX). Research on UX shifted attention from the usability of tools to the quality of the experience as a whole, including its affective, aesthetic and meaningful dimensions, and treated that experience as something to design deliberately rather than leave to chance (Hassenzahl and Tractinsky, 2006). In cultural and commercial settings this has been described as the staging of memorable experiences that engage people personally (Pine and Gilmore, 1998), which bears directly on brand and cultural exhibitions. One strand of this work is especially relevant here. Human perception is multisensory, yet interactive systems have long privileged vision and, to a lesser degree, hearing, while touch and the bodily senses remain comparatively underused. Designing deliberately across the senses is held to produce richer and more engaging experiences with technology (Obrist et al., 2016). Within museums and galleries, a growing literature argues for multisensory curation that engages touch, sound, smell, memory and the experience of space rather than sight alone, as a route to more immersive and more inclusive visits (Levent and Pascual-Leone, 2014). This matters for an exhibition whose subject is sound. DdDd places the auditory sense in the foreground and ties it to the visual, tactile and spatial qualities of the room, the device and the body, so that multisensory coherence becomes a central design concern.
The Visit as a Designed Experience Falk and Dierking (2013) model the museum experience as the meeting of personal, sociocultural and physical contexts, and stress that what a visitor takes away depends as much on movement and company as on content. Studies of conduct in galleries show that engagement is social and collaborative, with visitors shaping one another’s attention and action around exhibits (vom Lehn, Heath and Hindmarsh, 2001). Research on attention and dwell time (Serrell, 1997; Bitgood, 2013) makes a related point: attention is a scarce resource that has to be designed for rather than assumed.
Integrating Material and Digital Media A substantial line of work holds that the strongest exhibition experiences are hybrid, weaving physical and digital resources together (Bannon et al., 2005). Petrelli et al. (2013) argue for returning materiality and physical interaction to the foreground of the visit while still conveying digital content, and Hornecker and Ciolfi (2019) survey two decades of human– computer interaction research in museums in the same spirit. The literature on mobile guides is more divided. Well-designed mobile interpretation can deepen engagement, but a recurring worry is that the device pulls attention away from the room and the object, so that it “displaces the object” it was meant to illuminate (vom Lehn and Heath, 2003). Two further ideas recur in design discussion. Flow describes a state of absorbed, intrinsically motivated engagement (Csikszentmihalyi, 1990). Affordance describes the relationship between an artefact’s perceptible properties and the actions it invites (Norman, 2013).
Integrating Three Modalities into One Experience 31 Personal Devices and the Mobile-Mediated Visit
The role of the mobile layer in exhibitions has shifted from museum-issued guide devices to the personal phones that visitors bring with them. Early human–computer interaction research already showed that a mobile device is not an add-on separate from the visit but is woven into its social fabric. Grinter et al. (2002) found that pairs of visitors used a guide device through shared listening, independent use, following one another and checking in on each other, and that these patterns depended on their social relationships and existing visiting strategies. Aoki et al. (2002) went further, designing mobile audio to support face-to-face conversation rather than to replace it. This line of work offers a more balanced position than the idea that the device displaces the object (vom Lehn and Heath, 2003): a personal device can scatter attention, but it can also act as social glue, and the difference lies in design.
The tension becomes sharper once visitors use their own phones. Collins, Mulholland and Zdrahal (2009) note that mobile technology has long been treated as an information device, which several evaluation studies found encourages a heads-down interaction with the screen rather than a hands-on engagement with exhibits; in response they designed an application that lets visitors use their own phones to bookmark, by text message, the things that interest them in the gallery, so that the personal device serves attention to the physical exhibits instead. A personal phone, familiar and already to hand, can thus become an extension of the self rather than an institutional prosthesis, but whether it integrates into the overall experience rather than intruding on it depends on whether the interaction is designed to direct attention back to the physical space.
On this basis, gamified and task-based design on the personal device has been used to turn looking into participating. Rubino et al. (2015) integrated a location-based mobile game into the visit and evaluated its effect on visitor behaviour and learning, and Ha et al. (2021) showed empirically that, compared with a non-gamified version, a gamified mobile question-asking app significantly increased visitors’ willingness to take part in group discussion. This echoes the broader turn in museum studies from the passive audience to the active participant (Simon, 2010). On this evidence, the contemporary mobile interface is not simply a channel for delivering information; it is part of the overall user experience, able to confer agency, connect people socially and reach across the time before, during and after a visit.
One dimension remains comparatively under-theorised, namely the orchestration of modalities. Benford et al. (2009) introduced the notion of trajectories, understanding complex user experiences as coherent journeys through hybrid structures, punctuated by a series of transitions and held together by orchestration. Yet most existing research studies a single modality in depth, whether tangible interfaces, mobile guides or immersive environments, rather than how physical, mobile and embodied modalities are composed into one visit. This paper takes up that gap by reading a single exhibition as an exercise in integrating the three modalities into one curve of agency.
METHODOLOGY
The study is a qualitative, interpretive single-case study (Yin, 2018) combined with a reflective analysis of design decisions. A single-case design suits the aim, which is analytical rather than statistical generalisation. The point is to develop and illustrate a framework that designers can carry to other exhibitions, not to estimate how often a phenomenon occurs across a population.
DdDd was chosen as a critical, information-rich case. It uses all three modalities of interest, physical, mobile and embodied, and it received external recognition through Shopping Design’s Taiwan Design Best 100, in the Annual Concept Exhibition category. That recognition marks it as an example of considered practice rather than an arbitrary instance. For studying integration, a case that integrates the modalities by design is more useful than a stronger example of any one modality on its own.
The analysis draws on direct experience of the exhibition, including a walkthrough of its seven themed rooms and the concluding interactive installation. It also draws on primary design documentation and published materials from the curatorial team (ULTRACOMBOS) and the spatial-design studio (UMEADOW), among them project statements, the visual-identity system and production credits.
The material was analysed in four steps. Each of the three modalities was first mapped on its own terms. The transitions between modalities and the overall sequence were then traced as an arc. The integration was read against the exhibition’s core metaphor, asking how each modality expressed it. Finally, four transferable design principles were drawn from recurring design decisions. The main limitation is that this is an interpretive analysis rather than a controlled visitor study. There are no systematic dwell-time, observational or survey data, so the claims about visitor experience are analytical inferences. The Conclusion returns to this point as a direction for future work.
RESEARCH RESULTS The Case: DdDd
DigiWave is an annual Taiwanese festival built around a single technological theme. The 2024 edition, DdDd, took as its subject the cultural evolution of sound from the analogue era to the digital present. The exhibition was presented under the guidance of the Ministry of Digital Affairs and its Administration for Digital Industries, executed by the Taiwan Industry- Academia Collaboration Association, curated by the studio ULTRACOMBOS (2024), and designed spatially by the studio UMEADOW. The title encodes the theme. The capital D evokes the major scale, while the lowercase d stands both for the half-note associated with Debussy and for “digital”, a nod to the widening range of what sound can be.
A linear display of recording devices and players would, the designers judged, feel didactic. Instead the team turned the whole venue into a single giant instrument. Across seven themed rooms, which run from the sonic logos of brands and the changing media that have carried recorded music
Integrating Three Modalities into One Experience 33 to instrument design and the use of artificial intelligence in sound, visitors operate small “adjusters” that shape the sound forming around them. Sixteen characters, each standing for a facet of sound, hold the interaction together across the rooms and are driven by software so that visitors can manipulate them. The organising idea, that the space is an instrument and the visitor is its player, is the core metaphor against which the three modalities can be read.
Physical Interface: Encoding the Core Metaphor in Built Space The most pervasive physical interface is the architecture itself. The walls are clad in custom perforated melamine panels whose hole pattern is taken directly from the grilles of audio equipment, so that the surfaces of the room read as the surface of a speaker and sound seems to pass visually through the space. In the terms of Hornecker and Buur (2006) this is a tangible- interaction decision, in which material form and spatial configuration are not a backdrop but a carrier of the exhibition’s meaning. Before a visitor has touched a screen or reached the finale, the room has already stated the central metaphor in a purely physical register: the gallery is the body of an instrument (see Figure 1). The seven themed rooms then give that metaphor narrative content, moving from the recognisable sound marks of everyday brands through the material history of recorded sound to questions about instruments and machine-generated audio.
Figure 1: Perforated melamine wall panels, their hole pattern derived from audio- equipment grilles, treating the gallery surfaces as the body of an instrument (physical interface). Source: UMEADOW, 2024 DigiWave—DdDd (spatial design), https://www. umeadow.com/project/dw2024.
Mobile Interface: Adding Agency Without Displacing Attention The personal device enters DdDd as a layer of play rather than as a narrator, and notably this layer runs on the visitor’s own phone. As noted above, mobile interpretation is often recorded as drawing attention away from exhibits, encouraging a heads-down rather than a hands-on interaction (vom Lehn and Heath, 2003; Collins, Mulholland and Zdrahal, 2009); DdDd avoids this risk partly because it uses a device the visitor already knows and carries, so the phone is closer to an extension of the self than to a borrowed museum
34 Huang device, which lowers the cost of learning the interface. Icons in the shape of musical staves are concealed in unexpected corners of the rooms, and visitors must use their phones to find and unlock them, each one opening a sound experience tied to its location; because the puzzle is hidden in the physical environment, the device sends the visitor looking outward at the room rather than downward at a feed. Attention is thereby directed back into the physical space, so that the mobile layer adds agency without displacing the object it was meant to illuminate.
The mobile interface also turns the visitor from a viewer into an author. The visit becomes a game in which the rooms are levels and the visitor is a player, while the device keeps a record of what has been found and, from the particular path the player has taken, generates a personalised digital instrument. This instrument is something the visitor has made rather than information passively received, which echoes the turn in museum studies from the passive audience to the active participant (Simon, 2010) and is consistent with evidence that gamified tasks on the personal device convert looking into participation (Rubino et al., 2015; Ha et al., 2021). If the physical interface lets the visitor perceive the instrument with the body, the mobile interface teaches them to compose with it.
The mobile interface matters most, however, because it does not keep the visitor on a private screen; it sends the personalised result back to the group. The generated digital instrument is portable and can be played with others, so the phone connects people rather than isolating them. Earlier research had already shown that a mobile device can be woven into the social fabric of a visit and can support, rather than replace, face-to-face interaction (Grinter et al., 2002; Aoki et al., 2002), and DdDd is built in just this way: the mobile layer is the hinge between individual agency and collective, embodied performance (see Figure 2). It is, then, the point in the visit where reception becomes authorship and the individual is drawn back into the group, rather than only one modality among three.
Figure 2: The smartphone puzzle: visitors locate hidden staff-notation icons that unlock sound experiences and, across the visit, generate a personalised digital instrument (mobile interface). Source: image courtesy of ULTRACOMBOS, 2024 DigiWave— DdDd, reproduced from ShoppingDesign (2024), https://www.shoppingdesign.com. tw/post/view/10976.
Integrating Three Modalities into One Experience 35 Embodied Experience: Relocating Agency to the Collective Body
The exhibition ends in the body rather than on the screen. In the final room a large interactive sound installation invites visitors to reshape the music of the space together while the same actions drive a live projection (see Figure 3). The personal device recedes, and agency shifts from the individual phone to the shared presence of the people in the room, whose movements and choices compose a common, momentary result. This is embodied and social interaction in the sense of Dourish (2001) and of the gallery-conduct studies (vom Lehn, Heath and Hindmarsh, 2001), in which meaning is made through bodies acting together in space. The metaphor reaches its conclusion here. Having understood the instrument as physical space and learned to compose with it through the device, the visitor now plays it with others through the body.
Figure 3: The concluding interactive sound installation, in which visitors collectively shape the music of the space and the live projection while the personal device recedes (embodied experience). Source: image courtesy of ULTRACOMBOS, 2024 DigiWave— DdDd, reproduced from ShoppingDesign (2024), https://www.shoppingdesign.com. tw/post/view/10976.
One Experience: The Integrating Arc The three modalities are best read not as parallel attractions but as a single connected arc held by one metaphor, expressed first in architecture, then on the device, then in the body. The visit moves through rising degrees of agency, from perceiving the instrument, to composing with it through the personal device, to playing it with others once the device is set aside, and the personal device is the hinge in this progression. The transitions are built to feel like discoveries rather than administrative steps: the visitor is never told to open an application, finds the mobile layer by noticing hidden notation, and leaves it behind at the climax when the body takes over. In the vocabulary of trajectories (Benford et al., 2009), the contribution of DdDd lies less in the strength of any single modality than in how it orchestrates the physical, mobile and embodied into one sustained curve of agency. In this sense three modalities form one experience: their integration comes less from technical handoffs than from a consistent metaphor and a single curve of agency running from perceiving, through composing, to performing together.
Four transferable design principles follow from the way DdDd composes its three modalities. Each is anchored in a specific decision already described above and then stated in general terms.
Distribute the core metaphor across registers. Each modality should express the central metaphor in its own material, so that the visitor meets one concept in several forms rather than several unrelated concepts. In DdDd the walls give the instrument its body, the device its act of composition, and the finale its performance, which treats physical form and spatial configuration as carriers of meaning rather than neutral containers (Hornecker and Buur, 2006). Stage transitions as discoveries. Junctions between modalities should be embedded in acts of discovery rather than signalled as procedural steps. Overt handoffs, such as a directive to scan a code or move to the next station, foreground the apparatus and break the sense of a continuous visit, whereas discovery sustains the absorbed, intrinsically motivated engagement associated with flow (Csikszentmihalyi, 1990).
Augment rather than displace attention. The mobile interface should deepen engagement with the physical space rather than replace it with a screen. Because mobile interpretation tends to draw attention away from the object it is meant to illuminate (vom Lehn and Heath, 2003), making progress depend on close attention to the room, as DdDd does, turns the device outward rather than inward.
Retire the device at the climax. A device-mediated exhibition works best when the device is not present throughout. The closing experience should be embodied and social, with the personal screen having done its work earlier in the sequence, so that agency passes to the shared activity of the visitors present (Dourish, 2001; vom Lehn, Heath and Hindmarsh, 2001).
CONCLUSION
Contemporary exhibitions increasingly work across physical, mobile and embodied modalities at once, and their success turns on integration rather than on the strength of any one modality. Read through the lens of three modalities and one experience, DdDd shows how a single core metaphor, the space as an instrument and the visitor as its player, can be carried in three registers and arranged as a rising curve of agency that ends in collective, embodied creation, and it is from this case that the four design principles set out above were derived.
These principles also point to a wider trajectory in exhibition user experience. The direction is towards orchestrated multi-modality, in which physical, mobile and embodied elements are composed into one directed arc and the personal device adds a layer of agency before stepping aside. The same observations carry into brand and cultural exhibitions, where designers usually have a built environment, a companion application and a signature interactive moment to coordinate, and where the temptation to let the application dominate the visit is strongest.
Integrating Three Modalities into One Experience 37 The study’s main limitation is its method. It is an interpretive analysis of one case rather than an empirical investigation of visitor behaviour. Future work should test the four principles through systematic visitor studies, including observation, dwell-time measurement and interviews, and should examine further cases at different points on the spectrum from environmental immersion to device-dominant interpretation, in order to see how widely the principles hold.
REFERENCES
Aoki, P.M., Grinter, R.E., Hurst, A., Szymanski, M.H., Thornton, J.D. and Woodruff, A. (2002) ‘Sotto Voce: exploring the interplay of conversation and mobile audio spaces’, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’02). New York: ACM, pp. 431–438. Bannon, L., Benford, S., Bowers, J. and Heath, C. (2005) ‘Hybrid design creates
innovative museum experiences’, Communications of the ACM, 48(3), pp. 62–65. Benford, S., Giannachi, G., Koleva, B. and Rodden, T. (2009) ‘From interaction to trajectories: Designing coherent journeys through user experiences’, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). New York: ACM, pp. 709–718. Bitgood, S. (2013) Attention and Value: The Keys to Understanding Museum Visitors.
Walnut Creek, CA: Left Coast Press. Ciolfi, L. (2015) ‘Embodiment and place experience in heritage technology design’, in Macdonald, S. and Rees Leahy, H. (eds.) The International Handbooks of Museum Studies. Chichester: Wiley. Collins, T.D., Mulholland, P. and Zdrahal, Z. (2009) ‘Using mobile phones to map
online community resources to a physical museum space’, International Journal of Web Based Communities, 5(1), pp. 18–32. Csikszentmihalyi, M. (1990) Flow: The Psychology of Optimal Experience. New York: Harper & Row. Dourish, P. (2001) Where the Action Is: The Foundations of Embodied Interaction.
Cambridge, MA: MIT Press. Falk, J.H. and Dierking, L.D. (2013) The Museum Experience Revisited. Walnut Creek, CA: Left Coast Press. Grinter, R.E., Aoki, P.M., Hurst, A., Szymanski, M.H., Thornton, J.D. and
Woodruff, A. (2002) ‘Revisiting the visit: Understanding how technology can shape the museum visit’, in Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (CSCW ’02). New York: ACM, pp. 146–155. Ha, J., Pérez Cortés, L.E., Su, M., Nelson, B.C., Bowman, C. and Bowman, J.D. (2021) ‘The impact of a gamified mobile question-asking app on museum visitor group interactions: An ICAP framing’, International Journal of Computer-Supported Collaborative Learning, 16(3), pp. 367–401. Hassenzahl, M. and Tractinsky, N. (2006) ‘User experience – a research agenda’,
Behaviour & Information Technology, 25(2), pp. 91–97. Hornecker, E. and Buur, J. (2006) ‘Getting a grip on tangible interaction: A framework on physical space and social interaction’, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’06). New York: ACM, pp. 437–446. Hornecker, E. and Ciolfi, L. (2019) Human–Computer Interactions in Museums. San
Rafael, CA: Morgan & Claypool. Levent, N. and Pascual-Leone, A. (eds.) (2014) The Multisensory Museum: Cross- Disciplinary Perspectives on Touch, Sound, Smell, Memory, and Space. Lanham, MD: Rowman & Littlefield.
New York: Basic Books. Obrist, M., Velasco, C., Vi, C., Ranasinghe, N., Israr, A., Cheok, A.D., Spence, C. and Gopalakrishnakone, P. (2016) ‘Sensing the future of HCI: touch, taste, and smell user interfaces’, Interactions, 23(5), pp. 40–49. Petrelli, D., Ciolfi, L., van Dijk, D., Hornecker, E., Not, E. and Schmidt, A. (2013)
‘Integrating material and digital: A new way for cultural heritage’, Interactions, 20(4), pp. 58–63. Pine, B.J. and Gilmore, J.H. (1998) ‘Welcome to the experience economy’, Harvard Business Review, 76(4), pp. 97–105. Rubino, I., Barberis, C., Xhembulla, J. and Malnati, G. (2015) ‘Integrating a
location-based mobile game in the museum visit: Evaluating visitors’ behaviour and learning’, Journal on Computing and Cultural Heritage, 8(3), Article 15. Serrell, B. (1997) ‘Paying attention: The duration and allocation of visitors’ time in museum exhibitions’, Curator: The Museum Journal, 40(2), pp. 108–125. Simon, N. (2010) The Participatory Museum. Santa Cruz, CA: Museum 2.0. ULTRACOMBOS (2024) 2024 DigiWave—DdDd [exhibition]. Taipei: Administration for Digital Industries, Ministry of Digital Affairs. vom Lehn, D. and Heath, C. (2003) ‘Displacing the object: mobile technologies and
interpretive resources’, in Proceedings of ICHIM 2003 (International Cultural Heritage Informatics Meeting). Paris: Archives & Museum Informatics. vom Lehn, D., Heath, C. and Hindmarsh, J. (2001) ‘Exhibiting interaction: conduct and collaboration in museums and galleries’, Symbolic Interaction, 24(2), pp. 189–216. Yin, R.K. (2018) Case Study Research and Applications: Design and Methods. 6th
edn. Thousand Oaks, CA: SAGE.
SEARCH THE AHFE DIGITAL BOOKSTORE BY ISBN
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2026)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.