Chenlu Li, Songhua Gao (2023). Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context. In: Usability and User Experience. AHFE Open Access, vol. 110, pp. 454–464. https://doi.org/10.54941/ahfe1003220
This paper focuses on the issue of how to design cultural relic information interaction in a mixed reality context. Taking the perspective of cultural relic information interaction design as the entry point, and combining the mixed reality context and characteristics, a design strategy for cultural relic information interaction design are proposed, which provides a basis and reference for cultural relic information interaction design and helps to improve the user experience. Using interdisciplinary cross-research and deductive methods, the design strategy are proposed at the level of strategy, scope, structure and presentation, and a practical application is carried out to validate the design strategy, using the Luohua bird embroidered jacket as an example. By analysing the concept and characteristics of mixed reality contexts, new opportunities for digital product design from the three dimensions of computer, human and environment are identified. The design of natural interaction techniques, design theories of natural human interaction behaviour, and theoretical support that can enhance user experience in rich application scenarios provide for the design of cultural relic information interaction. At the same time, it further extends the application of mixed reality contexts in the field of cultural relic information dissemination. Expand the explanatory power of cultural relic information. From the level of public awareness: it helps the public to better understand cultural relic information, gradually love traditional culture, improve their awareness of cultural relic information, and then develop a sense of cultural identity. From the level of cultural popularisation: to improve the breadth and depth of popularisation of cultural relics information, to help promote the dissemination and transmission of cultural relics information, while expanding the effective transmission of cultural relics information.
Mixed Reality (MR), also known as Hybrid Reality, emerged in the 1990s and is defined as any technology that lies between the real world and the digital world. The real world and the virtual world are mixed together to cre- ate a new digital environment. Virtual reality, augmented reality and mixed reality are all broadly defined as virtual reality. The experience where gra- phics are superimposed on a video stream of the physical world is called
Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context 455
augmented reality, and the experience where the user does not feel the phy- sical world is called virtual reality. Between these two extremes lies Mixed Reality. Mixed reality is the sum of technologies that lie between physical and digital reality. The main difference is that mixed reality emphasises natural interaction. Natural interaction can be summarised in three main input forms and four spatial technologies, including Gaze, Gesture and Voice, as well as Collaborative Sharing, Spatial Mapping, Spatial Coordinate and Spatial Sound.
Interaction is the communication of information between users and inte- ractive objects. Therefore, information interaction design in the context of Mixed Reality focuses more on the naturalness of the interaction and fits bet- ter with the user’s natural behaviour and cognitive model. It can make full use of the experience and skills users have acquired in real life and reduce their cognitive costs. Changing the traditional static display state, breaking the one-way nature of information transmission, realising real-time inte- raction and communication between people and objects, and meeting the interactivity of people’s behaviour. Rather than relying solely on traditional human-computer interaction such as clicking and swiping, users can interact ergonomically in a natural way with multimodal senses, meeting the natu- ral movements of people close to the real world when experiencing virtual information. The ultimate goal is to achieve perfect integration and real-time interaction between virtual information and the real world, providing users with a more natural and efficient way to interact.
When a user uses a product, the user interacts with the environment around them, which also creates a mixed reality environment consisting of a combi- nation of human, technological and environmental factors. Context refers to the environment in which events take place, hence the creation of mixed rea- lity contexts. The creation of new contexts opens up endless opportunities for design.
The current presentation of cultural relics is mostly in museums, exhibition halls and other environments, mostly in the form of display cabinets, exhibi- tion stands and other physical presentation, resulting in some audiences not being able to fully access the information of cultural relics, and the informa- tion of cultural relics not being fully interpreted, resulting in the phenomenon and problem of the public’s recognition of the cultural value of cultural relics information being reduced.
In response to this problem, the mixed reality context offers new opportu- nities for the design of human, computer and environmental factors. Firstly, it emphasises the human behavioural interaction and the communication of content carriers, usage scenarios, user needs. Secondly, it emphasises the understanding of the relationship between people and information. Mixed reality technology overlays existing technologies such as augmented reality and virtual reality on top of each other to create a mixed reality environ- ment. It provides a more natural way of interacting with people and focuses on understanding the relationship of the content itself to the information environment. Thirdly, it emphasises the way information is disseminated in a more artistic and natural way based on traditional reality, with diverse
456 Li and Gao
means of information dissemination to meet the needs of users and cultural communication needs in the current context.
INTERACTION DESIGN STRATEGY FOR CULTURAL RELIC INFORMATION IN A MIXED REALITY CONTEXT Based on the above theoretical research, this paper proposes design stra- tegies at the strategic, scope, structure and performance levels by applying interdisciplinary cross-research and deductive methods.
On the one hand, based on the five elements of the user experience theory proposed by Jesse James Garrett, the elements of user experience are progres- sively developed from the abstract to the concrete. The strategic layer focuses on satisfying user needs and achieving product goals, the scope layer classi- fies needs and translates them into corresponding functions and contents,. The structural layer visualises functionality and content into a conceptual framework through interaction design and information architecture; the framework layer defines the detailed interface appearance, navigation and information design, the presentation layer brings together functionality, con- tent and aesthetics in a logical structure and framework to produce the final design.
On the other hand, according to Nathan Shedroff (1999), a user experie- nce expert, the concept of information interaction is a new concept that is a cross between information design, interaction design and perceptual design. Information design is about the information that the design producer wants to express, interaction design is about the response of the product to the design consumer, and perceptual design is about design from the perspective of aesthetics and perceptual psychology.
Strategic Level Design Strategy Strategic level: in terms of product targeting, proposing refined user require- ments, clarifying the design strategy for content types.
This link is a prerequisite for information interaction design. As Cultu- ral Relic Information involves a variety of Interdisciplinary areas such as archaeology, behaviour and art, the research content is large and complex. When collected, the content needs to be categorised to help filter and orga- nise the content at a later stage. It provides a new design strategy for heritage researchers and enthusiasts to facilitate the generalisation of Cultural Relic Information and the design of applications.
Strategy 1: Refine user needs. Users are mainly located in the “brow- sing visitors”, “cultural relic researchers”, “cultural relic enthusiasts” three types of users. Through the analysis of these three types of users, refi- ning user needs. First, browsing visitors. Such users are mainly tourists, for the purpose of entertainment and leisure, quickly browse information, only superficial and understanding of information, generally based on the display of information ontology. Second, cultural relic researchers. Such users are mainly cultural conservation staff, scientific researchers and digital desi- gners. Not only do they have a need to display ontological information, but they also have a need for the deeper cultural content that can be derived
Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context 457
from ontological information. The content is filtered and refined from the perspective of conservation and design. Thirdly, cultural relic enthusiasts. This type of user is interested in cultural relic information and needs to under- stand the connotations of cultural relic in depth. The content is presented in a more visual and interesting way. The use of artistic means to generate aesthe- tic experience and emotional awareness, providing users with the opportunity to actively participate in the exploration of interactive communication.
Strategy 2, clarifying content types: the content of cultural relic infor- mation is categorised and interpreted as a way to provide pre-content planning preparation for mixed reality information interaction design. First, introduce and describe the ontological information of the garment. This includes basic physical information such as size, colour and shape, pat- tern, production process and cultural value. Secondly, introduce rele- vant information derived from the ontology extension. This includes both historical and social information and cultural and artistic information. Historical and social information includes: social influence, historical and cultural values, etc. Cultural and artistic information includes: aesthetic concepts, design ideas, cultural psychology, cultural spirit, connotation stories, etc.
Scope Layer Design Strategy Scope layer: In terms of content and functional design, proposes design strategies for screening key content, Refining key content and Mining transformative content.
The content of cultural relic information is complex and huge, and after the content has been classified at the pre-strategic level, the presented content needs to be transformed. On the one hand, combined with the key words of cultural relic information, the content of in-depth excavation, comprehen- sive and systematic collection of information. The information is collected through a variety of means, such as field research visits and access to lite- rature. Not only to collect explicit information, but also to dig into the implicit information contained in cultural relic information, to establish a cultural relic information content library, use coding to refine informa- tion, using specific categories, knowledge content and graphic elements as keywords, to create information refinement tables. Analyse information to further simplify it and distill valuable information. Reduce the cognitive load of the huge amount of information and refine the focus of the presenta- tion. Prepare the groundwork for the Information interaction design. On the other hand, after the collection and analysis of information in the early stages, a content distillation table of cultural relic information was obtai- ned. Using the name, cultural content, behavioural habits, technical features and artistic presentation as keywords, a content transformation table was created to transform a single piece of information into experiential infor- mation. Reduce the cognitive cost of the user experience and deepen the memory and understanding of the information. Provide cultural relic rese- archers and enthusiasts with a more in-depth way of thinking and depth of research.
458 Li and Gao
Structural Layer Design Strategy Structural layer: In terms of interaction design and implementation, the fol- lowing strategies are proposed: enhancing interaction efficiency based on the principles of natural interaction, strengthening multimodal sensory coopera- tion to enhance interaction experience and mobilise subconscious behaviour to enhance interaction participation.
Strategy 1: Increase the efficiency of interaction based on the principles of natural interaction. On the one hand, the range of interactive activities is designed according to the comfort zone of movement. Based on Alex Chu’s research on the comfortable range of motion, the available field of view and the range of head movement were determined. On the other hand, follows the concept of information anchoring. This is based on the concept of infor- mation anchoring by graphic designer Niteesh Yadav, where the anchoring controls the performance in different situations, in two natural ways: infor- mation is fixed in front of the head and information are fixed in space, thus increasing the efficiency of interaction.
Strategy 2: Enhancing multimodal sensory coordination to improve the interaction experience. Multimodal sensory interaction refers to the design of interactions that open up the single senses and optimize the user experience of working with multiple senses, such as vision, hearing and touch to achieve a more natural and intuitive interaction. On the one hand, natural intera- ction methods are used to strengthen the fit. Manipulation of virtual objects and communicate with virtual objects is achieved through simple and natural interaction methods such as gaze, gesture and voice. On the other hand, Enh- anced fit through multimodal sensory interaction script. The mixed reality multimodal interaction script is constructed from three aspects: user emo- tion, user intention and user interaction modality to enhance the interaction experience.
Strategy 3: Mobilise subconscious behaviour to enhance interactive enga- gement. Subconscious behaviour refers to behaviour that has a clear goal, but no clear motivation, i.e. Natural behaviour that users perform out of insti- nct and habitual experience. The technical support provided by the mixed reality context enables the design to bring a natural interaction experience to the user, thus enhancing interactive engagement. On the one hand, guided design is added to mobilise subconscious behaviour. By setting nodes, transi- tions, sound effects and other guidance, the design has directional feedback, guidance that fits the atmosphere of the scene and the special effects expres- sion to mobilise subconscious behaviour. On the other hand, in mixed reality contexts spatial sound technology can be used in combination with natural interaction methods to increase natural and reasonable feedback design and mobilise subconscious user behaviour. In interaction design, it is necessary to consider how users go about interacting and how to guide them so that they have a good interaction experience. Not only do designers need to consider the interface layout and information display, but they also need to combine the support provided by technology to realize the design and take advan- tage of technology to ultimately achieve the purpose of enhancing interactive participation.
Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context 459
Expression Layer Design Strategy Expression layer: In terms of visual expression design, the design strategies of scenario-based design to create a sense of reality, narrative design to build a sense of presence and artistic means to enhance expressiveness are proposed.
The visual design of the presentation layer is crucial. A realistic, natural and immersive visual experience is designed to help a person develop a better understanding of the Cultural Relic Information.
Strategy 1: Scene design creates a sense of authenticity. Scene design needs to be analysed according to the type and characteristics of different infor- mation and choose an appropriate presentation to present it. Reasonable arrangement of the positional relationship between the user and the virtual object. In scenario design first establish the location relationship between the environment and the user, then determine the size and location of objects in the environment according to the user, and finally determine the interface. In this way will determine the relative coordinates fastest and reduce varia- bles. Rationalising the position of the user in relation to the virtual objects improves the perceptibility and experience of the cultural space. This reduces the cognitive load on the user and enables them to understand and concise information in the actual space at high speed and convenience. Information should always be kept at a distance from the user and within the user’s visual range to facilitate access to the information. Allowing the user to have a three-dimensional, multi-layered experience of the scene.
Strategy 2: Narrative design builds a sense of immediacy. A narrative can be “thought of as a set of events that follow a particular sequence from beginning to end”. Narrative structures can be divided into linear narratives, hybrid narratives and exploratory narratives. The purpose of the narrative is to enhance the user’s memory and understanding of the information. Builds a storyline is established, scenarios are created, information representation is increased and the user has a sense of control over the experience. Grab the user’s attention with some highlighted design elements that prompt explo- ration, such as moving to that location point to trigger an event. Efficiently completes the transfer of information between the user and the system, deepe- ning the user’s memory. Through realistic rendering effects, agile and natural scene reproduction, and with more emotionally rich guided speech, users feel that these are part of the real world. Combined with a multi-sensory interactive experience, it quickly engages the experience, creates an overall atmosphere and conveys emotions and ideas.
Strategy 3: Artistic means to enhance expression. Artistic design can enhance expressiveness through multi-level artistic innovations such as pro- portional innovation, perspective innovation, style innovation and sound innovation, highlighting the shocking effect of details, echoing the theme and attracting users. Borrowing from the principles of artistry, clear details are displayed by enlarging the proportions of objects or magnifying local ele- ments. The digital model authentically restores the cultural relic, not only retaining the original information about the cultural relic, but also without losing the artistic underpinning. The first-person perspective that the user’s mixed reality device has, combined with the typical artistic elements detailed
460 Li and Gao
Figure 1: Luohua bird embroidery jacket (adapted from https://mp.weixin.qq.com/s/t3I- hPpf72BRCQcS0Q8_1g).
in the content layer, creates a surprising experience. The overall Style of work design is enhanced by extracting the inherent colours of the cultural relic to echo the theme of design. The use of spatial sound technology, which simu- lates the sound of nature and dynamic objects, enhances the Performance effects in terms of sound innovation. The design is artistic and innovative, based on the authenticity of the cultural relic, to enhance the visualisation.
Design Practice Information interaction design of Luohua bird embroidery jacket in mixed reality context, design theme is the Luohua Bird Embroidery Jacket, Luo refers to a traditional Chinese manufacturing process. The embroidery of birds and flowers refers to the content of the embroidered item. The design Jacket refers to the style of the lapel coat in clothing. The aim is to explore how the design strategy can be applied to the mixed reality interaction design of the Luohua bird embroidered jacket as far as possible, in order to validate the design strategy.
The design strategies were applied to the actual project practice. Firstly, in terms of user needs and content categories: three categories of users were identified, namely ‘browsing visitors’, ‘cultural relic researchers’ and ‘cultural relic enthusiasts’. The needs of the corresponding users were analysed. The content of the Luohua bird embroidered jacket was categorised and presen- ted. The content was prepared in advance for the Interaction design of the cultural relic. Secondly, in terms of content and functional design: a content library was created for the Luohua bird embroidered binder, the key content of the Luohua bird embroidered binder was extracted, and the content was transformed into a design. Third, in terms of interaction design and realisa- tion: The scene layout was designed according to the range of comfortable
Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context 461
Figure 2: High-fidelity interaction flow chart (painted by the author, 2022).
Figure available in the original PDF paper.
Figure 3: Mixed reality multimodal interaction script (painted by the author, 2022).
use, and the position of the user in relation to the virtual objects was reasona- bly arranged. The concept of information anchoring is followed to strengthen the efficiency of information visualisation. Natural interaction methods such as gestures, voice commands and eye-tracking are used, together with mixed reality multimodal interaction script, to strengthen multimodal sensory coo- peration and enhance the visualisation experience of Interaction information.
462 Li and Gao
Figure 4: Diagram of the presentation of the work (painted by the author, 2022).
Fourthly, in terms of visual performance design: a lotus flower, a goose and a butterfly are used as clues for storytelling and string design to enhance the sense of presence during user experience, realistic scenes are created through anthropomorphic design and realistic task design to increase the sense of rea- lity during user experience, by means of exaggerated physical dimensions, the display of information details, together with the overall style and dynamic design, enhances the visual impact of the user experience.
CONCLUSION
Through the analysis of the concept and characteristics of mixed reality contexts, this paper identifies new opportunities for digital product design from the computer dimension, the human dimension, and the environmen- tal dimension. It provides design theories on natural interaction techniques, natural human interaction behaviour, and theoretical support that can enha- nce user experience in rich application scenarios of cultural relic information interaction design. It provides feasible design strategies for the later stage of cultural relic information interaction design. Combining the mixed rea- lity context and features, the design strategy for cultural relic information interaction in the mixed reality context is proposed. At the strategic level, a design strategy is proposed to refine user needs, and to clarify the presentation content. At the scope level, a design strategy is proposed to filter key con- tent, refine key content, and explore transformed content. At the structural level, design strategies are proposed to enhance interaction efficiency based on natural interaction principles, enhance multimodal sensory cooperation to improve interaction experience, and mobilise subconscious behaviour to enhance interaction participation. In the performance layer, the design stra- tegies of scenario-based design to create a sense of reality, narrative design to build a sense of presence, and artistic design to enhance expression are proposed. Through design practice, the feasibility of the design strategies in practical application is verified.
Interaction Design Strategies for Cultural Relic Information in a Mixed Reality Context 463
At the same time, the mixed reality context has been further expanded in the field of cultural relic information dissemination. Extending the interpre- tive power of cultural relic information. From the level of public awareness: it helps the public to better understand cultural relic information, to gradually love traditional culture, to enhance their awareness of cultural relic infor- mation, and to develop a sense of identity with national culture. From the level of cultural promotion: Enhancing the breadth and depth of the popula- risation of cultural relic information helps to promote the dissemination and transmission of cultural relic information, while expanding the effective ways of transmission of cultural relic information.
Research limitations: Not enough attention has been paid to how to improve the efficiency of information interaction in a mixed reality context. The low maturity of the hardware device application process, among other things, hinders the promotion and application of mixed reality technology. More exploration is still needed at this stage on how to test the efficiency of information interaction even further.
ACKNOWLEDGMENT
The Ministry of Education Humanities and Social Sciences Planning Fund Project “Research on Immersive Interpretation System of Ancient City Ruins” Project No. 21YJAZH0211.
REFERENCES
Arnheim R. (2004) Visual Thinking, California: University of California Press. Aso Hajirasouli, Saeed Banihashemi, Anoma Kumarasuriyar, et al. (2021) Virtual Reality-based Digitisation for Endangered Heritage Sites: Theoretical Framework and Application x [J]. Journal of Cultural Heritage, No. 3 pp. 10. Azuma R, Baillot Y, Behringer R, et al. (2001) Recent Advances in Augmented Reality,
IEEE Computer Graphics and Applications Volume 21 No. 6. pp. 34–47. Büschel W, Chen J, Dachselt R, Drucker S, Dwyer T, Görg C, Isenberg T, Kerren A, North C and Stuerzlinger W. (2018). ‘Interaction for immersive analytics’. Mar- riott K, Schreiber F, Dwyer T, Klein K, Riche N H, Itoh T, Stuerzlinger W and Thomas B H, eds. Immersive Analytics. Cham: Springer, pp. 95–138. Bourguet M L. (2003). Designing and prototyping multi-modal commands. Proce-
edings of the IFIP TC13 International Conference on Human-Computer Intera- ction. Zurich, Switzerland: IOS Press, pp. 717–720. Garrett J. (2010) The Elements of User Experience: User-Centered Design for the Web and Beyond, California: New Riders Press. Isabelle Verhulst, Andy Woods, Laryssa Whittaker, et al. (2021) Do VR and AR Ver-
sions of an Immersive Cultural Experience Engender Different User Experiences? Computers in Human Behavior No. 7. pp. 10. Julius Pettersson, Petter Falkman. (2020) Human Movement Direction Classification using Virtual Reality and Eye Tracking. Procedia Manufacturing, No. 10 p. 10. Kathryn Smith, Mark Roughley, Samantha Harris, et al. (2020) From Ta-Kesh to Ta-
Kush: the Affordances of Digital, Haptic Visualisation for Heritage Accessibility. Digital Applications in Archaeology and Cultural Heritage, No. 8 pp. 10. LI Yang, HUANG Jin, TlAN Fang, et al. (2019) Gesture Interaction in Virtual Reality.
Virtual Reality & Intelligent Hardware, No. 1 pp. 84–112.
464 Li and Gao
Marie-Monique Schaper, Maria Santos, Laura Malinverni, et al. (2018) Learning about the Past through Situatedness, Embodied Exploration and Digital Augmen- tation of Cultural Heritage Sites. International Journal of Human-Computer Studies, No. 1 pp. 10. Richard Laing. (2020) Built Heritage Modelling and Visualisation: the Potential to
Engage with Issues of Heritage Value and Wider Participation. Developments in the Built Environment, No. 6 pp. 10. Ware C. (2012) Information Visualization: Perception for Design, Amsterdam: Elsevier. Yukang YAN, Xin YI, Chun YU, et al. (2019) Gesture-based Target Acquisition in
Virtual and Augmented Reality. Virtual Reality & Intelligent Hardware No. 3 pp. 276–289.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Usability and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-86-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional