Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Visual Narrative Design of Text in Augmented Reality Interactive Experience

China Academy of Art, School of Design and Art, Hangzhou 310000, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1004226

Abstract

With the rapid development of digital society, augmented reality technologies are creating rich and engaging user experiences that are changing the way information is transmitted. As a key element in conveying information, telling stories, and constructing user experiences, the visual narrative design of the text is facing the challenge of moving from a traditional medium to a digital interactive medium. In the context of augmented reality, many researchers have focused on how to effectively use text for visual narrative design to create appealing visuals with clear and easy-to-read messages and themes. This paper examines the characteristics and functions of different types of text from the perspective of visual narrative, taking the origin, development, and construction of text as the starting point. Based on the properties of augmented reality technology, a model of textual visual narrative design strategy applicable to interactive application scenarios is constructed. Based on the theoretical study, an evaluation is carried out with actual augmented reality interaction design cases. Based on user feedback and the effectiveness of the design cases, the design model is optimized and improved to ensure that augmented reality technology can bring immersive and imaginative interaction experiences to the visual narrative design of text. This study helps to deeply understand the inner logic and process of textual visual narrative, enrich and expand its application theory in the field of interaction, improve the efficiency and visual effect of textual information dissemination, and provide some guidance and reference for the future development of the textual narrative.

INTRODUCTION

There are two ways of textual narrative: verbal narrative and visual narra- tive. The textual narrative explored in this paper is based on vision, combined with the technical characteristics of AR, and is carried out through three links: text construction, visual transformation, and AR interaction. Compared with traditional textual narrative, the core of textual interactive narrative lies in the visual representation of text and the role transformation of interactive narrative. AR text design is an emerging field. The application of interac- tive technologies in virtual environments can stimulate cognitive curiosity and have a positive impact on the reading of texts and the communication of information (Cheng, 2023). The design of digital interactive narratives can create many new storytelling structures and forms of representation that

© 2023. Published by AHFE Open Access. All rights reserved. 93

94 Zeng and Zheng

result in unique experiences (Murray, 2023). The intervention of digital inter- active narrative and AR technology has brought more diverse and flexible forms of visual expression to text design, providing audiences with a per- sonalized experience and enriching the depth and interactivity of textual information conveyance. Although interactive digital narrative has a long history of development, there is less scholarly knowledge on the subject and there is a need to utilize qualitative or quantitative design methods to improve the state of research (Roth & Koenitz, 2017). Therefore, how to develop a sound methodological theory in the field of design that combines interactive technology and textual narrative has become the focus of research.

In this paper, the text is mainly categorized into two types: ideograms and phonograms, and Chinese characters and English, which are representative of the two types of text, are selected as research objects. Firstly, by analysing the visual narrative features of the characters, the construction ideas of different characters for story texts are established. Secondly, we compare the differen- tiated performance of commonly used narrative modes in AR interaction and traditional carriers. We use this as a basis to explore different perspectives of text visual design in interactive environments and maximize the advantages of interactive media. Finally, based on exploring the previous theoretical meth- ods, the interactive exhibition work “Jiangnan Made of Water” was created with the narrative theme of Jiangnan Water Town. By applying the ideas and methods of constructing story text and interactive narrative plot proposed in this paper in the design process, the narrative ability and visual expres- siveness of text in the interactive field are improved, and how to create more colourful text visual narrative design is further explored.

CHARACTERIZATION AND CONTENT CONSTRUCTION OF TEXTUAL VISUAL NARRATIVES Visual Narrative Characteristics of Ideogram and Phonogram

Scripts diverged from early development to later stages, where some of them began to deviate from the visual and became phonograms. In comparison, another category of ideograms continued to develop along with visuality, which can be perceived and interpreted visually and intuitively (Ma, 2020). Different ideas can be provided for establishing narrative texts by comparing ideograms and phonograms.

Chinese characters are typical representatives of ideograms characterized by symbolism, composite, and spatiality. Oracle bone inscriptions serve as the primary source of Chinese characters, and the shapes of many Chinese characters originate from specific images. Although Chinese characters have been simplified, they can still express the objects or concepts represented by the semantics of the words more intuitively. Secondly, the composite char- acterization stems from the way Chinese characters are constructed. Many Chinese characters consist of different radicals and parts. A radical represents a component, a thing, or a phenomenon. By splitting and restructuring, the meaning of a Chinese character can be further visualized, thus showing richer meanings. In addition, Chinese characters are characterized by spatiality. As a square character, the staggering and turning of the strokes produce the

Visual Narrative Design of Text in Augmented Reality Interactive Experience 95

space of division and negative form. In calligraphy, the weight and urgency of the strokes, the angle of the strokes, and the intensity of the ink and brush strokes can create a vivid sense of space between the brush and the paper. The solid narrative nature of Chinese characters gives an inherent advantage to introducing narrative thinking, providing rich inspiration and possibilities for visual design (Li, 2015).

Regarding the direction of development and composition, the visual nar- rative basis of phonograms could be more substantial, with a certain degree of abstraction. Taking English as an example, separate letters have no spe- cific meaning, and the narrative mainly relies on the combination of letters to correspond to a specific language, and only after combining them into words and sentences do they have the ability to narrate. Therefore, the individual use or random combination of letters can only be used as a purely abstract visual element with no direct information or emotion. The radical departure of this class of sound words from their original visual representation is an excellent challenge for visual narratives. It requires a specific approach to add visual character to their properties.

Content Construction of Textual Narratives Establishing a narrative theme is the basis for building a story text. Design- ers need to analyse and study what they want to describe, then refine and summarize the relevant information text to propose a narrative theme that best highlights the characteristics of the object. After completing this stage of work, the construction of the textual narrative content can be carried out (Ma, 2018).

In the face of different narrative themes, the source of the story text can either be the analysis and expression of the semantics of the theme itself, or the theme meaning can be used as the story’s background to find other relevant texts. At the same time, according to the narrative characteristics of different types of texts, there are various choices in the amount of tex- tual content (see Figure 1). Ideograms can tell a complete story by expressing the theme of the narrative through a single character due to their symbolic solid and composite characteristics. In contrast, the individual letters of the phonograms cannot express specific content and require the construc- tion of complete utterances when expressing themes. Therefore, ideograms and phonograms have different ways of expressing narrative content, and different choices should be made according to the specific situation.

Figure 1: The link between textual, visual narrative features and content construction.
Original PDF Figure 1:

96 Zeng and Zheng

NARRATIVE AND TEXTUAL VISUAL DESIGN FOR AR INTERACTION AR Interactive Narrative

Before the visualization design of text, it is necessary to understand the nar- rative mode of AR interactive media so that the content theme, visual form, and media application of text can achieve a perfect combination. As interac- tive digital narratives become more interesting and creative, their experiential mechanisms become more closely aligned with narrative goals, generating patterns that differ from traditional narratives (Millard, 2023). Narrative in AR, on the one hand, can be accomplished through the viewer’s experi- ence, that is, the interactive narrative mode. On the other hand, dynamic and three-dimensional visual effects can be used to show vivid and three- dimensional narrative scenes. The plot setting with interactive features has gradually become the mainstream of narrative design, mainly manifested in the triggered plot and randomized plot.

In the AR scene, the trigger plot will set up the plot nodes and story direc- tion in advance, allowing users to trigger events to advance the development of the storyline. Randomized plots set up different contents or scenes of the narrative, and the user can randomly choose the plot’s direction. These inter- active plot settings enable users to participate more actively in and shape the direction of the narrative, providing a richer and more fascinating narrative experience. At the same time, AR interactions also allow for more flexible expression of traditional narrative carriers such as forward, backward, paral- lel, and interlude narratives, providing a variety of perspectives for modeling and arranging textual content. The plot of positive and negative narratives can also utilize the back-and-forth relationship of space to plan the sequen- tial order of the text, thus guiding the browsing of the text. Interactive media make up for the limitations of traditional visual narrative media in the dimen- sions of time and space. Text can be used to tell stories in the fusion of virtual and natural environments, and narrative plots can be more fully and multidi- mensionally expressed with the characteristics of nonlinearity, spatiality, and interactivity (see Figure 2).

Figure 2: Plotting in AR interactive narratives.
Original PDF Figure 2:

Text Visual Design in AR As the subject of visual narrative, the text needs to be converted from a conceptual form of thinking to a thematic visual form of expression (Zhou,

Visual Narrative Design of Text in Augmented Reality Interactive Experience 97

2020). Different textual forms have their own visual narrative qualities and expressions in the visual design of textual narratives. The internal structure of text, text glyphs, and the combination of multiple texts can all be objects of design. According to this categorization, text narrative design mainly has three ways: text disassembled narrative, text form narrative, and multiple text arrangement and combination narrative.

The narrative of text disassembly is to disassemble or reorganize one or more characters. Chinese characters, composed of different components, can be disassembled from morphological, semantic, historical, and cul- tural perspectives. The disassembled strokes and components can be used as visual elements for graphic composition design to excitingly deepen people’s understanding of words and texts.

The design of glyphs can be based on the meaning of the text or the nar- rative theme. In the case of Chinese character glyphs, typeface design can be done in a way that echoes form and meaning. Based on mastering the meaning of the Chinese characters, the form of the strokes and structure of the characters can be changed to produce a visual image that is closer to the meaning of the characters. For the design of English glyphs, although the individual letters themselves do not point to the meaning, they can be com- bined with the story’s theme and the connotation of the words and phrases in the visual design to achieve the unity of meaning and glyph design. Using graphics, colour, texture, and other visual elements to express style, emotion, or imagery makes it possible to make the form of the font convey a specific message or story.

The typographic design focuses on multi-text narratives and applies to all text types. Multi-text narratives are more closely related to linguistic narra- tives, and complex and diverse narrative activities can be realized through the arrangement of words. Most of the traditional choreography design is based on books and magazines. In contrast, the text choreography in AR can adopt dynamic design means and combine with natural space scenes to make the text layout have a three-dimensional form. In the AR environment, the text is no longer flat or static. However, it can form a dynamically changing entity by changing the viewer’s perspective and position, providing more creative space for text visual design.

TEXTUAL VISUAL DESIGN IN AR INTERACTIVE NARRATIVES In modern society, when people create new user experiences through interac- tive products and services as a medium, it has an artistic infectious power that transcends reality and can subvert the traditional manipulation experience. The application of interactive narratives for verbal communication is also becoming more and more widespread, and its artistic attributes are constantly changing and extending. Based on in-depth research on textual narrative features, content construction, AR narrative mode, and visual design, this paper explores the interactive narrative application of textual elements in presenting exhibition information. The authors created an interactive experi- ence of the exhibition space “Jiangnan Made of Water”. This work takes the poems expressing the four seasons of the Jiangnan water town as the narrative

98 Zeng and Zheng

text and combines them with AR interactive technology to bring the view- ers an immersive experience of the Jiangnan water town. Compared with the presentation of textual information in traditional exhibitions, the narra- tive in AR pays more attention to the interactive relationship between the theme, text, narrative experience, and acceptance. It penetrates different sce- narios and application fields, generating diversified visual art and practical, functional values.

Design Ideas for “Jiangnan Made of Water” Ancient Chinese poems are closely related to historical and cultural elements. These poems are deeply rooted in China’s historical and cultural traditions and extensively absorbed elements from the surrounding natural landscape, incorporating them into rich and vivid poetic expressions. The water towns of Jiangnan, with their rich historical and cultural heritage and beautiful natural scenery, have also inspired many Chinese poems. At the same time, ancient poets often inscribed poems on the walls while having fun so that the words and the spatial context could blend. These fundamental text elements in 3D physical space can provide rich references and inspirations for the text design of AR scenes, making the connection between the virtual world and the natu- ral world closer and enhancing the immersion experience of users. At present, many exhibitions of Jiangnan culture mostly show the historical and cultural life of the past through ancient artifacts and relics, and there are very few ways of expression that use text as the visual subject for exhibition narra- tives. Therefore, the exhibition space design work “Jiangnan Made of Water” expresses the culture and scenery of the Jiangnan water town from the unique perspective of poetry, shows the embedded cultural imagery in the AR virtual scene, and lets the viewers feel the history, culture and natural beauty of the Jiangnan water town in an immersive way.

Narrative Text Construction and Interactive Plot Setting for the Jiangnan Water Town Theme Throughout the ages, the literati have depicted and praised Jiangnan with- out water. Water is a classic business card of Jiangnan culture and the core of Jiangnan culture. It has infiltrated all aspects of society, culture, and life in Jiangnan for thousands of years. In the combing stage of the information text, the development of water-related written records in Jiangnan is shown through infographics, which deepens the knowledge of Jiangnan water cul- ture (see Figure 3). The construction of narrative content will extract the poems of Jiangnan water town from the information text of the preliminary research. In selecting poems, to more completely and vividly depict the differ- ent landscapes of Jiangnan water towns, the author takes the four seasons of Jiangnan as the time clues and narrative logic and extracts four poems that can represent the four seasons of Jiangnan.

In the setting of the narrative plot, randomized and triggered plots from interactive narratives were used. In the ring-shaped space, the author sets up different plot directions. On the one hand, one can view the changing textual landscape of the four seasons sequentially through the clockwise or

Visual Narrative Design of Text in Augmented Reality Interactive Experience 99

counterclockwise viewing flow according to the setting of the triggered plot. On the other hand, one can also walk around freely in the space, scanning the images on different screens to form the textual landscapes of Jiangnan poems in the corresponding seasons (see Figure 4). Different viewing routes can lead to the formation of different storylines. Poetry is traditionally presented on paper carriers for readers to read in order. In the AR interactive application, viewers can watch the poems of different seasons in a non-linear way, and the text generates random interactions and communication with the viewers in the virtual interface. The narrative mode moves from linear narrative to spatial multi-linear narrative. In Jiangnan’s Made of Water exhibition space, text reading no longer relies on linear or sequential logic. However, it allows readers to comprehend information in multiple sequences and perspectives according to their choices, needs, or contextual relevance.

Figure 3: Extraction of textual elements related to the Jiangnan water town culture.
Original PDF Figure 3:
Figure 4: Schematic of the different viewing routes in the AR experience.
Original PDF Figure 4:

Textual Interactive Landscape Presentation of “Jiangnan Made of Water” Entering the exhibition hall, the screens around the room show images of the scenery in each of the four seasons in Jiangnan. In Jiangnan, in spring, the West Lake is shrouded in a haze of light rain, reflecting the beautiful green mountains. Summer in Jiangnan shows the season of ripening plums. In the fall, when the tide comes in, phoenixes and rain float in the air. The win- ter in Jiangnan shows the “Broken Bridge and Broken Snow” scene in the

100 Zeng and Zheng

Ten Scenes of West Lake. In the face of constantly changing dynamic images, the viewer can use the cell phone to scan and identify different on-screen landscapes, and the cell phone interface will present the corresponding text poems (see Figure 5). AR technology makes the text information in these virtual interfaces produce different perspectives and typography, providing users with a more personalized and participatory visual experience. In terms of visual design, through the design of the text’s direction, dynamics, and spa- tial layout, the text in AR and the image of the Jiangnan water town combine to form a poetic scene. The text no longer just exists in a two-dimensional plane, but a virtual text landscape with three-dimensional properties. This fusion of natural space and text elements makes users feel as if they are in the story and creates a new and colourful visual narrative experience for them.

Figure 5: Virtual landscape of Chinese character poetry texts in AR interaction.
Original PDF Figure 5:

CONCLUSION

This study aims to discuss the design of text from the perspective of AR interactive narrative and analyzes the visual narrative of the text itself and AR interactive applications, respectively. Because different texts have differ- ent emphases on visual narrative, the construction of narrative content needs to be based on the characteristics of the text itself. At the same time, AR interaction facilitates the expression of diversified narratives and expands the dimension of visual design. In the textual narrative of AR interaction, the boundary between the narrator and the narrated is broken, resulting in a com- munication mode different from the traditional linear narrative, which has excellent potential for development. Therefore, this paper attempts to sort out design ideas that integrate textual, visual narrative, and AR interaction to provide a richer and more vivid narrative experience. With the continuous development of the design field and the evolution of interactive applications, more diverse application strategies must be researched and explored to create typography works that establish deeper narrative connections with users.

ACKNOWLEDGMENT

This dissertation is the stage result of the 2021 National Social Science Foundation major project on art, “Research on Chinese Character Inher- itance and Innovative Design” (approval number: 21ZD26) sub-project: “Cross-cultural Communication of Chinese Character Innovative Design”.

Visual Narrative Design of Text in Augmented Reality Interactive Experience 101

REFERENCES

Cheng, K. H. (2023). An epistemic curiosity-evoking model for immersive virtual reality narrative reading: User experience and the interaction among epistemic curiosity, transportation, and attitudinal learning. Computers & Education, 201, p.104814. Li, H. (2015). The introduction of narrative thinking in Chinese character typeface

design. Packaging Engineering, 36(8), 30–33. Ma, Q. (2018). Research on narrative strategy of visual communication design (Doctoral dissertation, Nanjing: Nanjing Arts Institute). Ma, Q. (2021). Research on textual narrative design method. Packaging Engineering,

42(18), 274–281. Millard, D. E. (2023). Strange Patterns: Structure and Post-structure in Interactive Digital Narratives. The Authoring Problem: Challenges in Supporting Authoring for Interactive Digital Narratives, pp. 147–169. Murray, J. H. (2018). Research into interactive digital narrative: a kaleidoscopic view.

In Interactive Storytelling: 11th International Conference on Interactive Digital Storytelling, ICIDS 2018, Dublin, Ireland, December 5–8, 2018, Proceedings 11 (pp. 3–17). Springer International Publishing. Roth, C., & Koenitz, H. (2017). Towards creating a body of evidence-based interactive digital narrative design knowledge: approaches and challenges. In Proceedings of the 2nd International Workshop on Multimedia Alternate Realities (pp. 19–24). Zhou, Q. (2020). Semiotic Interpretation of Narrative Design. Journal of Nanjing

Arts Institute (Art and Design) (04), 127–131.

Cover of the proceedings volume Human-Centered Design and User Experience

Explore the complete digital volume

Human-Centered Design and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-90-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →