Ana Faria, Célia Pinto, Emanuel Sousa (2022). 'On mobile augmented reality and user experience: A reflection and future research agenda. In: Usability and User Experience. AHFE Open Access, vol. 39, pp. 146–152. https://doi.org/10.54941/ahfe1001702
The ubiquity of mobile phones and the advancement of their computational capabilities is making Augmented Reality more accessible to the general public. Being actively used in diverse domains (retail, mapping and navigation, education, maintenance and industry, entertainment, social media, etc.) Mobile Augmented Reality (MAR) can bring value to companies and provide enhanced user experience, when compared with traditional interfaces. Consequently, designers have been exploring the potential of this technology to assist people with their daily life activities and create meaningful experiences.When it comes to designing MAR experiences, what concepts should be considered? What are the best practices? What skills need to be mastered? In order to answer these questions a literature review on user studies that measure the User Experience of MAR applications was conducted. User studies were purposefully targeted because a user centered design approach has consistently shown that understanding peoples’ needs and expectations can improve existing products or assist the development of new ones. The aim of this study is to report a set of well rounded recommendations to help future researchers and practitioners to design better MAR solutions, present research gaps, and provide further avenues for research.Taking the objectives of the study into account, various search strings were constructed with key search terms (e.g. “augmented reality” AND “evaluation” AND “interface design” AND “mobile”). Based on the title and abstract, the search was conducted in September 2021 in Scopus, Web of Science and IEEE Xplore databases for peer review papers from journals between 2011 and 2021. Publications in small formats, literature reviews and technology-centric performance evaluations of algorithms were excluded. The initial research retrieved 62 papers — the abstract of each paper was analyzed. After removing duplicates, and applying the inclusion and exclusion criteria, 38 documents were excluded. Hence, 24 papers were eligible for analysis.After being assembled, the articles were read in full and reviewed in terms of their scope, research methods, procedures and outcomes. 29% of the publications dedicate a considerable amount of the document to report issues related to how the applications were built from a technological standpoint, and in some cases (37.5%) the sections that focus on the user studies lack detail. Therefore, it was not possible to draw elaborate conclusions from these papers in terms of the design process and evaluation. In contrast, few papers present highly detailed descriptions regarding methodological procedures. This information resulted in a list to support measuring MAR experiences. The role of emotions in MAR has captured the interest of researchers. Moreover, topics such as audio-augmented reality, affordances, storytelling and design principles are beginning to emerge in the literature. A critical reflection on these themes and the findings from the studies altogether led to setting out guidelines to create better MAR experiences. To conclude, this literature review shows that researchers are making efforts to measure the user experience of MAR applications. However, the design process is still often disregarded. Future work should therefore explore how to design useful and usable MAR interfaces.
The adoption of extended reality technologies, namely, virtual reality and augmented reality has increased due to the coronavirus pandemic, and the market for retail is expected to grow 68.5% till 2027 (Vardomatski, 2021). Alongside retail, augmented reality has been explored in several domains such as mapping and navigation, education, maintenance and industry, entertain- ment, social media, among others. Part of this growth may be explained by the steady increase of mobile augmented reality (MAR) users over the years (Statista, 2021).
However, there is still limited understanding of how to take advantage of the 3D interaction design space to develop useful and meaningful applicati- ons. What design elements should be considered? What are the best practices? To answer these questions a literature review on user studies that measure the User Experience (UX) of MAR applications was conducted. User studies were purposefully targeted because a user centered design approach has consisten- tly shown that understanding peoples’ needs and expectations can improve existing products or assist the development of new ones. The aim of this study is to report a set of key recommendations to help in designing MAR solutions, present research gaps, and suggest areas for future research.
On Mobile Augmented Reality and User Experience 147
LITERATURE REVIEW AND ANALYSIS The search for peer-reviewed journal papers was conducted in September 2021, using Scopus, Web of Science and IEEE Xplore databases, and limited to publications issued between 2011 and 2021. The search was based on queries made of relevant keywords applied to the title and abstract of the publications (see Table 1). The initial search retrieved 62 papers that were screened from the abstract. After removing duplicates, publications in small formats, literature reviews and technology-centric performance evaluations, 38 documents were excluded. Hence, 24 papers were eligible for analysis.
Table 1. Search terms used in the databases.
Table available in the original PDF paper.
No Query 1 “augmented reality” AND evaluation AND “user interface design” AND mobile 2 “augmented reality” AND evaluation AND “user experience” AND mobile 3 “augmented reality” AND recommendations AND “user experience” AND mobile 4 “augmented reality” AND “user experience” AND “mobile” AND “user study”
All eligible papers were read in full and reviewed in terms of their scope, research methods, and outcomes. The analysis addressed three goals: 1) iden- tify the components of UX that have been explored; 2) categorize different approaches to measure the UX of MAR applications, to support future rese- arch, and 3) compile guidelines and best practices on how to design MAR experiences.
RESULTS AND DISCUSSION
Seven of 24 publications (29%) dedicate a considerable effort to report issues related to how the applications were built from a technological standpoint. In some cases (9 of 24, 37.5%), the report on the user studies lack detail. Notwithstanding, some publications highlighted interesting starting points for future research. This section emphasizes the themes that emerged, draw- ing on relevant literature to offer explanations or distinct perspectives of the subject.
Components of User Experience in MAR Literature The analysis of the user studies was partially based on “The Components of UX” suggested by Hartson and Pyla (2018). According to the authors, UX combines four factors: 1) Usability, 2) Usefulness, 3) Emotional impact, and 4) Meaningfulness. Table 2 presents the components of UX that were found in the literature. Some studies comprise more than one.
As anticipated, “Usability” is the most studied UX component. In general, ensuring the usability of the design seems to be a matter of concern. Indeed, usability plays a vital role in MAR and “It has to be right before the other UX design building blocks can fall into place” (Hillmann, 2021, p. 44), otherwise
Table 2. Different components of UX.
Table available in the original PDF paper.
UX Component N° of studies Percentage of total (%) Usability 18 75 Usefulness 7 29 Emotional impact 6 25 Total 24 100
the other UX components might not even be considered (Hartson & Pyla, 2018, p. 10). The studies included in the component “Usefulness” involve several areas such as education, industrial application, tourism, etc. In most cases the goal was to explore new MAR application concepts and evaluate their utility, which means that researchers are exploring the potential of the technology. This finding is in line with Dirin and Laine (2018), who listed some opportu- nities provided by MAR applications, namely for marketing purposes, to aid the promotion of products and services, to create innovative user interfaces, among others.
Close to “Usefulness,” the user studies that comprise the UX component “Emotional impact” approach emotions from different perspectives. Some studies focus mainly on emotions (see Dirin and Laine (2018), Redzuan et al. (2019), Zhang et al. (2019)), whereas others considered emotions, but only as part of the broader evaluation (see Dhir and Al-Kahtani (2013), Kou- routhanassis et al. (2015), Magnenat et al. (2015)). This finding is in line with Hillmann (2021), who considers that persuasion, emotion, and trust are equally significant as usability.
It was noticed that 6 of 24 studies explored both “Usability” and “Use- fulness” components. This is probably unsurprising considering that percei- ved usefulness and perceived ease-of-use (closely related with usability) are two fundamental predictors of technological acceptance (Davis, 1989). Furthermore, two studies contemplate three UX components: “Usability,” “Usefulness,” and “Emotional impact.”
Lastly, no studies explored the UX component “Meaningfulness.” This component refers to “how a product or artifact becomes meaningful in the life of a user” (Hartson & Pyla, 2018, p. 16). It is perhaps the most dif- ficult dimension to explore, since it requires a longitudinal analysis of the interaction with the device/application. Nonetheless, there are authors sug- gesting that future research should focus on longitudinal studies (see Cen et al. (2020), Dirin and Laine (2018) and Mishra et al. (2021)). Addressing this component might not make sense for all user studies, but it is recom- mended for those that aim to observe users’ over a period of time to identify changes in behavior and collect insightful data.
Measuring the User Experience Both quantitative and qualitative approaches were employed in the repor- ted studies. As can be observed in Table 3, the most prevalent method is the questionnaire. The widespread adoption of standardized questionnaires was
On Mobile Augmented Reality and User Experience 149
Table 3. Main Methodologies used.
Table available in the original PDF paper.
Main methodology used N° of studies Percentage of total (%) Customized questionnaire 13 54 Standardized questionnaire 9 38 Interview 6 25 Usability metrics 4 17 Think aloud 3 13 Video recordings 2 8 Other 1 4 Total 24 100
expected since they are easy to apply data collection tools. Also as Sauro and Lewis explain (2016, p. 232) “questionnaires are of potential value to usa- bility practitioners due to psychometric qualification indicating significant reliability, validity, and sensitivity.” Despite being advantageous, this method does not always allow to identify the underlying causes of the results. Indeed, Davidaviˇcien˙e et al. (2019) mention the latter as a shortcoming of the user experience questionnaire (UEQ).
Whilst some studies adopted one method, others opted for two or more. For instance, Dhir and Al-Kahtani (2013) demonstrate how to use a tri- angulation strategy combining quantitative and qualitative data obtained from three distinct methods: SUXES, Emocard, and AttrakDiff. The auth- ors argue that these methods complement each other and offer rich insights. Certainly, using multiple sources of data translates into more reliable results (Whitenton, 2021).
Guidelines and Best Practices The recommendations derived from the review were grouped into five topics, based on important UX dimensions found throughout the studies (see also Hillmann (2021)): content, interaction, environment and spatial components, sensory input, and emotions.
With regard to “content,” the authors advise taking advantage of MAR technology, i.e. deliver content that is relevant for tasks and provide it based on collected contextual information to mitigate cognitive overload (Kourouthanassis et al., 2015). On this subject, Kim et al. (2017) add that displaying videos and images before text may also aid to reduce cognitive overload. Furthermore, Kourouthanassis et al. (2015) found that MAR tech- nology raises privacy concerns, and to avert negative emotions they advocate for transparency. This issue also emerged as a factor that might influence the acceptance of MAR applications (tom Dieck & Jung, 2018).
The topic “interaction” involves the recommendations that focus on how to communicate to users how they can interact with content. Overall, the authors argue that MAR apps must provide good usability. In particular, it is suggested that users must be informed about the current state of the system, and familiar icons or interaction metaphors should be employed (Kourouthanassis et al., 2015). Moreover, Gjøsæter (2014) offers guidance
on different types of affordances, and suggests combining them with design guidelines to enhance user experience.
Concerning “environment and spatial components,” the findings derive from the user studies that examined issues about the context in which peo- ple interact with MAR applications. Specifically, Dirin and Laine (2018) draw attention to public spaces, arguing that designers should take users’ safety into account. Then, Sekhavat and Parsons (2018) and Brata and Liang (2020) analyzed location-based AR. The findings suggest that location-based AR may generate more favorable results when compared to marker-based AR (Sekhavat & Parsons, 2018), and digital maps are likely to prevail over location-based AR in pedestrian navigation because of familiarity (Brata & Liang, 2020).
In terms of “sensory input,” two studies are worthy of note. The first one directed attention to auditory displays. By comparing 4 types, the auth- ors suggest that spatial audio alongside earcons may increase the level of immersion (Vazquez-Alvarez et al., 2012). The second study investigated multisensory AR, and concluded that it has the potential to enhance expe- riences and, thus, should be further explored (Marto et al., 2021). These findings might inspire and encourage new paths of research beyond visual clues.
The topic “Emotions” includes the studies that provide guidance on how to establish emotional connections through MAR apps, and how to incre- ase user engagement. The authors highlighted the potential of narratives to encourage content exploration (Kim et al., 2017), and to create useful and involving experiences in the realm of entertainment and education (Nam, 2015). To develop emotional attachment some of the suggested best practi- ces include: personalized experiences, realism, and spatial correspondence (Dirin & Laine, 2018).
Additionally, it was noticed that few studies documented the design pro- cess. As a case in point, Nam (2015) explained the rationale behind the prototypes stating the design factors and the types of narrative layers used. Nonetheless, none of the publications mentions considering the users and their needs in the design process. As a matter of fact, Dirin and Laine (2018) suggest user centered design methods to identify users’ needs, and Kim et al. (2017) consider that users’ personal context and preferences should be addressed.
CONCLUSIONS AND RESEARCH IMPLICATIONS To design better MAR applications, practitioners and designers should ensure good usability, and simultaneously take advantage of AR technology to create engaging experiences. In the literature topics such as affordances, audio- augmented reality, multisensory AR and narratives are beginning to arise. Also, the role of emotions in MAR has captured the interest of researchers. These topics seem to be unexplored in the realm of MAR applications and future studies are strongly encouraged.
To conclude, this literature review shows that researchers and developers are making efforts to measure the user experience of MAR applications.
On Mobile Augmented Reality and User Experience 151
However, the design process is still often disregarded. Future work should therefore describe how MAR apps are designed. Also, in the author’s view, regardless of the interaction technology in use, user centered design frame- works should be employed in the design process to create useful, usable and satisfying experiences.
ACKNOWLEDGMENT
This work was executed under the project Cognitive CMMS - Cognitive Computerized Maintenance Management System, NUP: POCI-01-0247- FEDER-033574, co-funded by the Incentive System for Research and Tech- nological Development, from the Thematic Operational Program Competiti- veness of the national framework program - Portugal2020.
REFERENCES
Brata, K. C., & Liang, D. (2020). Comparative study of user experience on mobile pedestrian navigation between digital map interface and location-based augmen- ted reality. International Journal of Electrical and Computer Engineering, 10(2), 2037–2044. https://doi.org/10.11591/ijece.v10i2.pp2037-2044 Cen, L., Ruta, D., Al Qassem, L. M. M. S., & Ng, J. (2020). Augmented Immersive
Reality (AIR) for Improved Learning Performance: A Quantitative Evaluation. IEEE Transactions on Learning Technologies, 13(2), 283–296. https://doi.org/10. 1109/TLT.2019.2937525 Davidaviˇcien˙e, V., Raudeli¯unien˙e, J., & Viršilait˙e, R. (2019). User experience evaluation and creativity stimulation with augmented reality mobile applications. Creativity Studies, 12(1), 34–48. https://doi.org/10.3846/cs.2019.3576 Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance
of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10. 2307/249008 Dhir, A., & Al-Kahtani, M. (2013). A case study on user experience (UX) evaluation of mobile augmented reality prototypes. Journal of Universal Computer Science, 19(8), 1175–1196. Dirin, A., & Laine, T. H. (2018). User experience in mobile augmented reality:
Emotions, challenges, opportunities and best practices. Computers, 7(2). https: //doi.org/10.3390/computers7020033 Gjøsæter, T. (2014). Affordances in mobile augmented reality applications. International Journal of Interactive Mobile Technologies, 8(4), 45–55. https://doi.org/10. 3991/ijim.v8i4.4051 Hartson, R., & Pyla, P. (2018). The UX book: Agile UX design for a quality user
experience (2nd ed.). Morgan Kaufmann. Hillmann, C. (2021). UX for XR. Apress. https://doi.org/10.1007/978-1-4842- 7020-2 Kim, H., Matuszka, T., Kim, J. I., Kim, J., & Woo, W. (2017). Ontology-based
mobile augmented reality in cultural heritage sites: information modeling and user study. Multimedia Tools and Applications, 76(24), 26001–26029. https: //doi.org/10.1007/s11042-017-4868-6 Kourouthanassis, P. E., Boletsis, C., & Lekakos, G. (2015). Demystifying the design of mobile augmented reality applications. Multimedia Tools and Applications, 74(3), 1045–1066. https://doi.org/10.1007/s11042-013-1710-7
Magnenat, S., Ngo, D. T., Zünd, F., Ryffel, M., Noris, G., Rothlin, G., Marra, A., Nitti, M., Fua, P., Gross, M., & Sumner, R. W. (2015). Live Texturing of Augmented Reality Characters from Colored Drawings. IEEE Transactions on Visualization and Computer Graphics, 21(11), 1201–1210. https://doi.org/10. 1109/TVCG.2015.2459871 Marto, A., Melo, M., Goncalves, A., & Bessa, M. (2021). Development and Evalua-
tion of an Outdoor Multisensory AR System for Cultural Heritage. IEEE Access, 9, 16419–16434. https://doi.org/10.1109/ACCESS.2021.3050974 Mishra, A., Shukla, A., Rana, N. P., & Dwivedi, Y. K. (2021). From “touch” to a “multisensory” experience: The impact of technology interface and product type on consumer responses. Psychology and Marketing, 38(3), 385–396. https://doi. org/10.1002/mar.21436 Nam, Y. (2015). Designing interactive narratives for mobile augmented reality.
Cluster Computing, 18(1), 309–320. https://doi.org/10.1007/s10586-014-0354- 3 Redzuan, F., Khairuddin, A. N. A., & Daud, N. A. (2019). Emotional augmented reality-based mobile learning design elements: A kansei engineering appro- ach. Indonesian Journal of Electrical Engineering and Computer Science, 14(1), 413–420. https://doi.org/10.11591/ijeecs.v14.i1.pp413-420 Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics
for user research. Morgan Kaufmann. Sekhavat, Y. A., & Parsons, J. (2018). The effect of tracking technique on the quality of user experience for augmented reality mobile navigation. Multimedia Tools and Applications, 77(10), 11635–11668. https://doi.org/10.1007/s11042-017-4810- y Statista. (2021). Number of mobile augmented reality (AR) active users worldwide
from 2019 to 2024 (in billions). https://www.statista.com/statistics/1098630/glob al-mobile-augmented-reality-ar-users/ tom Dieck, M. C., & Jung, T. (2018). A theoretical model of mobile augmented reality acceptance in urban heritage tourism. Current Issues in Tourism, 21(2), 154–174. https://doi.org/10.1080/13683500.2015.1070801 Vardomatski, S. (2021). Augmented And Virtual Reality After Covid-19. Forbes. https://www.forbes.com/sites/forbestechcouncil/2021/09/14/augmented- and-virtual-reality-after-covid-19/?sh=a16f2ad2d97c Vazquez-Alvarez, Y., Oakley, I., & Brewster, S. A. (2012). Auditory display design
for exploration in mobile audio-augmented reality. Personal and Ubiquitous Computing, 16(8), 987–999. https://doi.org/10.1007/s00779-011-0459-0 Whitenton, K. (2021). Triangulation: Get Better Research Results by Using Multiple UX Methods. Nielsen Norman Group. https://www.nngroup.com/articles/triangu lation-better-research-results-using-multiple-ux-methods/ Zhang, J., Kamioka, E., & Tan, P. X. (2019). Emotions detection of user experience
(Ux) for mobile augmented reality (mar) applications. International Journal of Advanced Trends in Computer Science and Engineering, 8(1.4 S1), 63–67. https: //doi.org/10.30534/ijatcse/2019/1081.42019
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Usability and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-15-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional