Bernardo Marques, Samuel Silva, Betriz Sousa Santos (2024). Measuring Remote Collaboration Supported by Extended Reality: A Multidimensional Evaluation Approach. In: Accessibility, Assistive Technology and Digital Environments. AHFE Open Access, vol. 121, pp. 19–26. https://doi.org/10.54941/ahfe1004610
Remote collaboration supported by eXtended Reality (XR) is a paradigm that transforms how geographically dispersed teams collaborate on shared tasks by leveraging immersive technologies that bridge physical distances. This approach goes beyond traditional communication tools, by creating shared, three-dimensional spaces that enhance the depth and effectiveness of collaboration. Regardless, it also introduces new challenges, like evaluating how collaboration occurs during such contexts, which literature shows is not straightforward. This work delves into evaluation in the realm of remote XR collaboration by proposing that a multidimensional approach is used, considering the level of information understanding, communication effectiveness, or social presence, among others. By analyzing how evaluation could be performed, we intend to offer insights to steer research and development in this dynamic field. These assessments can contribute not only to refining XR-system development but also deepen comprehension of how teams collaborate over time.
Collaboration is a fundamental aspect of human interaction. In particular, remote collaboration refers to the act of physically distributed team members working together on shared tasks. This happens when on-site individuals can- not handle a given situation, forcing them to request assistance from remote experts. To accomplish this, various technologies and tools can be used to facilitate communication, information sharing, and cooperation. The abil- ity to collaborate remotely is especially crucial in today’s globalized world, as it allows efficient cross-border communication, knowledge sharing, and teamwork. These activities have particular relevance in a wide range of sce- narios, including industrial, medical, and educational domains, among others (de Belen et al., 2019; Ens et al., 2019; Sereno et al., 2020; Wang et al., 2021).
In recent years, remote eXtended Reality (XR) has witnessed unprece- dented growth, with the potential to create a common group and rev- olutionize the way teams work together across geographical boundaries. This umbrella term encompasses Virtual, Augmented and Mixed Reality
(VR/AR/MR) (Speicher et al., 2019; Ratcliffe et al., 2021; Rauschnabel et al., 2022). All of these can be applied to collaborative scenarios, by creating immersive and interactive environments that bring remotely sepa- rated individuals together. For example, after being captured and shared, the on-site environment can be visualized by remote experts through VR. These individuals can enhance their view with virtual content (e.g., draw- ings, pointers, gaze, gestures, arrows, etc.), explaining what to do and where to act. Later, on-site individuals using AR can visualize superimposed instructions on top of the real-world context, improving situation under- standing, awareness and alertness. This paradigm shifts from traditional instant messaging, document editing, or video conferencing alternatives introduce unique challenges and opportunities for the research community (Sereno et al., 2020; Marques et al., 2021; Marques, Silva, Dias, et al., 2022; Marques, Ferreira, et al., 2023).
To elaborate, thus far, remote XR is positioned between the replication and empiricism stages of the BRETAM model, evidencing the need for addi- tional work toward achieving a more advanced stage of maturity (Marques, Teixeira, et al., 2022). An essential step to achieve this is considering evalua- tion of the collaborative process, where limited research has been conducted, given the number of variables that affect how these multi-user experiences occur. An improved understanding of how collaboration unveils can provide the tools to perform a more contextualized interpretation and define more robust conclusions, not only regarding the performance of the technology but also how XR contributes to an effective/efficient work effort (Dünser et al., 2008; Dey et al., 2018; Merino et al., 2020; Marques, Silva, Teixeira, et al., 2022; Marques, Silva, et al., 2023). This work explores the significance of considering distinct dimensions of collaboration when conducting user evaluations in the context of remote- XR collaboration. To harness the full potential of XR, it is imperative to better comprehend the collaborative process, which encompasses vari- ous dimensions of interest (e.g., level of information understanding, ease of communication, level of social presence, among others). By examining the multifaceted aspects of collaboration in this setting, we aim to provide insights that can guide research and development in this rapidly evolving field. These evaluations not only inform the development of XR-systems but also contribute to a deeper understanding of how remote collaboration can be enhanced. By addressing the distinct dimensions of collaboration, the community can also create more inclusive, and effective XR-remote collabo- ration experiences that benefit a broad spectrum of users and industries, thus shaping the future of remote collaboration.
A MULTIDIMENSIONAL APPROACH FOR EVALUATING XR-REMOTE COLLABORATION The literature shows that research on remote collaboration supported by XR (see Figure 1) must evolve from solving technical issues, towards more mean- ingful studies on collaboration itself (de Belen et al., 2019; Merino et al.,
Measuring Remote Collaboration Supported by eXtended Reality 21
2020; Ratcliffe et al., 2021; Marques, Silva, et al., 2023). Regardless, eval- uating these scenarios is a difficult endeavor since many aspects may affect how teams collaborate. Similarly, there is a lack of theories to guide how the collaborative process may be characterized, and beyond that, present evalu- ation frameworks are also not specifically designed to evaluate the dynamics of XR-remote collaboration, falling short to provide the necessary data to inform a more thorough analysis.
Figure 1: Illustration of a scenario of remote collaboration supported by XR, having an on-site individual located in an industrial scenario being assisted by a remote team- member.
As described in previous work, studies evaluating scenarios of XR-remote collaboration often rely on single-user methodologies, primarily centered on comparing technological aspects or interaction mechanisms, as well as quan- tify effectiveness in completing tasks, often lacking diversity, difficulty, and ecological validity. However, such approaches may not be the most suit- able for multifaceted solutions intended to support collaboration among distributed teams. A significant portion of existing studies concentrates solely on the performance of individual collaborators (on-site or remote). Conse- quently, the evaluation tends to overlook important aspects of collaboration (Marques, Silva, Dias, et al., 2022; Marques, Silva, Teixeira et al., 2022; Marques, Teixeira, et al., 2022; Marques, Silva, et al., 2023).
Besides, most studies in this field take place in laboratories, collecting objective and subjective data at the tasks’ conclusion through standard prac- tices such as scale-based questionnaires (e.g., System Usability Scale (SUS), NASA Task Load Index (TLX), among others) or direct observation. Fur- thermore, the lack of guidelines hampers researchers from determining which dimensions of collaboration should be collected and how to effectively gather them.
Next, we propose a list of important dimensions of collaboration, that should be considered. These were selected carefully after survey existing lit- erature to create a list of important topics facing the lack of methodologies and frameworks. This list was also displayed to experts in XR/VR/AR/MR, Human-Computer Interaction (HCI), Human-Centered Design (HCD), and Computer-Supported Cooperative Work (CSCW), who had an important role
in selecting, analyzing and filtering said topics of collaboration by voting about the ones they considered more relevant. To elaborate, we took inspi- ration from works by (Kim et al., 2014, 2018, 2019, 2020; Aschenbrenner et al., 2018; Kim, Billinghurst and Lee, 2018; Huang et al., 2019; Pium- somboon et al., 2019; Teo et al., 2019; Kim, Billinghurst and Kim, 2020; Marques, Silva, Teixeira, et al., 2022; Marques, Ferreira, et al., 2023).
With this proposal, we aim to provide a structured list of relevant arti- facts, that can be measured all at once. These characteristics are important since collaboration may occur at many levels and depends on several factors that may directly impact the collaborative outcomes. Nevertheless, other sub- sets of alternative dimensions can be considered according to the evaluation scope, as long as they are not used in an isolated manner. By only using one of these dimensions, findings reported may be misleading or of limited value in these scenarios.
To elaborate, the proposed list includes the following dimensions of collaboration:
level of attentional allocation - the degree of cognitive focus or mental concentration allocated to a specific task;
effectiveness in perceived information understanding - the extent to which information is comprehended and interpreted by an individual;
effectiveness in express ideas properly - the proficiency in conveying thoughts, concepts, or messages accurately and coherently in a manner that aligns with the communicator’s intent;
level of satisfaction - The extent to which an individual experiences fulfillment in the process of engaging with a particular activity or task;
level of frustration - The degree of dissatisfaction experienced by an individual in response to the collaborative process;
level of visual complexity - The degree of intricacy present in the visual elements of the XR tool being used during the collaborative process;
effectiveness to communicate - The ability to convey information clearly, accurately, and comprehensively to other collaborators;
level of social presence - The perceived degree of connection, interaction, or involvement with others in a XR- mediated environment;
effectiveness in completing the intended tasks - The degree to which a team successfully accomplishs the specified goals;
level of mental effort - The amount of cognitive workload experienced by an individual while engaging in a particular task or activity;
level of physical demand - The extent of physical effort required to perform a given task or activity;
level of temporal effort - The amount of time-related investment required to complete a task or achieve a particular objective.
These dimensions may be collected at the end of the collaborative process, using a Likert-type scale (1- Low; 7 - High). Having this list of dimensions may also help researchers create data visualizations to maximize its usefulness during the data analysis process. For example, Figure 2 illustrates a possible visualization, i.e., a radar char representation, in which all dimensions may be
Measuring Remote Collaboration Supported by eXtended Reality 23
analyzed at the same time, for all team members involved in the collaborative process.
Figure 2: Illustration of a possible visualization for the proposed dimensions that may be considered during an evaluation of a scenario of remote collaboration supported by XR. The radar chart enables to have a better understanding of all dimensions at once.
Enhancing our comprehension of the collaborative process through mul- tiple dimensions equips researchers with the tools for a contextualized inter- pretation of results, leading to more robust conclusions and efficient work efforts. Therefore, conducting more comprehensive collaborative studies is crucial, enabling the retrieval of the necessary amount of data for a thor- ough analysis that sheds light on various factors influencing XR-supported collaboration. The proposed dimensions and visualization may also be used over long periods of time, allowing to compare how the same team behaves between collaborative sessions. This information may be extremely relevant if outliers start to emerge, allowing to understand what exactly is affecting the team, e.g., the XR tool, one element in particular, the task being handled, etc. In essence, a profound understanding of these aspects is vital to ensure genuine characterization. By achieving this, the community may be able to better assess a wide range of information, including individual and team per- sonalities, performances, and behaviors, thereby facilitating a more insightful analysis, and establishing conclusions.
Final Remarks and Next Steps The significance of remote collaboration supported by XR lies in its ability to break down geographical barriers, fostering efficient cross-border communi- cation, knowledge sharing, and teamwork. For instance, using VR, remote experts can visualize and engage with on-site environments in real-time,
providing guidance and insights. On the other hand, AR allows on-site indi- viduals to receive superimposed instructions or information directly within their real-world context.
This collaborative model finds applications across various domains, including industrial, medical, and educational sectors. The immersive and interactive nature of XR not only enhances situational awareness but also facilitates a more natural and engaging form of communication among team members. As XR technologies continue to advance, the potential for creating realistic, shared experiences across distances expands, leading to a paradigm shift in how teams collaborate remotely.
In the rapidly evolving landscape of remote XR collaboration, the impor- tance of developing and implementing new evaluation models cannot be overstated. Traditional methods often fall short in capturing the nuanced dynamics of collaborative processes facilitated by immersive technologies. The unique blend of existing realities introduces novel domains such as spa- tial awareness, interactive visualization, or social presence, which demand innovative evaluation approaches. Relying on established models may hinder our ability to fully grasp the impact and effectiveness of XR in remote col- laboration scenarios. As we navigate the uncharted territories of remote XR collaboration, given its level of maturity, crafting new evaluation approaches becomes paramount. By embracing these new evaluation paradigms, we may be able to truly unlock the transformative potential of remote XR collabo- ration and foster more meaningful, efficient, and user-centric collaborative interactions across diverse industries.
Moving forward, we intend to prioritize conducting extended and for- mal user studies, especially within industrial scenarios such as maintenance scenarios. These studies could provide valuable insights into the real-world applicability of the proposed approach, enabling the collection of a compre- hensive and robust dataset for in-depth analysis and continual improvement. Industry settings present complex and dynamic environments that could further validate the effectiveness of the approach in practical, high-stakes situations.
Another avenue for future exploration involves extending the proposed approach to multi-user scenarios, involving multiple team members in the collaborative process. This expansion would assess the scalability of the approach and its ability to facilitate seamless interaction among various team members in a shared XR environment. Evaluating how the system performs when supporting collaboration among multiple users is crucial for ensuring its practicality and effectiveness in diverse team-based settings.
Furthermore, it is important to dedicate efforts to developing a visualiza- tion tool tailored for remote XR collaboration. This would empower team leaders and decision-makers with the means to make more informed deci- sions. It could offer features such as real-time data visualization, collaborative data annotation, as well as interactive analysis, having an overview of how collaboration unfollowed.
Measuring Remote Collaboration Supported by eXtended Reality 25
ACKNOWLEDGMENT
We thank everyone involved for your time and expertise. This research was supported by IEETA - Institute of Electronics and Informatics Engineering of Aveiro, funded by National Funds through FCT, in the context of the project [UIDB/00127/2020].
REFERENCES
Aschenbrenner, D., Rojkov, M., Leutert, F., Verlinden, J., Lukosch, S., Latoschik, M. E. and Schilling, K. (2018) ‘Comparing different augmented reality support applications for cooperative repair of an industrial robot’, in 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR- Adjunct). IEEE, pp. 69–74. de Belen, R. A. J., Nguyen, H., Filonik, D., Del Favero, D. and Bednarz, T. (2019) ‘A systematic review of the current state of collaborative mixed reality technologies: 2013–2018’, AIMS Electronics and Electrical Engineering, 3(2), pp. 181–223. Dey, A., Billinghurst, M., Lindeman, R. W. and Swan, J. E. (2018) ‘A systematic
review of 10 years of augmented reality usability studies: 2005 to 2014’, Frontiers in Robotics and AI, 5, p. 37. Dünser, A., Grasset, R. and Billinghurst, M. (2008) ‘A survey of evaluation techniques used in augmented reality studies’, in Human Interface Technology Laboratory. Ens, B., Lanir, J., Tang, A., Bateman, S., Lee, G., Piumsomboon, T. and
Billinghurst, M. (2019) ‘Revisiting collaboration through mixed reality: The evo- lution of groupware’, International Journal of Human-Computer Studies, 131, pp. 81–98. Huang, W., Kim, S., Billinghurst, M. and Alem, L. (2019) ‘Sharing hand gesture and sketch cues in remote collaboration’, Journal of Visual Communication and Image Representation, 58, pp. 428–438. Kim, S., Billinghurst, M. and Kim, K. (2020) ‘Multimodal interfaces and commu-
nication cues for remote collaboration’, Journal on Multimodal User Interfaces. Springer, pp. 313–319. Kim, S., Billinghurst, M., Lee, C. and Lee, G. (2018) ‘Using freeze frame and visual notifications in an annotation drawing interface for remote collaboration.’, KSII Transactions on Internet & Information Systems, 12(12). Kim, S., Billinghurst, M. and Lee, G. (2018) ‘The effect of collaboration styles
and view independence on video-mediated remote collaboration’, Computer Supported Cooperative Work (CSCW), 27, pp. 569–607. Kim, S., Lee, G., Billinghurst, M. and Huang, W. (2020) ‘The combination of visual communication cues in mixed reality remote collaboration’, Journal on Multimodal User Interfaces, 14, pp. 321–335. Kim, S., Lee, G., Huang, W., Kim, H., Woo, W. and Billinghurst, M. (2019) ‘Evaluat-
ing the combination of visual communication cues for HMD-based mixed reality remote collaboration’, in Proceedings of the 2019 CHI conference on human factors in computing systems, pp. 1–13. Kim, S., Lee, G., Sakata, N. and Billinghurst, M. (2014) ‘Improving co-presence with augmented visual communication cues for sharing experience through video conference’, in 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, pp. 83–92. Marques, B., Ferreira, C., Silva, S., Dias, P. and Santos, B. S. (2023) ‘Is social presence
(alone) a general predictor for good remote collaboration? comparing video and augmented reality guidance in maintenance procedures’, Virtual Reality, pp. 1–14.
Marques, B., Silva, S., Alves, J., Araujo, T., Dias, P. and Santos, B. S. (2021) ‘A conceptual model and taxonomy for collaborative augmented reality’, IEEE Transactions on Visualization and Computer Graphics, 28(12), pp. 5113–5133. Marques, B., Silva, S., Dias, P. and Santos, B. S. (2022) ‘Evaluating augmented reality
based remote collaboration: A contextualized approach’, in Human-automation interaction: mobile computing. Springer, pp. 267–280. Marques, B., Silva, S., Dias, P. and Santos, B. S. (2023) ‘How to Evaluate If Collaborative Augmented Reality Speaks to Its Users’, IEEE Computer Graphics and Applications, 43(5), pp. 107–113. Marques, B., Silva, S., Teixeira, A., Dias, P. and Santos, B. S. (2022) ‘A vision for
contextualized evaluation of remote collaboration supported by AR’, Computers & Graphics, 102, pp. 413–425. Marques, B., Teixeira, A., Silva, S., Alves, J., Dias, P. and Santos, B. S. (2022) ‘A critical analysis on remote collaboration mediated by Augmented Reality: Making a case for improved characterization and evaluation of the collaborative process’, Computers & Graphics, 102, pp. 619–633. Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D. and Weiskopf, D.
(2020) ‘Evaluating mixed and augmented reality: A systematic literature review (2009-2019)’, in IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, pp. 438–451. Piumsomboon, T., Dey, A., Ens, B., Lee, G. and Billinghurst, M. (2019) ‘The effects of sharing awareness cues in collaborative mixed reality’, Frontiers in Robotics and AI, 6 Ratcliffe, J., Soave, F., Bryan-Kinns, N., Tokarchuk, L. and Farkhatdinov, I. (2021)
‘Extended reality (XR) remote research: a survey of drawbacks and opportunities’, in Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–13. Rauschnabel, P. A., Felix, R., Hinsch, C., Shahab, H. and Alt, F. (2022) ‘What is XR?
Towards a framework for augmented and virtual reality’, Computers in human behavior, 133, p. 107289. Sereno, M., Wang, X., Besançon, L., Mcguffin, M. J. and Isenberg, T. (2020) ‘Collaborative work in augmented reality: A survey’, IEEE Transactions on Visualization and Computer Graphics, 28(6), pp. 2530–2549. Speicher, M., Hall, B. D. and Nebeling, M. (2019) ‘What is mixed reality?’, in
Proceedings of the CHI conference on human factors in computing systems, pp. 1–15. Teo, T., Lee, G. A., Billinghurst, M. and Adcock, M. (2019) ‘Investigating the use of different visual cues to improve social presence within a 360 mixed reality remote collaboration’, in Proceedings of the International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp. 1–9. Wang, P., Bai, X., Billinghurst, M., Zhang, S., Zhang, X., Wang, S., He, W., Yan,
Y. and Ji, H. (2021) ‘AR/MR remote collaboration on physical tasks: a review’, Robotics and Computer-Integrated Manufacturing, 72, p. 102071.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Accessibility, Assistive Technology and Digital Environments
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-97-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional