Jiahe Zhang, Yuan Xiang (2023). User Interface Consistency Experience in Composite Cross-device Interaction. In: Ergonomics In Design. AHFE Open Access, vol. 77, pp. 826–836. https://doi.org/10.54941/ahfe1003443
The complexity of cross-device interaction brings challenges to the design of user experience. Related research has investigated the elements of cross-device scenarios and the design principles of different scenarios. To explore the consistency of user experience for a single user in multiple temporal cross-device scenarios, this study first defined the concept of composite cross-device scenarios that include cross-device interactions with asynchronous and synchronous operations. Then, based on the principle of cross-device consistency, three interface distribution strategies under three composite cross-device scenarios, mirroring, split view, and split interaction, were constructed. Finally, through experiments, the user experience of the above three interface distribution strategies is tested under the two tasks of "operation-oriented" and "observation-oriented". Experimental results show that there are differences in user preferences for different interface distribution strategies in two different tasks. In each group, the mirrored pattern received the highest score. Through semi-structured interviews, it is found that users pay the lowest learning cost for performing cross-device operations in mirror mode. Separation of view and separation of interaction score relatively high in operation-oriented and observation-oriented task scenarios, respectively. This difference indicates that the cross-device interface distribution method needs to meet the interaction requirements of the task.
With personal smart devices’ popularity, individual users increasingly use multiple devices to complete daily computing tasks (Sørensen et al., 2014). While the use of multiple devices expands the usage scenarios of smart devi- ces, due to the increase in the number of devices and the complexity of the relationship, cross-device user experience design and related research face many challenges. To cope with the difficulty of designing interactions across devices and the complexity of adapting to UI standards for different platforms (Dong, Churchill and Nichols, 2016), related research has contributed in two aspects. Some researches attempt to sort out various interaction elements in cross-device scenarios and help researchers and designers better under- stand cross-device interactions by establishing a theoretical framework. In this regard, related researches mainly explore the cross-device user experience when tasks are migrated from one device to another, or when multiple devi- ces collaborate. However, in actual use cases, what constitutes a cross-device
User Interface Consistency Experience in Composite Cross-Device Interaction 827
context may be a composite cross-device interaction scenario formed by a combination of devices with multiple relationships. In addition, a large num- ber of studies have explored the design principles of cross-device interaction for specific cross-device scenarios. Among them, consistency across devices is considered an important design principle (Nichols, Richter and Gajos, 2006). Maintaining consistency across devices reduces the user’s learning cost in per- forming tasks across devices, but since different devices in a heterogeneous cross-device system have different capabilities and UI standards, complete consistency is unrealistic and May hinder a good user experience. There- fore, exploring how consistency affects cross-device user experience helps to use the design principle of consistency more reasonably (Pyla, Tungare and Pérez-Qu, 2006).
In this study, we will first summarize cross-device scene elements and cross-device consistency related research. Then based on related research, the concept of a “composite cross-device scenario” will be proposed, and the interface distribution strategy in this scenario will be constructed based on the principle of consistency. To verify the user experience of different inter- face distribution strategies in the composite cross-device scenario, this study implemented experiments including “operation-oriented” and “observation- oriented”. In the last two chapters, by analyzing the experimental results, the impact of the consistent interface distribution strategy on the user experience of composite cross-device scenarios is obtained from both quantitative and qualitative perspectives.
RELATED CROSS-DEVICE USER EXPERIENCE RESEARCH Compared with the case of using a single device, the interaction of cross- device systems has a higher complexity (Dong, Churchill and Nichols, 2016). Therefore, to explore the cross-device user experience, it is necessary to clarify the relevant elements that affect the cross-device user experience. This study first refers to cross-device usability and research related to cross-device user experience.
Charles Denis and Laurent Karsenty defined the concept of inter-usability (Denis and Karsenty, 2005), which refers to the ease with which certain functions can be transferred to other devices. Inter-usability has two dimen- sions: knowledge continuity and task continuity. Three design principles can be applied to each dimension: Inter-device Consistency, Transparency and Dialogue adaptability. Minna Wäljas and Katarina Segerståhl provided an initial framework of user experience for cross-platform services through field research, which consists of three important components: Appropriate system composition, Fluency of content and task migration, and Service consistency (Wljas et al., 2010). In (Shin, 2016), the key elements of cross-platform UX include access, mobility and coherence. The idea of inter-usability for user- centered system design is proposed. The case for generating visual separation in mobile multi-display is described in (Cauchard et al., 2011). The results of the study show that the spatial distribution of devices in a multi-display environment affects the user experience.
828 Zhang and Yuan
Research related to cross-device user experience and cross-device usabi- lity inspired this research in two aspects: on the one hand, the cross-device composition and service provision, and on the other hand, cross-device consistency.
Cross-Device Composition and Service Provision In the related research on cross-device user experience, the device compo- sition of cross-device system is an important factor. Related studies discuss device composition across devices from different aspects.
Charles Denis and Laurent Karsenty defined three levels of device redun- dancy according to the data accessibility, including Redundant devices, Complementary devices and Exclusive devices (Denis and Karsenty, 2005). Minna Wäljas and Katarina Segerståhl defined Multichanneled and Cros- smedial service delivery methods according to the way services are provided across devices (Wljas et al., 2010). Through the study of relevant works of literature, Frederik Brudy et al. proposed a cross-device feature framework, summarizing Temporal, device Configuration, Relationship, Dynamics, Scale and Space (Brudy et al., 2019).
Related studies have discussed the device composition and service pro- vision methods in a single cross-device temporal scenario. However, in the actual cross-device using scenario, there are more complicated situations, such as: using mobile phones to watch videos during commuting, then casting video to TV at home, and using a mobile phone for control. Therefore, this study proposes the concept of composite cross-device scenarios to describe this cross-device usage scenario that includes multiple service relationships.
Cross-Device Consistency Cross-device consistency is considered to be an important design principle (Nichols, Richter and Gajos, 2006), and related research will be discussed from two aspects: the dimensions included in cross-device consistency and how cross-device consistency affects user experience.
Relevant researches generally divide cross-device consistency according to the level of human-computer interaction. In (Denis and Karsenty, 2005), Charles Denis and Laurent Karsenty defined the dimensions of cross-device consistency: Perceptual consistency, Lexical consistency, Syntactical consiste- ncy, and Semantic consistency. In the research of Minna Wäljas, Katarina Segerståhl et al. in (Wljas et al., 2010), consistency is defined as Perceptual consistency, semantic consistency and interaction consistency.
Cross-device consistency has an important impact on user experience. Martina Ziefle et al. investigated the impact of young and old mental models on performance when interacting with mobile devices, and found that when the navigation of the device interface conforms to the user’s mental model It can reduce the cost of user learning (Ziefle, Arning and Bay, 2006). Consi- stency between devices helps users migrate their previous experience of using devices to new devices and provides a seamless user experience by reducing user learning costs. In (Oliveira and Rocha, 2007), the following priorities are proposed to apply consistency in a multi-device context: Task Perception,
User Interface Consistency Experience in Composite Cross-Device Interaction 829
Task Execution, and Task Personalization. At the same time, many studies have shown that consistency cannot dominate the cross-device design, and seamless cross-device experience is the key to user experience (Pyla, Tungare and Pérez-Qu, 2006). Sung Woo Kim et al. (Kim, Jo and Ha, 2011) described the concept of different UIs with the same UX, and proposed that in a cross- device scenario, in order to adapt to the specifications of different devices, UI inconsistencies cannot be avoided, but the same user experience should be maintained.
CONSISTENCY-BASED COMPOSITE CROSS-DEVICE INTERFACE DISTRIBUTION STRATEGY Based on the concepts of Temporal and Configuration in the cross-device feature framework proposed by Frederik Brudy et al. (Brudy et al., 2019), the composite cross-device scenario defined in this study refers to a tem- poral cross-device interaction context contains other timing cross-device interactions.
There are many combinations for composite cross-device scenarios. In this study, we mainly discuss the cross-device usage situation where the task is migrated from A device to A+B devices, as shown in Figure 1. For exam- ple, one edits emails during commuting and then uses computers and mobile phones to work together to complete email editing when he arrives at the office.
In the above composite cross-device scenario, there are two processes, one is the process in which the user connects the task on the mobile phone to the mobile phone + computer, and the other is the process in which the user uses the mobile phone + computer to jointly complete the email editing pro- cess. When tasks are migrated from one device to multiple devices, how to distribute the interfaces on multiple devices and maintain a good cross-device interaction experience has become a key issue.
This study constructs a cross-device interface distribution strategy based on the principle of consistency by studying related interface distribution stra- tegies. According to perception consistency, interaction consistency and data consistency, three interface distribution strategies are constructed, as shown in Figure 2.
Figure 1: One of the composite cross-device interaction scenarios.
830 Zhang and Yuan
Figure 2: Cross-device interface distribution strategy based on the principle of consistency.
Mirrored The Mirrored mode will maintain maximum consistency before and after the migration. The devices after migration will share the interface in real-time in the Mirrored mode.
Separated View Arrange different interface elements on a single device to different multi- device, but keep the interaction modality, interaction event triggering method and operation sequence consistent.
Separated Operation Partially migrate views from a single device to devices with larger screens, and migrate interactions to interactable screen devices in a way that fits natively.
STUDY To explore the relationship between consistency and user experience in composite cross-device scenarios, this study designed and implemented two groups of experiments.
Tasks Relevant studies have shown that the configuration of cross-device systems should match cross-device tasks (Wljas et al., 2010). To explore the impact of different consistent distribution strategies on cross-device user experience, according to the information and operating characteristics of devices, this study divides cross-device tasks into The following two tasks designed for “Operation-oriented” and “Observation-oriented”: 1) Picture collection task According to the prompts of the main tester, the participants used a sin- gle device and cross-device respectively to complete the browsing of specific pictures in a group of photos and finally switched to favorites to browse the favorite pictures. 2) Picture observation task The participants used a single device and a cross-device method to observe whether the pictures and text descriptions in the interface matched.
User Interface Consistency Experience in Composite Cross-Device Interaction 831
Methods and Tools The experiment was carried out in the form of an online experiment. Under the guidance of the experimenter, the participants completed the experiment by logging in to the website experiment page. The operation interface of the participants will be recorded.
The experiment uses the System Portability Questionnaire (STQ) to mea- sure the participants’ cross-device experience (Yunchen et al., 2013). Each group of experiments will conduct a semi-structured interview to compreh- ensively collect the participants’ subjective attitudes and reasons for different experimental schemes.
Participants Each of the two groups invited 30 participants, aged between 20 and 30, including 12 males and 18 females in the first group; 16 males and 14 females in the second group. Most of the participants were postgraduates, and some of them were postgraduates in the field of interaction design.
Prototype Both groups of experiments were tested online, and the experimental pro- totype was a web page specially developed for the experiment that supports running on mobile phones and computers. The synchronization between devices relies on the WebSocket protocol to be forwarded by the server, and the delay in synchronization between tested devices is difficult to notice.
To avoid the influence caused by the order in which the equipment was used, the participants were numbered and the order in which the interfaces of different strategies appeared was evenly disrupted.
Procedure Before the start of the experiment, the participants were invited to enter the online meeting and read the experimental instructions, and filled in relevant personal information. After the experiment started, the participants of the two groups of experiments followed the guidance of the main experimenter, first using a single device and then using a combination of cross-device to experience three cross-device interaction schemes with consistent strategies. After the test of each program is completed, the participants will be invited to fill in the questionnaire to record the participants’ subjective evaluation of the cross-device experience. After the completion of each group of experiments, the participants will be asked about the test plan and the reason for the best experience and the worst experience, and the experimenter will record these views.
RESULTS
The experiment analyzes two aspects: on the one hand, whether there are differences among the three consistency strategies in the same task mode, and on the other hand, whether there are differences in the same consistency strategy between two groups of experiments. The results of subjective scores are shown in Table 1 and Table 2 below.
832 Zhang and Yuan
Table 1. Results of the subjective scores of the first group.
Table available in the original PDF paper.
TE OE CP FP
Equations and mathematical formulas are available in the original paper PDF.
Table 2. Results of the subjective scores of the second group.
Table available in the original PDF paper.
TE OE CP FP
Cross-Device User Experience To compare the impact of three consistency strategies on user experience in the same group of cross-device tasks, this study adopted a one-way analysis of variance ANOVA and detected specific differences through post-hoc multiple comparisons.
In the results of ANOVA for the subjective scores of the first group, tran- sfer experience TE F(2, 87) = 4.860 p = 0.01<0.05, overall experience OE F(2, 87) = 3.057 p = 0.052>0.05 and CP (Consistency Perception) F(2, 87) = 8.103 p = 0.001<0.05 was significant, and functional perception FP F(2, 90) = 1.729 p = 0.183>0.05 was not significant.
Post-hoc multiple comparisons revealed significant differences in TE scores between the mirrored mode and the split interaction pattern. The CP score of mirrored mode was significantly higher than that of the other two modes, and there was no significant difference in CP between detached view and detached interaction.
In the results of the second group of experiments, transfer experie- nce TE F(2, 90) = 11.931 p = 0.000<0.05, overall experience OE F(2, 90) = 7.820 p = 0.001<0.05 and perceptual consistency CP F(2, 90) = 19.282 p = 0.000<0.05 is significant, functional perception FP F(2, 90) = 1.612 p = 0.205>0.05 is not significant.
Post-hoc multiple comparisons revealed significant differences in TE and OE of the mirrored mode versus the detached view mode, and significant differences between the CP mirrored mode and detached view and detached interactions in the second set of experiments.
User Interface Consistency Experience in Composite Cross-Device Interaction 833
Comparison Between Groups To explore the impact of different task types on user experience and consi- stency, a paired sample t-test was conducted on the subjective scores of TE and CP of two groups of participants with the same consistency strategy.
Comparing the mirrored modes of the two groups, the results of the paired sample t-test are shown in Table 3, and there was no significant difference in TE and CP in the two patterns.
Comparing the separated view modes of the two groups, the results of the paired sample t-test are shown in Table 4. In the two modes, the TE of task 1 is significantly higher than that of task 2, and the CP of task 1 is also significantly higher than that of task 2 The CP.
Comparing the separated operation modes of the two groups, the results of the paired sample t-test are shown in Table 5, and there was no significant difference in TE and CP in the two modes.
Table 3. Results of the paired sample t-test in mirrored mode.
Table available in the original PDF paper.
Table 4. Results of the paired sample t-test in separated view mode.
Table available in the original PDF paper.
Table 5. Results of the paired sample t-test in separated operation mode.
Table available in the original PDF paper.
834 Zhang and Yuan
DISCUSSION
Operation-Oriented Task According to the experimental results, the migration experience of the mir- rored mode in this scenario is significantly higher than that of the separated view and separated operation, and the average score of the separated view is higher than the average value of the separation interaction score, but there is no significance. In user interviews, it was concluded that the relevant reason is that the mirrored mode is consistent with the original device interface and interaction method, and can be used without additional learning.
Some participants said that the mirrored mode is not perfect. Although the cost of learning and adapting to new devices is reduced during the migration process, it does not make full use of the ability of multiple devices.
Some participants also said that if it is used many times or for a long time, the learning cost caused by inconsistency is acceptable.
Observation-Oriented Task From the statistical results of users’ subjective ratings, the TE of the mirror image mode in this task scenario is significantly higher than that of the sepa- rated view, and the average value is slightly higher than that of the separation interaction, but it is not significant. Separated view scored significantly lower than the remaining two modes. Through the interviews with the second group of experimenters, it was found that in the observation-oriented task scenario, adopting the method of separating the UI led to visual separation when users observed the interface content (Cauchard et al., 2011), and at the same time resulted in lower consistency in the participants’ perception.
In this mode, the mean TE score of the separated interaction method was close to that of the mirror image mode. The participants reported that in the observation-oriented task, the participants could focus their visual focus on the screen after a short period of adaptation and learning by using the sepa- rated interaction method. At the same time, use blind operation to control the equipment.
User Experience With Different Task Scenarios By comparing the scores of the same mode in the two tasks, it can be found that there is a significant difference between the two modes in the separa- ted view mode. According to the content of user interviews, no matter in which task, the separated view will cause the user’s visual focus to be con- stantly switching between the two devices. For observation-oriented tasks, users are more sensitive to this switching of visual focus, and affect the user’s perception of consistency.
CONCLUSION AND FUTURE WORK This paper explores the user experience of two types of tasks in the com- posite cross-device scenario by building a consistent interface distribution strategy. On the whole, maintaining consistency in compound cross-device tasks is conducive to providing a better cross-device user experience. Even if
User Interface Consistency Experience in Composite Cross-Device Interaction 835
users can adapt to a certain degree of inconsistency, any inconsistency will increase the learning cost of the first use. Therefore, this study believes that in a composite cross-device scenario, if there are no other restrictions and functional requirements, adopting a design scheme with higher consistency as much as possible can help ensure a seamless user experience. At the same time, different tasks will affect the user experience of users, and users of observation-oriented tasks are more sensitive to interface incoherence.
In addition, this study found that there may be differences in users’ percei- ved sensitivity to consistency across dimensions, with users more likely to report differences at the interface level, whereas differences at the interaction level may not be distinguishable from interface differences.
Still to be explored in the following areas: Composite cross-device scenarios include multiple combinations of devi- ces and multiple correspondences in the number of devices. This study only discusses the case of including a synchronous device in the context of asynch- ronous operations, and more possible composite cross-device scenarios To be discussed.
Sequence can be a factor that affects cross-device user experience. This study explores the situation of migrating from one device to multiple devices. The operation situation may be different if it is opened from the opposite order.
ACKNOWLEDGMENT
This research is supported by the Philosophy and Social Sciences Foundation of Hunan, China (No. 22YBA038).
REFERENCES
Brudy, F., Holz, C., Rädle, R., Wu, C. J. and Marquardt, N. ‘Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices’. the 2019 CHI Conference. Cauchard, J. R., Löchtefeld, M., Irani, P., Schöning, J. and Subramanian, S. ‘Visual
separation in mobile multi-display environments’. Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA, October 16-19, 2011. Denis, C. and Karsenty, L. (2005) Inter-Usability of Multi-Device Systems – A Conceptual Framework. Multiple User Interfaces: Cross-Platform Applications and Context-Aware Interfaces. Dong, T., Churchill, E. F. and Nichols, J. ‘Understanding the Challenges of Designing
and Developing Multi-Device Experiences’. the 2016 ACM Conference. Kim, S. W., Jo, H. K. and Ha, D. Y. ‘Different UI, Same UX: A Design Concept for Implementing a Locally-Optimized and Globally-Unified User Experience’. Design, User Experience, and Usability. Theory, Methods, Tools and Practice - First International Conference, DUXU 2011, Held as Part of HCI International 2011, Orlando, FL, USA, July 9-14, 2011, Proceedings, Part II. Nichols, J., Richter, K. and Gajos, K. ‘The many faces of consistency in cross-platform
design: A whitepaper’. CEUR Conference Proceedings. Oliveira, R. D. and Rocha, H. ‘Consistency on Multi-device Design’. Springer Berlin Heidelberg.
836 Zhang and Yuan
Pyla, P. S., Tungare, M. and Pérez-Qu, M. (2006) ‘Multiple user interfaces: Why consistency is not everything, and seamless task migration is key’, Proceedings of the Chi Workshop on the Many Faces of Consistency in Cross Platform Design. Shin, D. H. (2016) ‘Cross-Platform Users’ Experiences Toward Designing Interu-
sable Systems’, International Journal of Humancomputer Interaction, 32(7-9), pp. 503–514. Sørensen, H., Raptis, D., Kjeldskov, J. and Skov, M. B. (2014) ‘The 4C Framework: Principles of Interaction in Digital Ecosystems’, ACM. Wljas, M., Segersthl, K., Vnnen-Vainio-Mattila, K. and Oinas-Kukkonen, H. (2010)
‘Cross-platform service user experience: a field study and an initial framework’, ACM. Yunchen, Huang, Lesley and Strawderman (2013) ‘A System Transferability Questionnaire (STQ)’, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57(1), pp. 1663–1667. Ziefle, M., Arning, K. and Bay, S. (2006) ‘Cross platform consistency and cognitive
compatibility: The importance of users’ mental model for the interaction with mobile devices’.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Ergonomics In Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-53-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional