Xiaoli Chen, Guoyan Wang, Yongheng Lu, Can Yao, Guocheng Hao (2022). Optimal Interaction Design of Remote Operator Interfaces: A Case study of Patrol Separation Interfaces for Corridor Robots. In: Human Factors and Systems Interaction. AHFE Open Access, vol. 52, pp. 313–321. https://doi.org/10.54941/ahfe1002171
Although the displayed interface has been emphasized in the study of robotic teleoperation systems, the role played by image information in it has not been fully elucidated. This study designed the Patrol-inspection Separation (PIS) interaction interface using the inspection robot of utility tunnel (IR-UT) as the experimental object. To verify the PIS interaction interface usability, we proposed the IR-UT displayed interface usability evaluation after summarizing the previous studies and assessing the interface combination's degree of excellence. Six different combinations of displayed interfaces were set up, and 17 subjects controlled the IRUT and completed the shooting tasks under the six interfaces. At the end of each interface shooting task, subjects filled out a cognitive load assessment scale and a usability evaluation questionnaire. The results are as follows: the PIS interface and the pan-tilt camera's live image can optimize the operational performance; the dual pan-tilt images can decrease the subjects' performance compared with the single pan-tilt image. The above findings are intended to provide meaningful references for optimizing the teleoperation interface.
Teleoperated robots’ stability, reliability, and high precision have been signi- ficantly improved in recent years. In past studies, some researchers found that it is hard for the operator to perceive the distance to a target or obstacle because the operator has to perceive the robot’s position and surroundings only through the operation interface, which is similar to understanding the environment through a ‘keyhole’ (Fong et al., 2003), (Nielsen et al., 2007), (Woods et al., 2004). Many researchers have made efforts to address this issue. Hunglin Chi et al. (2012, p. 641–652) designed a four-view teleo- perated system for the crane, including top, left, right, and global, and it can show guidance information based on augmented reality technology. The study found that the interface improved the efficiency of the teleoperated control with less mental load. However, Voshell et al. (2005, p. 442–446) found that the operator would mistake for the operation speed and reduce
operational performance under the multiple-view interface. In addition, the interface’s display type and manner influenced the operator performance a lot (Young and Peschel, 2020), so the interface should try to minimize the distra- ction of the operator and allow visual redundancy of important information (Endsley, 1998). Lamb et al. (2005, p. 31–37) found that the operators per- form better on the teleoperated control when they avoid collisions and adapt to complicated tasks centered on self-perspective. They would obtain global information on the robotic arm path when aligning and capturing functions centered on external perspective. Olmos (2000, p. 247–271) found that it is difficult for operators to integrate information from the egocentric and exo- centric perspectives. If the operators mishandled this information, they will misjudge the current situation and reduce operational performance.
There are few studies on IR-UT displayed interface, mainly focusing on IR- UT simulation and control. In this study, we designed the Patrol-inspection separation interface and proposed the IR-UT interface usability evaluation to verify its usability and assess the interface combination’s degree of excellence. We simulated the working environment of IR-UT inspection and asked the subjects to complete the inspection task with six different image feedbacks. We obtained and analyzed the subjects’ subjective perception and objective performance indicators to provide meaningful references for optimizing the IR-UT displayed interface and personnel assessment.
METHODS
Experimental Platform In this study, the experiments were conducted under the self-developed IRUT. We built the inspection robot telerobotic system of utility tunnel (IRTS-UT) based on the US Microsoft Windows system. The displayed interface is pre- sented on a Founder FG981-WT 1440*900 60Hz displayer. In addition, the IR-UT is controlled by the Qt Creator, which is the remote-control system application.
Experimental Design Subjects This study recruited 17 subjects (mean age = 20 years), equally distribu- ted across gender. Subjects had normal or corrected visual acuity. According to their report, they were right-handed and had no operating robotic arms experience.
PIS Interface We designed the patrol-inspection separation (PIS) interface as follows. The end-of-arm camera’s live image and the pan-tilt image could automatically switch relative sizes on the screen depending on the experimental task’s obje- ctive. In the patrol phase, the end-of-arm camera’s live image is small and the pan-tilt image is large, as in the inspection phase, the end-of-arm camera’s live image is large, and the pan-tilt image is small (as shown in Fig. 1). As a
Optimal Interaction Design of Remote Operator Interfaces 315
Figure 1: Description of the PIS image feedback mechanism.
side note, when all three images are presented on the image screen, all images will no longer switch sizes automatically.
The IR-UT Display Interface Usability Evaluation Based on Genetic Robotic Training (GRT) manual, we proposed the IR-UT displayed interface usability evaluation to verify the usability of the PIS. The IR-UT displayed interface usability evaluation includes subjective perception and objective performance. The subjective perception indicators referred to the cognitive load and the interface usability evaluation. These two indicators were calculated based on the cognitive load assessment scale and usability evaluation questionnaire completed by the subjects. We divided the objective performance indicators into four categories: time indicators, process indi- cators, efficiency indicators, and success indicators. This study divided the objective performance indicators into four categories. The time indicator was the total time. The process indicators included the joint limit times and the initialization times. The efficiency indicators included the robotic arm opera- ting efficiency, end-position adjustment efficiency, and end-angle adjustment efficiency. The success indicator was the target completion rate.
Tasks In this study, six teleoperation interfaces were set up according to the differe- nces in the image presented (as shown in Table 1). The three types of images used in the experiment are shown in Figure 2. The subjects completed the inspection task under six kinds of displayed interfaces at a fixed distance. The experimental process was divided into teaching training and formal experi- ment. The subjects all received the same teaching training. After completing all experimental tasks, the subjects completed the questionnaires.
RESULTS
We used spss26.0 to process the experimental data. The statistics showed that all subjects’ target completion rate was 100%, so this indicator was not involved in the correlation analysis. The six displayed interfaces were divided into two main categories: groups A+B and A+C. The group aims to learn how the PIS interfaces impact a single number of heads. The second category, groups A+B, A+C, A, and A+B+C, aims to learn how pan-tilts’ mode and number affect without PIS interface.
Figure 2: The three types of images.Table 1. Description of the content of the image feedback of the displayed interface.
Table available in the original PDF paper.
Group number
Image combination Method
With or without PIS image Feedback mechanism
1 A+B Yes 2 A+B No 3 A+C Yes 4 A+C No 5 A No 6 A+B+C No
The Indicators of Objective Performance and Subjective Perception Objective Performance Indicators The end adjustment efficiency (end-position adjustment efficiency, end-angle adjustment efficiency) proposed in this study was significantly correlated with most metrics. The end-position adjustment efficiency was significan- tly negatively correlated with cognitive load (p<0.001), total time (p<0.001), joint limit times (p = 0.019), and initialization times (p = 0.012). In addition, it significantly positively correlated with robotic arm operating effi- ciency (p< 0.001), end-angle adjustment efficiency (p<0.001), and usability evaluation (p<0.001).
There was no statistically significant difference between the end-angle adjustment efficiency and initialization times. The end-angle adjustment effi- ciency was significantly negatively correlated with cognitive load (p<0.001), total time (p<0.001), and joint limit times (p = 0.018). In addition, it was significantly positively correlated with robotic arm operating efficiency, end-position adjustment efficiency, and usability evaluation (p<0.001).
The only indicator correlated with the joint limit times was cognitive load, while the initialization times correlated with total time and usability evaluation.
Subjective Perception Indicators Cognitive load was significantly correlated with all indicators and had a Pear- son correlation coefficient of -0.741 with usability evaluation, the strongest of all indicators. On the other hand, usability evaluation was significantly correlated with all indicators except for the joint limit times.
Optimal Interaction Design of Remote Operator Interfaces 317
Table 2. Independent samples t-test (with PIS).
Table available in the original PDF paper.
Groups Indicators Cognitive Total time Robotic arm End-position
t ±3.159 ±3.542 3.293 2.558 2.185 p 0.003 0.001 0.004 0.015 0.036 A+C With PIS 3.71±0.71 778.06±137.18 0.39±0.07 7077±3.94 3.62±1.38 With-out PIS t 2.189 2.909 ±3.019 ±1.946 ±2.979 p 0.036 0.007 0.005 0.06 0.005
Interface Presentation Analysis Patrol-Inspection Separation Due to the nature of the patrol-inspection separation (PIS) image, only groups A+B and A+C contained the PIS interface. We compared each metric with and without the PIS interface (Table 2).
Total time: The significance level of the total time was p = 0.001<0.05 in group A+B (end-of-arm camera’s live image + pan-tilt camera live image); and p = 0.007<0.05 in group A+C (end-of-arm camera’s live image + pan- tilt live model image), indicating that the total time spent was significantly different in both with and without PIS interface. In group A+B, there was a significant decrease in total time in the PIS interface compared to the interface without PIS. However, in group A+C, the interface with PIS increased the total time.
Cognitive load, robotic arm operating efficiency, and usability evaluation: All three indicators differed significantly in the interface with and without PIS (p<0.05). In the A+B group, the interface with PIS significantly reduced the cognitive load and improved the robotic arm operating efficiency. The usability evaluation score was higher for the interface with PIS. However, in the same situation in the A+C group, there was a decrease in the indicators above. The PIS interface increased the subjects’ cognitive load and decreased the robotic arm efficiency and ease of operation.
End position adjustment efficiency: In group A+B, the end-position adju- stment efficiency was higher in the PIS interface than in the interface without PIS (p<0.05), but in group A+C, the interface with and without patrol sepa- ration did not have much effect on the end-position adjustment efficiency. The rest of the metrics were not significantly different in the interface with or without PIS.
Interface Presentation Analysis-1 In the absence of the PIS interface, the effect of pan-tilt mode and the number of pan-tilts were analyzed using an independent samples t-test.
Comparing the two kinds of Pan-tilt mode under a single Pan-tilt mode situation. We were comparing group A+B and group A+C, the cognitive load (p<0.001) and total time (p<0.001) were much less in the A+B group than in the A+C group. Furthermore, the robotic arm operating efficiency (p<0.001), end-position adjustment efficiency (p<0.001), end-angle adjustment efficie- ncy (p = 0.007), and usability evaluation (p<0.001) were all higher in the A+B group than in the A+C group.
Comparing the single Pan-tilt mode with the dual Pan-tilt mode. Compa- ring group A+B and group A+B+C, the cognitive load was lower in group A+B than in group A+B+C (p = 0.049). In addition, the robotic arm opera- ting efficiency (p = 0.029), end-position adjustment efficiency (p = 0.031), and end-angle adjustment efficiency (p = 0.016) were higher in group A+B.
However, comparing group A+C and group A+B+C, cognitive load (p = 0.003) and total time (p<0.001) were higher in group A+C than in group A+B+C, and robotic arm operating efficiency (p<0.001), end-position adjustment efficiency (p = 0.036), and usability evaluation (p<0.001) were lower in group A+C.
Comparing the no Pan-tilt mode with the single Pan-tilt mode. Compa- ring group A+B and group A, group A+B showed significant differences (p<0.05) from group A in all metrics, except for the joint limit times. Com- pared to group A with no Pan-tilt mode, group A+B had a lower cognitive load, less total time, and higher ratings of robotic arm operating efficiency, end adjustment efficiency, and ease of use.
However, comparing group A+C and group A, there were no significant differences (p>0.05) between group A+C and group A for all metrics with the inclusion of the pan-tilt live model image, and group A+C was worse than group A in terms of data performance.
DISCUSSION
In this study, we verified the usability of the PIS interface, examined the opera- tional impact of the number and the mode of pan-tilt to advise on optimizing the teleoperation interface’s design.
The impact of the Patrol Inspection Separation Interface The PIS interface is more in line with the subject’s tendency to focus on the content of the operation and intermittently based on the operation. We found that in a single pan-tilt mode, the use of the PIS interface in the A+B group (end-of-arm camera’s live image + pan-tilt camera live image) significantly improved subjects’ robotic arm operating efficiency and end adjustment effi- ciency, reduced subjects’ total time and cognitive load, and improved subjects’ usability evaluation scores (Table 2).
The more utilized image is enlarged in the PIS interface while the secon- dary image is reduced. However, in group A+C (the end-of-arm camera’s live image + the pan-tilt live model image), the use of PIS would increase the subjects’ cognitive load, reduce the subjects’ robotic arm operating efficie- ncy and the end adjustment efficiency, and the subjects’ usability evaluation score would be reduced. We hypothesize that the live model images could
Optimal Interaction Design of Remote Operator Interfaces 319
increase the cognitive load on the subjects, as they had to spend more time understanding the model’s imaging principles. Therefore, we suggest that the PIS interface should not be introduced when the interface contains live model images, which would overload the subjects’ cognition. The PIS interface can be added under the pan-tilt camera live image, which improves operational performance.
Pan-Tilt Pattern and Quantitative Impact Comparing the camera’s live image, the presentation of the live model image would make it more difficult for the subjects to interpret the image informa- tion. In fact, in the case of the end-of-arm camera’s live image matching with the single pan-tilt image, the group A+C with the pan-tilt live model image performed worse than the group A+B without any live model image about all indicators.
Further analysis revealed that the virtual components in the live model image differed from the real scene. The operators needed to align the two with calibration errors, which increased the cognitive load on operators and decre- ased performance. Similar studies such as Vozar et al. (2012, p. 448–455) found that performance decreased although the operator’s distance perce- ption was better in the augmented reality interface than in the video-only interface.
Although the pan-tilt image information is helpful for operational tasks, the dual pan-tilt images increase the burden on the subjects’ cognitive load during processing overlapping information. Under the A+B+C group of dual pan-tilt images (pan-tilt camera live image and pan-tilt live model image), the performance of the indicators was better than group A+C and worse than group A+B. According to N. Matsui et al.’s fuzzy control model of human attention allocation behavior, the operator can allocate attention optimally when the number of messages displayed is three or four (Matsui et al., 1988), (Matsui et al., 1986). Therefore, the amount of pan-tilt image information should not be too much. In the design of the IR-UT interface, the appropriate amount of displayed pan-tilt image information needs to concern the purpose and content of the operational task.
The lack of pan-tilt image information would place higher demands on the subjects’ cognitive abilities, such as spatial orientation and mental rotation, implying an increase in the task’s difficulty. The experimental results sup- ported this inference. All indicators of group A with no pan-tilt image were worse than group A+B with the pan-tilt camera live image.
Personnel Evaluation Our study provides an empirical basis for evaluating personnel’s ease of using the teleoperation interface.
Subjective Perception Indicators The Pearson analysis shows that cognitive load is significantly correlated with all indicators, and usability evaluation is significantly correlated with
all indicators except the joint limit times. These two indicators are calcu- lated based on the questionnaires filled out by the subjects after operation, which can directly reflect the influence of the teleoperated interface on the subjective feelings of the personnel. The cognitive load is significantly and negatively correlated with the usability evaluation. It means that the greater the cognitive load on the subject during the experimental task, the worse the experience of operating the teleoperated interface. It has a positive effect on designing efficient IR-UT teleoperated user interfaces by reducing the num- ber of interaction steps and complex thinking of the operator (Interaction, 1995), (Shneiderman and Technology, 1988).
Objective Performance Indicators The end of the robotic arm requires a high degree of accuracy in both posi- tion movement and angle change when reaching the target shooting point. The end-position adjustment efficiency and end-angle adjustment efficiency proposed in this study could reflect the subjects’ efficiency in adjusting the robotic arm’s joints. The larger their values, the fewer key taps are required to compensate for the unit deviation. From the Pearson analysis results, the end-position adjustment efficiency correlated with all indicators significan- tly, and the end-angle adjustment efficiency was associated with all indicators except the initialization times. So, the two indicators could be used as a refe- rence to evaluate how easy it is for the operator and optimize the teleoperated interface of IR-UT in the future.
CONCLUSION
This study examined the operational impact on personnel of introducing a patrol-inspection separation interface in the IR-UT teleoperated interface. Our study found that it is helpful to introduce the patrol-inspection sepa- ration in the camera’s live image. The operators could more quickly and accurately access information when the primary image zoomed in support of the patrol-inspection separation mechanism. However, In the case of the live model image, introducing a patrol-inspection separation interface for inspection was unsuitable because of its specificity. Moreover, the live model image was not ideal for the IR-UT teleoperated interface. Furthermore, dual pan-tilt images were a burden because the conflicting information confused the operator. In contrast, no pan-tilt images would add more difficulties for the operator. These findings above would provide important implications for optimizing the design of the teleoperated interface.
REFERENCES
Chi, H. L., Chen, Y. C., Kang, S. C. & Hsieh, S. H. J. A. E. I. 2012. Development of user interface for tele-operated cranes - ScienceDirect. 26, 641–652. Endsley, M. R. Design and Evaluation for Situation Awareness Enhancement. Human
Factors Society, Meeting, 1998. Fong, T., Thorpe, C. & Baur, C. J. I. T. O. I. E. 2003. Multi-robot remote driving with collaborative control. 50, 699–704. Interaction, D. N. J. R. I. H. C. 1995. The psychopathology of everyday things.
Optimal Interaction Design of Remote Operator Interfaces 321
Lamb, P. & Owen, D. Human performance in space telerobotic manipulation. Proceedings of the ACM symposium on Virtual reality software and technology, 2005. 31–37. Matsui, N., Bamba, E. J. S. & Japan, C. I. 1988. Evaluative cognition and attention
allocation in human interface. Matsui, N., Bamba, E. J. T. O. T. S. O. I. & ENGINEERS, C. 1986. Consideration of the Attention Allocation Problem on the Basis of Fuzzy Entropy. 22, 623–628. Nielsen, C. W., Goodrich, M. A. & Ricks, R. W. J. I. T. O. R. 2007. Ecological
Interfaces for Improving Mobile Robot Teleoperation. 23, 927–941. Olmos, O., Wickens, C. D. & Chudy, A. J. I. J. O. A. P. 2000. Tactical Displays for Combat Awareness: An Examination of Dimensionality and Frame of Reference Concepts and the Application of Cognitive Engineering. 10, 247–271. Shneiderman, B. J. J. O. T. A. F. I. S. & Technology 1988. Designing the User Interface:
Strategies for Effective Human-Computer Interaction. 39, 603–604. Voshell, M., Woods, D. D., Phillips, F. J. H. F. & Proceedings, E. S. A. M. 2005. Overcoming the Keyhole in Human-Robot Coordination: Simulation and Evaluation. 49, 442–446. Vozar, S. & Tilbury, D. M. Improving UGV teleoperation performance using novel
visualization techniques and manual interfaces. Unmanned Systems Technology XIV, 2012. SPIE, 448–455. Woods, D. D., Tittle, J., Feil, M., Roesler, A. J. I. T. O. S. M. & C, C. P. 2004.
Envisioning Human–Robot Coordination in Future Operations. 34, 210–218. Young, S. N. & Peschel, J. M. J. I. T. O. H.-M. S. 2020. Review of Human–Machine Interfaces for Small Unmanned Systems With Robotic Manipulators. pp. 1–13.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Systems Interaction
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-28-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional