Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

5G Remote Control in Failure Situations of Transport Robots in Challenging Hospital Environments

Fraunhofer Institute for Material Flow and Logistics, Dortmund, NRW 44227, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1003594

Abstract

Autonomous Mobile Robots (AMRs) continue to facilitate the work of physicians and hospital nurses by releasing those professionals from time consuming transport tasks within hospitals. Nonetheless, AMRs still often face challenges when situations occur, which result in a failure of the navigation system. In this paper, we present an analysis and an implementation of a remote-control mechanism using 5G networks to enable an operator to control an AMR, in our example within a hospital, to support an AMR in situations, where an autonomous navigation faced challenges, that cannot be solved autonomously. In detail, four major challenges are faced when implementing a remote control for failure situations – the data connection itself, the sensor data acquisition and compression, the delivery of the current robot state for a user and the controllability of the robot. For autonomous driving, the AMR is equipped with a 128-layered 3D-Lidar sensor. An RGB-D camera facilitates video feedback for the operator to navigate the AMR manually. Additionally, the point cloud of the 3D-lidar provides a detailed in-depth view of the environment, which recognizes present persons or also allows the operator to drive backwards. To establish a connection between an AMR and a remote operator, a stable and low latency data connection is required. Since the Wi-Fi requirements of hospitals usually do not fit the requirements of remote-controlled robots regarding data security, network coverage, connection latency and bandwidth, the usage of the Wi-Fi network of hospitals is not appropriate. These challenges can be overcome using 5G cellular network to guarantee a low latency, high bandwidth connection which is independent from the regulations and limitations of the local Wi-Fi network. Nonetheless, by selecting the cellular 5G network as the remote operation network, further challenges arise – e.g. coverage of the 5G network or the stable and secure accessibility of the robot. Since hospital building structures are complex and usually are constructed using reinforced concrete, 5G radio waves are reflected or absorbed. In addition, the bandwidth is limited, since a public cellular connection is used. Due to these limitations, data compression is required for transmitting large chunks of sensor data, such as RGB camera streams or point clouds. The RGB video compression is implemented using the H.264 codec, which again can be accelerated using hardware. The point cloud is compressed through an octree implementation. As a result, the sensor data is transmitted with low latency and less lag. Despite using data compression algorithms, which are not lossless, the quality of the sensor data, received by the operator, is still sufficient for remote control operations. For a safe and controlled remote control of an AMR using the above explained technology stack, a data connection with less to no data transfer loss is required.

INTRODUCTION

To further increase the efficiency of their employees, hospitals are increasingly using autonomous mobile robots (AMRs) (Holland et al., 2021). Nonethe- less, AMRs still often face challenges when situations occur, which result in a failure of the navigation system. In this paper, we present an analysis and an implementation of a remote-control mechanism using 5G networks to enable an operator to control an AMR, in our example within a hospital, to support an AMR in situations, where an autonomous navigation faced challenges, that could not be solved autonomously.

In detail, four major technical challenges are faced when implementing a remote control for failure situations – the data connection itself, the sensor data acquisition and compression, the delivery of the current robot state for a user and the controllability of the robot.

The data connection itself, which relies on a cellular 5G connection, is described together with our network stack in the connection section, while the compression of the acquired data is further discussed in the compression section. The complete workflow of our remote operation implementation from a human robot interaction perspective is described in the remote ope- ration part while a technical overview of our proposed system is described in the system design section.

REMOTE OPERATION Remote Operation is an important method for error state resolution when the robot is not able to escape an unforeseen or unknown situation and fulfil its

© 2023. Published by AHFE Open Access. All rights reserved. 96

5G Remote Control in Failure Situations of Transport Robots 97

Figure 1: User flow for the user story.
Original PDF Figure 1:

task. These situations may occur when obstacles block the navigation path or mobile objects physically interact with the robot – like humans blocking the path intentionally or unintentionally. For common hospital environments we identified clusters of roles that share similar behaviours and intentions, which lead to conflicts with the robot or in case of remote operation also solve these situations. The roles are divided into two groups – major and minor roles – from the remote operation point of view. The major roles include the operator and the transporter of goods because they are mainly involved in interaction with the robot and contribute to proper workflow. The minor roles are dispa- tcher of goods, recipient of the goods, technicians, medical staff, patients and visitors. To design adequate human robot interaction, we identified user sto- ries that describe typical interaction and desires of the respective roles. From that point on user flows are elaborated. These flows consider all roles, goals, intentions, involved technologies and means of actions to accomplish tasks in human robot interaction.

The user story for the overall system from the operator’s perspective is stated as: As Operator I have received a Notification, that support is necessary. Now I want to set up a connection from my PC to the mobile robot AMR and support the User on site.

SYSTEM DESIGN As displayed in Figure 2, our system contains three entities. An autonomous mobile robot (AMR), a remote operator and a central virtual private network (VPN) server. The AMR, as described above, has the task to fulfil various transportation tasks, the remote operator steps in, when the AMR faces issues that the AMR can not resolve autonomously and the VPN Server ensures a safe connection between the AMR and a remote operator. The details on the connection between an AMR and a remote operator is further described in the communication section.

In our system, a transport robot has multiple sensors, namely a 128-layer, 360

  • Ouster OS0 laser scanner and multiple RGB HD cameras to navigate through the hospital environment. Furthermore, the AMR is enabled to navi- gate autonomously by the Robot Operating Systems (ROS) navigation stack and its behaviour is controlled by a high level multitask behaviour framework
Figure 2: System overview.
Original PDF Figure 2:

(Hoose et al., 2020). To be able to communicate with other entities, the AMR consists of a 5G router. The pipeline, that compresses the data, which again can be acquired by a remote operator is further described in the compression section.

The remote operator, who can step into situations, where the AMR has issues to continue its task autonomously, has access to a desired AMR using 5G by using a 5G router and a VPN connection as described in the com- munication section. The remote operator is also able to receive RGB camera images and Point Clouds in near real time, which again are recorded by a connected AMR. The remote operator can than monitor this sensor data to send control commands to the connected AMR using a device of their choice as discussed in the communication section.

COMMUNICATION To establish a connection between an autonomous mobile robot (AMR) and a remote operator, a stable and low latency data connection is required to guarantee a safe interaction between a remote operator and an AMR.

Most hospitals already employ Wi-Fi networks for wireless communi- cation and internet access. Nonetheless, those Wi-Fi networks often lack consistent network coverage and have a varying latency and bandwidth e.g., depending on the local signal strength or number of devices in the wireless network. In addition, depending on the network policies within the hospi- tal, remote access to a device within the hospital network is not allowed. For these reasons, we selected a 5G network as the wireless data transmis- sion technology for our use case. 5G has the advantage that it is built for many devices and is easily accessible. In detail, we use the public cellular 5G network to be independent from the hospital’s infrastructure. In addition, 5G networks, compared to 4G networks, provide the advantage of lower latencies (Al-Saadeh et al., 2018), which are crucial to stable remote-control operations.

5G Remote Control in Failure Situations of Transport Robots 99

Nonetheless, by selecting the public cellular 5G network as the remote operation network, further challenges arise – e.g., connection security risks and IP addresses shared with multiple devices. For this reason, we employ a virtual private network (VPN), since VPN ensures a secure connection and does not require a public IP address that is not shared with other users for the connected clients. In detail, we selected Wireguard (Mackey et al., 2020) as the VPN software of our choice because of its properties regarding speed, bandwidth, security, and reliability (Mackey et al., 2020). Nonetheless, to be capable of using a VPN, a third instance besides a remote operator’s device and an AMR is required – a VPN server. In our case, the VPN server also organizes the routing between all connected VPN clients.

In addition to VPN, two OSI application layer (Kurose and Ross, 20073) protocols are used to create a connection between an AMR and a device, used by a remote operator. First of all, GStreamer (Nimmi et al., 2014) creates a connection to transfer live RGB camera data to a remote operator, as further described in the compression section. Secondly, for control commands and compressed point cloud data transfer, the Data Distribution Service (DDS) (Pardo-Castellote, 2003) protocol is used. In detail, the CycloneDDS (Desbi- ens, 2023) implementation, available for ROS2 (Maruyama et al., 2016), has been used with UDP (Kurose and Ross, 2007) as configured transfer protocol. Furthermore, the ROS2 sensor transport quality of service (QoS) settings (Maruyama et al., 2016) have been used to ensure a connection with sufficient bandwidth.

COMPRESSION Since hospital building structures are complex and usually are constructed using reinforced concrete, 5G radio waves are reflected or absorbed. In addi- tion, the bandwidth is limited, since a public cellular connection is used. Due to these limitations, data compression is required for transmitting large chunks of sensor data, such as RGB camera streams or point clouds. The RGB video compression is implemented using the H.264 codec (Wiegand et al., 2003) by applying a GStreamer pipeline (Nimmi et al., 2014) (see figure 3), which again can be accelerated using hardware video encoders via va-api.

The point cloud is compressed through an octree (Meagher, 1982) imple- mentation as displayed in figure 4. The octree has been implemented as a

Figure 3: GStreamer pipeline - GStreamer pipeline for the camera-stream. The camera images are accessed through the v4l2src element and encoded with h264enc element via va-api. Then the video stream is sent via UDP through our communication setup, as described in the communication section.
Original PDF Figure 3:
Figure 4: Octree implementation - visualization of the octree compression with depth level of two with three points (left). On the right side the tree structure and the encoded byte-stream of the octree.
Original PDF Figure 4:

binary tree, where each node of the octree is described as one byte. A bit in each node refers either to a node in the next depth level or, when the maxi- mum depth level is reached, a leaf, which again, represents a point within the point cloud that is compressed by the octree. The octree is encoded as a byte-stream in depth first in-order traversal. The maximum depth level and length of the region of interest must be given. As a result, the sensor data is transmitted with low latency and less lag. Despite using data compression algorithms, which are not necessarily lossless, the quality of the sensor data received by the operator is still sufficient for remote control operations.

CONCLUSION

In conclusion, we defined a remote-control workflow from the point of view of a remote operator to support a transport robot in situations where said transport robot failed in its navigation. Additionally, we implemented the remote operation mechanism using DDS and direct UDP connections in cel- lular 5G networks in combination with data compression to ensure a stable and reliable remote operation experience.

OUTLOOK In our future work we plan to do further long-term tests in a real-world hospi- tal to evaluate the long-term reliability of our implemented remote-control mechanism. In addition, we plan to further extend our remote operation capabilities to use Augmented Reality (AR) and/or Virtual Reality (VR) glas- ses since an operator might profit from a more immersive way to control a remote operating autonomous mobile robot.

ACKNOWLEDGMENT

This work was funded by the Ministry of Economic Affairs, Industry, Climate Action and Energy of the State of North Rhine-Westphalia, Germany through the 5G.NRW research competition. https://www.5G-wettbewerb.nrw.

5G Remote Control in Failure Situations of Transport Robots 101

REFERENCES

Desbiens, F. (2023). DDS. In Building Enterprise IoT Solutions with Eclipse IoT Technologies (pp. 121–154). Apress, Berkeley, CA. Hoose, S., Sendzik, D., Jost, J., & Kirks, T. (2020). A Flexible High and Low Level

Multi Task Behavior Framework for Multi-agent Systems in Complex Environ- ments. In 2020 5th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS) (pp. 68–72). IEEE. Jane Holland, Liz Kingston, Conor McCarthy, Eddie Armstrong, Peter O’Dwyer, Fionn Merz and Mark McConnell Service Robots in the Healthcare Sector Robotics 2021. Kurose, James F., and Keith W. Ross. (2007) “Computer Networking: A Top-Down

Approach Edition.” Addision Wesley. Mackey, Steven, et al. (2020). “A Performance Comparison of WireGuard and OpenVPN”, Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy. Maruyama, Y., Kato, S., & Azumi, T. (2016). Exploring the performance of ROS2.

In Proceedings of the 13th International Conference on Embedded Software (pp. 1–10). Meagher, Donald. Geometric modeling using octree encoding. Computer graphics and image processing, 1982, 19. Jg., Nr. 2, S. 129–147. Nimmi, S., V. Saranya, and R. Gandhiraj. (2014) “Real-time video streaming using

GStreamer in GNU Radio platform.” 2014 International Conference on Green Computing Communication and Electrical Engineering (ICGCCEE). IEEE. Osama Al-Saadeh; Gustav Wikstrom; Joachim Sachs; Ilaria Thibault; David Lister, End-to-End Latency and Reliability Performance of 5G in London, 2018 IEEE Global Communications Conference (GLOBECOM). Pardo-Castellote, G. (2003). Omg data-distribution service: Architectural overview.

In 23rd International Conference on Distributed Computing Systems Workshops, 2003. Proceedings. (pp. 200–206). IEEE. Pekka Isto*, Tapio Heikkilä, Aarne Mämmelä, Mikko Uitto, Tuomas Seppälä, and Jari M. Ahola, 5G Based Machine Remote Operation Development Utilizing Digital Twin, De Gruyter, VTT Technical Research Centre of Finland Ltd, P. O. Wiegand, Thomas & Sullivan, Gary & Bjøntegaard, Gisle & Luthra, Ajay.

(2003). Overview of the H.264/AVC Video coding standard. Circuits and Systems for Video Technology, IEEE Transactions on. 13. 560–576. 10.1109/TCSVT.2003.815165.

Cover of the proceedings volume Human Factors and Systems Interaction

Explore the complete digital volume

Human Factors and Systems Interaction

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-60-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →