Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Challenges in Research Projects on Augmented Reality in the Domain of Mechatronic and Robotic Applications

Heilbronn University, Faculty of Mechanics and Electronics, Max-Planck-Str. 39, 74081, Heilbronn, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1002315

Abstract

The team of the Otto Rettenmaier Research Laboratory for Digitalization at Heilbronn University has been working for years on research projects for interaction with robots as well as a mechatronic factory system using augmented reality (and also virtual reality as well as quite simply via mobile devices - keyword BYOD). As AR Device usually a Microsoft Hololens is used, in the VR area a system of HTC Vive is available in the lab. Other mobile devices are smartphones as well as tablets from different manufacturers.A number of different implementation challenges have been identified in the projects, some of which are quite different. As a main issue is certainly the combination of industrial components such as the robots with the systems that are used for the augmented reality applications. The AR applications and systems typically originate from the gaming world and are not aligned with typical industrial communication standards in terms of issues such as data communication and more. As described, one technical hurdle that has emerged is the communication between the interaction devices and the mechatronic systems. In the industrial world - based on the topic of Industry 4.0 and the associated developments - the OPC UA communication architecture has emerged. However, OPC UA -quasi as an industrial standard- is of course established as a standard in entertainment technology.In addition, there are questions about the development environment. This is predetermined on the side of the mechatronic systems by the products from the control technology, but relatively free on the side of the AR/VR applications. The bridge between these worlds must be developed in the projects anyway. The bandwidth of the respective engines for the development of AR/VR applications, on the other hand, is very broad and very dynamic - meaning that the possible range of functions changes with every update.The biggest challenges, however, come from the actual application scenarios based on industrial production. The classical interaction concepts, which are given for example by the respective operating terminals of robots and are known to the user, can only be transferred to the virtual or extended space with great difficulty. Furthermore, the mapping of the actual production environment in the virtual world is of course a very big problem.The article explains the hurdles encountered in the various research projects and the solutions developed in the projects as well as the questions that are still open.

INTRODUCTION

The team of the Otto Rettenmaier Research Laboratory for Digitalization at Heilbronn University has been working for years on research projects for the interaction with robots (Ehmann and Wittenberg, 2018 (1); Ehmann and Wittenberg, 2018 (2)) as well as a mechatronic factory system using augmen- ted reality or as the well-known abbreviation AR (Rempel et al., 2019 (1); Rempel et al., 2019 (2); Wittenberg et al., 2020). It started with AR appli- cations on tablet computers inspired by game design (Buyer and Wittenberg 2012; Buyer and Wittenberg 2013; Buyer and Wittenberg, 2015) to control industrial PLCs. Next steps and parallel to the AR applications on tablet computers, smart glasses for AR applications and head mounted displays for VR application, were tested and analyzed (Zieringer et al, 2019; Zieringer et al., 2020; Bareis et al., 2022; Schloer et al., 2022). User analysis sho- wed that there are great expectations on AR applications in robotic tasks (Bläss and Wittenberg, 2017 (1), Bläss and Wittenberg, 2017 (2)). Augmen- ted and Virtual Reality are certainly among the promising technologies of the future in human-technology interaction (Broll et al., 2019). To meet these expectations, research is ongoing in the Otto Rettenmaier Research Laboratory.

As AR devices, today often smart glasses, the Microsoft HoloLens or the Microsoft HoloLens 2 are used in the laboratory. In this laboratory, as already mentioned, research is not only done in the field of AR, but also in the field of virtual reality (VR). Here, the areas of the digital twin and many more are being researched. In the VR area head mounted displays like the HTC Vive are available. Other devices which are used in the field of AR in the

© 2022. Published by AHFE Open Access. All rights reserved. 92

Challenges in Research Projects on Augmented Reality 93

Laboratory are mobile devices like smartphones as well as tablet computers from different manufacturers.

The main application for the research activities in our laboratory is the combination of a typical 6-axis industrial robot embedded in a production environment and the current most widely used AR device, the Microsoft HoloLens.

CHALLENGES Several different implementation challenges have been identified in the pro- jects, some of which are quite different. A main issue is certainly the combination of industrial components such as the robots with the systems that are used for the augmented reality applications. The AR applications and systems typically originate from the gaming world and are not aligned with typical industrial communication standards in terms of issues such as data communication and more.

These challenges lie primarily in the three areas of the development envi- ronment, communication, and application scenarios, which are described below.

Development Environments One of the challenges is the variety of development environments that arise when combining AR devices and industrial applications. AR devices were ori- ginally developed for the gaming market and are addressed via game engines. The industrial applications (control of robots, control of other units via PLC, communication configuration, etc.) represent another inhomogeneous field. While PLC programming is largely standardized (IEC 61131, John, Tiegel- kamp, 2009), there is great variability in the programming of robots as well as in the multitude of different bus systems. On this side the development environment is given by the products from the control technology.

But it is relatively free on the side of the AR and VR applications. The bridge between these worlds must be developed in the projects anyway. The bandwidth of the respective engines for the development of AR and VR appli- cations, on the other hand, is very broad and very dynamic - meaning that the possible range of functions changes with every update. However, the graphics engines for the development of VR and AR applications are mainly located in the gaming area. The Unreal Engine and Unity are worth mentioning here. As shown on Figure 1 and 2 the agony of choice is the challenge working with AR and VR applications and their devices. Furthermore, the interconnection between these different development environments is very rare.As can also be seen from the example on Figure 2, not all development environments are available for the various consumer devices from the AR or VR area.

Communication As described, one technical hurdle that has emerged, is the communication between the interaction devices and the mechatronic systems. In the industrial world - based on the topic of Industry 4.0 and the associated developments - the OPC UA communication architecture has emerged (Mahnke et al., 2009;

94 Wittenberg and Bauer

Figure 1: Example of the relationship of the IDEs and their industrial devices.
Original PDF Figure 1:
Figure 2: Example of the relationship of the Game Engines and AR/VR devices.
Original PDF Figure 2:

Lange et al., 2010). However, OPC UA - quasi as an industrial standard - is of course not established as a standard in entertainment technology.

Although OPC UA is a quasi-standard in the industry, at the beginning only OPC UA servers were executable on the PLCs, which were connected to the robot controllers via real-time bus. Since the PLCs are proprietary devices, this could not be adapted. For this reason, different approaches were pursued. In the beginning as shown in Figure 2, OPC UA clients were implemented on the consumer devices and several OPC UA client instances were executed on the consumer devices. The problem with this type of communication is the usability of the creation and processing of the connection and the adaptability to different devices. A different approach is therefore taken with middleware. As shown in Figure 3, the middleware is deployed between the PLCs and the consumer devices, as the name suggests. It takes over the functionality of the consumer device from the previous approach and runs multiple OPC UA clients on it.

One problem of communication between consumer devices from the AR or VR area and industrial devices such as robots is that this is not yet real- time capable according to the current state of the art. For this reason, the robot controllers in the Otto Rettenmaier research laboratory are currently not connected directly to the consumer devices, but rather via PLCs. The authors hope that this will change in the future through new technologies such as the use of OPC UA over TSN.

TSN was originally developed as a real-time communication technology for the entertainment industry, and then found to be suitable for industrial use. Strongly Ethernet based, it currently lacks access to AR devices, which

Challenges in Research Projects on Augmented Reality 95

Figure 3: Communication.
Original PDF Figure 3:
Figure 4: Communication middleware.
Original PDF Figure 4:

come from the gaming environment. However, real-time is a relevant topic for the control and monitoring of industrial robots. The development of an own real-time extension for the AR devices is considered too costly. As written above, the hope here lies in the further development and widespread use of OPC UA over TSN, escecially in the open source area (Pfrommer, Usländer, 2020) which can be easily adapted to the existing system.

Application Scenarios The biggest challenges, however, come from the actual application scenarios based on industrial production. The classical interaction concepts, which are given for example by the respective operating terminals of robots and are known to the user, can only be transferred to the virtual or extended space with great difficulty. Furthermore, the mapping of the actual production envi- ronment in the virtual world is of course a very big problem. At the beginning of the research, simple applications were developed with tablet computers (bring your own device: BYOD). These used the camera of a tablet computer and displayed broad information on it when viewing a QR tag, for example.

This improved the usability especially in the area of monitoring and displa- ying information about different devices. However, this did not yet allow the devices to be operated. Once AR devices like smart glasses became availa- ble, they were first used for the same purpose. However, the benefit of this

96 Wittenberg and Bauer

Figure 5: Using a tablet for interaction with a PLC (Buyer, Wittenberg, 2015).
Original PDF Figure 5:

was hardly higher than that of tablet computers. For this reason, early attem- pts were also made to implement the condition of industrial devices such as robots on smart glasses. The first idea was to virtually represent the available real handheld robotic devices and operate them using the input methods pro- vided by a smart glass. With HoloLens, for example, these are simple hand gestures to create mouse clicks that are executed where the head is pointing at. However, user surveys showed that the robots’ virtualized handheld devi- ces were very difficult to operate using the input methods of smart glasses, the real robotic handheld devices are far superior. Our next research activities focus precisely on this area.

CONCLUSION

As illustrated, there are several challenges that arise when AR is to be used in the field of robotics or in industry more generally. These are not limited to the challenges presented in the selection of the development environment or the type of communication or the choice of application scenarios. Pre- sumably, there are other challenges that arise in this environment; however, they are worth solving as the added value of using AR technology greatly improves the usability of such complex devices. For this reason, research in this area will continue in the Otto Rettenmaier Research Laboratory in the future.

REFERENCES

Bareis C., Uhl F., Zeyer M., Bauer B., Wittenberg C. (2022) Human-Machine Interaction: Controlling of a Factory with an Augmented Reality Device. In: Ahram T., Taiar R. (eds) Human Interaction, Emerging Technologies and Future Systems V. IHIET 2021. Lecture Notes in Networks and Systems, vol 319. Springer, Cham. https://doi.org/10.1007/978-3-030-85540-6_105

Challenges in Research Projects on Augmented Reality 97

Bläss, M., Wittenberg, C. (2017 (1)): An online user analysis regarding the usage of mobile Augmented and Virtual Reality devices in the field of robotics. In: S. Lackey, J. Chen (Ed.): Virtual, Augmented and Mixed Reality, Springer, Cham, 2017, pp. 383–393. Bläss, M., Wittenberg, C. (2017 (2)): Online-Nutzeranalyse zum Einsatz von

Augmented Reality (AR) und Virtual Reality (VR) im Bereich der Robotik. In: J. Reiff-Stephan (Hrsg.): Tagungsband AALE 2017, VDE-Verlag, Berlin, 2017, pp. 247–250. Broll, W., Dörner, R., Grimm, P., Jung, B. (2019): Virtual und Augmented Reality (VR/AR): Grundlagen und Methoden der Virtuellen und Augmentierten Realität. (2019). Deutschland: Springer Berlin Heidelberg. Buyer, S., Wittenberg, C. (2012): Digitale Spiele als Vorbild – Interfacedesign einer

Augmented Reality Anwendung im industriellen Umfeld. Useware 2012, VDI- Verlag, 2012, pp. 283–292. Buyer, S., Wittenberg, C. (2013): AR und Webpad – nur Spiele(reien)?. In: Grundlagen und Anwendungen der Mensch-Maschine-Interaktion – Tagungsband 10. Berliner Werkstatt Mensch-Maschine-Systeme, 2013, pp. 187–191. Buyer, S., Wittenberg, C. (2015): AR and maintenance - visualisation of process data

and engineering information. In: Constantine Stephanidis (Ed.): HCI International 2015 – Posters’ Extended Abstracts, Springer Cham, 2015, pp. 159–162. Ehmann, D., Wittenberg, C. (2018): The idea of Virtual Teach-In in the field of industrial robotics. Proceedings of the IEEE 14th International Conference on Control and Automation (ICCA), Anchorage, USA, pp. 680–685. Ehmann, D., Wittenberg, C. (2018): Erweiterte Anzeige- und Bedienkonzepte auf

Basis der virtuellen Realität für 6-Achsige Robotersysteme unterschiedlicher Her- steller. Tagungsband AALE 2018, VDE VERLAG GMBH, Berlin, pp. 355–358. John, KH., Tiegelkamp, M. (2009): Die Programmiersprachen der IEC 61131-3. In: SPS-Programmierung mit IEC 61131–3. VDI-Buch(). Springer, Berlin, Heidelberg. Lange, J., Burke, T. J., Iwanitz, F. (2010). OPC: von Data Access bis Unified

Architecture. Deutschland: VDE-Verlag. Mahnke, W., Leitner, S., Damm, M. (2009). OPC Unified Architecture. Deutschland: Springer Berlin Heidelberg. Pfrommer, J., Usländer, T. (2020). Open-Source Implementierung von OPC UA

PubSub für echtzeitfähige Kommunikation mit Time-Sensitive Networking. In: Jasperneite, J., Lohweg, V. (eds) Kommunikation und Bildverarbeitung in der Automation. Technologien für die intelligente Automation, vol 12. Springer Vieweg, Berlin, Heidelberg. Rempel, W., Bauer, B., Stache, N. & Wittenberg, C. (2019 (1)). First steps to control a digitalized factory via Augmented Reality. 14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems, 52, 1–6. https://doi.org/10. 1016/j.ifacol.2019.12.072 Rempel, W., Bauer, B. & Wittenberg, C. (2019 (2)). Der Einsatz von Augmented

Reality in der Industrie 4. 0 am Beispiel einer Modellfabrik. Tagungsband AALE 2019: Autonome und intelligente Systeme in der Automatisierungstechnik. VDE Verlag GmbH. Berlin. 395–401. Schloer N., Bauer B., Wittemberg C. (2022) Human-Robot-Interaction via AR: First Steps of Building a Human-Robot Interface on a Microsoft HoloLens. In: Ahram T., Taiar R. (eds) Human Interaction, Emerging Technologies and Future Systems V. IHIET 2021. Lecture Notes in Networks and Systems, vol 319. Springer, Cham. https://doi.org/10.1007/978-3-030-85540-6_104

98 Wittenberg and Bauer

Wittenberg C., Bauer B., Stache N. (2020). A Smart Factory in a Laboratory Size for Developing and Testing Innovative Human-Machine Interaction Concepts. In: Ahram T., Falcão C. (eds) Advances in Usability and User Experience. AHFE 2019. Advances in Intelligent Systems and Computing, vol 972. Springer, Cham. https: //doi.org/10.1007/978-3-030-19135-1_16 Zieringer, C., Bauer, B. & Wittenberg, C. (2019). Erstellung eines virtuellen Zwillings

mit OPC UA und der Unreal Engine 4. Tagungsband AALE 2019: Autonome und intelligente Systeme in der Automatisierungstechnik. VDE Verlag GmbH. Berlin. 347–351. Zieringer C., Bauer B., Stache N.C., Wittenberg C. (2020). Human-Robot Interaction via a Virtual Twin and OPC UA. In: Zallio M. (eds) Advances in Human Factors in Robots, Drones and Unmanned Systems. AHFE 2020. Advances in Intelligent Systems and Computing, vol 1210. Springer, Cham. https://doi.org/10.1007/978- 3-030-51758-8_14

Cover of the proceedings volume Human Factors in Robots, Drones and Unmanned Systems

Explore the complete digital volume

Human Factors in Robots, Drones and Unmanned Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-33-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →