Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Combining AR and ball-shape input interface to control remotely a robot-arm

1 Aoyama Gakuin University, Tokyo, Japan ↗
2 Turku University of Applied Sciences, Turku, Finland ↗
Publication year
Digital Object Identifier10.54941/ahfe1004628

Abstract

Robot arm technology has revolutionized the manufacturing industry, offering worker safety and productivity benefits. However, issues like unintended actions and difficulty in intuitive operation remain. This study proposes a remote-control system for robot arms that combines augmented reality (AR) and a multi-sensor interactive smart ball (PALL0) from Ai2Ai Corp. The AR application allows operators to synchronize the control of a virtual and physical robot arm using their hand and the PALL0 held in the hand.The AR application operates the robot arm by matching the position of its head with the operator's hand and the posture of the head with the acceleration and angular velocity of the PALL0. A button on the PALL0 controls the opening and closing of the gripper, while the physical robot arm also operates. The start and end of the operation, as well as the return of the robot arm to its initial posture, can be performed using buttons in the AR application.Future research should compare the method of controlling robot arms using PALL0 with other methods, such as those that use the movement of the body's upper or lower extremities, and invest resources in building the robot's digital twin to improve system portability.

INTRODUCTION

In recent years, improvements in robot arm technology have revolutionized the manufacturing industry and many other industries. Introducing robot arms in manufacturing and other industries offers various advantages, such as increased productivity and worker safety. As examples of introduction, robots that work in the same space as the worker while cooperating with the robot, which is said to be called a co-worker robot (Omron Indus- trial Automation), and that perform tasks such as assembling and receiving parts (Yaskawa Electric), are being introduced. Although robot arm tasks are becoming increasingly automated, some tasks are too complex to auto- mate completely, and such tasks need to be controlled by humans. Control techniques utilizing muscle potentials (Masafumi et al., 1995) and leg move- ments (Saraiji et al., 2018) have been proposed. However, certain challenges remain unresolved, including non-intuitive manipulation and the potential

© 2024. Published by AHFE Open Access. All rights reserved. 200

Combining AR and Ball-Shape Input Interface to Control Remotely a Robot-Arm 201

for unintended motions. This system makes it possible to control a physi- cal robot arm by controlling the robot arm in the application using an AR application, hand, and hand-held PALLO—control by matching the posture calculated using acceleration and angular velocity. The initiation and termi- nation of the operation, as well as the restoration of the robot arm to its original position, can be achieved through buttons within the augmented reality (AR) application.

PREVIOUS STUDIES Robotic Arm Using Virtual Reality Controlled Remotely

In (Blanco, 2021), we introduced a system which is using virtual reality to monitor a robotic arm’s (Braccio Tinkerkit) behavior and to fully control if remotely.

This remote control was accomplished thanks to a WebSocket server with authentication designed with the idea of accepting multiple controlled robots using VR. The user was also able to watch a live stream (on a custom Nginx server) of the movements of the real robotic arm, giving them visual feedback of the performed commands.

What made this project special is that it encompassed a wide range of tech- nologies, being the first of its kind by bringing the metaverse and robotics together. It managed to combine a virtual environment with an embed- ded system capable of affecting the real world (picking up objects) through commands managed by a server with database and authentication.

This study provided for us information and documentation about the required protocol support and data flow requirements for integrating further embedded system solutions.

Figure 1: Block diagram of the whole implemented system and the technologies/pro- tocols used on the Tinkerkit robot arm project.
Original PDF Figure 1:

Mobile Robot and Digital Twin Visualization in Metaverse The industrial metaverse environment developed utilizing ProVerse Interac- tive’s metaverse platform contained our laboratory, which was scanned by utilizing a laser scanner to enable a matching virtual environment for the tank robot. This metaverse platform has originally been developed at Turku University of Applied Sciences introduced in Luimula et al. (2022) where also metaverse was defined as a technology enabling social communication, hands-on-training, and digital twin integrations in an immersive three dimen- sional environment. An accelerometer was used to update the position of the virtual robot according to the movements of the physical robot. In addi- tion, massively multi-user operations were demonstrated at ERF 2023 by implementing a movement system through a web interface which allowed the audience to participate in the demonstration. This implementation is reported in detail in Kaarlela et al. (2023).

The aim was to create a self-contained system for teleoperated robots by using VR and smart gloves as the user interface based on the experiences collected from the Tinkerkit prototype. In this study, we focused on visual- izations by visualizing a digital twin of the physical robot (XiaoR Geek tank robot) to increase a realistic user experience.

Figure 2: Remote controlled robot environment was introduced in the European Robotics Forum (Kaarlela et al., 2023).
Original PDF Figure 2:

Collaborative Metaverse Solution to Control Cobots Remotely Just lately, we have focused on Universal UR5 cobot. This robot is again remotely operated with VR headsets and controllers. It is visualized again as a digital twin and real-time video information for the users. Our objective is to showcase opportunities of industrial metaverse emphasizing collabo- rative elements, seamlessly combining the phenomena of digital and real worlds.

Combining AR and Ball-Shape Input Interface to Control Remotely a Robot-Arm 203

In addition to that, artificial vision is used to identify and pick up wooden blocks so that they can then be moved and placed in the desired position by the user, demonstrating both automation and human operation.

Operating in the environment requires an access token that grants right to robot remote control for the operator. In practice, a collaborative con- struction task will be carried out with the robot, where users represented by avatars can take turns participating in the construction project independently of their real-world physical location. In other words, the solution enables work shifts to continue without interruption, regardless of time and place.

Figure 3: The remotely controlled cobot collaborative system.
Original PDF Figure 3:

THREE WAYS TO CONTROL COBOT IN XR Robot Arm Control Through Button UI

Controlling the robot arm using the button UI is similar to the conventional method. The user can place buttons to control the position and rotation of each XYZ axe, and the user can operate the robot arm using those buttons. The user can also operate the opening and closing of the gripper using the buttons.

Remote-Control System for Robot Arms That Combines AR and PALL0 Ai2Ai Version PALL0 by Ai2Ai Oy (see figure 4) represents a leap in interactive technol- ogy, merging the physicality of a ball with advanced digital sensors and multimodal feedback. This device, designed primarily for fostering physical activity and rehabilitation, has now found application in industrial control systems. Equipped with sensors that track acceleration, angular velocity, and external environmental factors, PALL0 offers a unique, intuitive interface for various applications.

Figure 4: PALL0 by Ai2Ai Oy.
Original PDF Figure 4:

System Overview We propose a system to remotely control a robot arm by combining aug- mented reality (AR) using a transparent head-mounted display and Ai2Ai’s multi-sensor interactive smart ball (PALLO). The robot arm is capable of synchronizing its movements with a virtual robot arm, allowing for remote control of the robot arm.

Figure 5 depicts the schematic of the virtual robot arm control technique. The virtual robot arm is controlled by aligning the position of the hand with the position of the robot arm’s head, and matching the PALLO posture with the posture of the robot arm’s head. The angles of each robot arm joint are calculated from the position and orientation of the head using inverse kine- matics. The buttons on PALLO are used to control the opening and closing of the gripper. The hand position is acquired using the hand tracking function of the transparent head-mounted display and visualized in the AR application as a butt cube. The AR application controls the virtual robot arm using the PALLO posture information received from the console application and the hand position acquired in the application.

Figure 5: Virtual robot arm control method diagram.
Original PDF Figure 5:

Figure 6 shows the AR application before the control arm robot. After the AR application is launched, the start, reset, and menu buttons are displayed. The start and reset buttons disappear by pressing the start button, and an

Combining AR and Ball-Shape Input Interface to Control Remotely a Robot-Arm 205

orange cube along with instructional text appears as the initial hand posi- tion (see Figure 7). The robot arm can be controlled by superimposing the hand position on the orange cube. The menu button is always displayed, and by pressing the menu button at any time during control, the control of the robot arm can be temporarily interrupted. The start and reset buttons can be displayed. The robot arm can be returned to its initial posture by press- ing the reset button. This prevents the robot arm from making unintended movements.

Figure 6: AR application before controlling robot arm starts.
Original PDF Figure 6:
Figure 7: AR application after pressing the start button.
Original PDF Figure 7:

Cobot Control Through XR Controllers, Digital Twin and Virtual Representation of Robot Pendant As a next natural step of the system mentioned above, PALL0 interaction methods will be replaced by Hololens controllers. In both of these versions, cobot’s digital twin visualization augmented on the top of the real cobot will enable user interaction by selecting joints to be moved.

The user interacts with the UR5 digital twin using the XR controllers and can see the real movements of the real robot reflected in it; that way it is easy to notice any unexpected behavior and the control can be more intuitive.

For the future, as a third version, we are planning as well to simulate a virtual accurate model of the UR5 pendant, so that the user can perform the same actions that are able to be performed in the real system, such as controlling and programming the cobot. The interaction with the pendant will be specially intuitive using the HoloLens AR.

Figure 8: Cobots programmed utilizing digital twin visualization of the UR-5 handheld control device (top left). cobot control using XR controllers and digital twin (bottom left). Ai2Ai PALL0 to be used in the upcoming experiment (right).
Original PDF Figure 8:

CONCLUSION

In this paper, we propose a remote control system for a robot arm that combines augmented reality and PALL0. In the proposed system, the AR application controls the robot arm by matching the position of the robot arm’s head, the position of the operator’s hand, and the posture calculated from the acceleration and angular velocity of the head and PALLO. This prevents unintended operation and at the same time enables intuitive opera- tion. In the future, we will verify the effectiveness of the proposed system by conducting user evaluation experiments and comparing each method.

ACKNOWLEDGMENT

The authors would like to thank everyone who have participated in thede- velopment of the preliminary, prototyping, and pilot studies. This work

Combining AR and Ball-Shape Input Interface to Control Remotely a Robot-Arm 207

was supported by Business Finland and Finnish Ministry of Education and Culture. This work was partly supported by a grant from the Center for Advanced Information technology Research, Aoyama Gakuin University.

REFERENCES

V. Blanco Bataller. (2021) “Remote Control of a Robotic Arm Using Virtual RealityVirtual”. Bachelor’s Thesis, Universitat Politècnica de València, Valencia, Spain, 2021. Available online: https://riunet.upv.es/handle/10251/176778 (accessed on 17.04.2023). Honour, Eric C. (2006) “A Practical Program of Research to Measure Systems Engi-

neering Return on Investment (SE-ROI)”, proceedings of the Sixteenth Annual Symposium of the International Council on Systems Engineering, Orlando, FL. Kaarlela, T., Pitkäaho, T., Pieskä, S., Padrão, P., Bobadilla, L., Tikanmäki, M., Haavisto, T., Blanco Bataller, V., Laivuori, N., and Luimula, M. Towards Meta- verse: Utilizing Extended Reality and Digital Twins to Control Robotic Systems. Actuators, Vol12(6), Article 219, 2023, 20p. Luimula, M., Haavisto, T., Vu, D., Markopoulos, P., Aho, J., Markopoulos, E., and

Saarinen, J. The Use of Metaverse in Maritime Sector – A Combination of Social Communication, Hands on Experiencing and Digital Twins, In: Proceedings of the International Conference on Creativity, Innovation and Entrepreneurship, Vol. 31, 24–28, July, 2022, New York, US, pp. 115–123. Meilich, Abe. (2008) INCOSE MBSE Initiative Status of HSI/MBSE Activity (Presentation). Omron Industrial Automation: Features of Collaborative Robots, https:// www.fa.omron.co.jp/product/robotics/lineup/collaborative/feature/ (accessed on 04.04.2024). M. Y. Saraiji, T. Sasaki, K. Kunze, K. Minamizawa, and M. Inami. (2018)

“MetaArms: Body Remapping Using Feet Controlled Artificial Arms”. Pro- ceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18), pp. 65–74 (Oct. 2018). Taubman, Philip. (June 25, 2008) Top Engineers Shun Military; Con- cern Grow. The New York Times Website: http://www.nytimes.com/ 2008/06/25/us/25engineer.html. Uchida, Masafumi. Ide, Hideto and Yokoyama, Shuichi. (1995). Control of a Robot Arm by Myoelectricity. Transactions of the Institute of Electrical Engineers of Japan, C (Journal of Electronics, Information and Systems) 115(3): 445–451. Yaskawa Electric: What is a cooperative robot?, https://www.yaskawa.co.jp/produc

t/robotics/collaborative (accessed on 04.04.2024).

Cover of the proceedings volume Accessibility, Assistive Technology and Digital Environments

Explore the complete digital volume

Accessibility, Assistive Technology and Digital Environments

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-97-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →