Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Evaluation of the Usability and Intuitiveness of a Teleoperated Forklift

Technical University of Munich, Chair of Materials Handling, Material Flow, Logistics, Boltzmannstr. 15, 85748 Garching, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1005417

Abstract

Forklifts are still one of the most popular transport vehicles in intralogistics. However, the number of inexperienced employees increases. Due to innovations in technologies like steer-by-wire and data transfer via Wi-Fi or 5G, teleoperated machines are constantly evolving. This possibility can solve many challenges in intralogistics. Still, there is also a risk that teleoperation reduces intuitiveness due to the lack of feedback from the machine's status, such as occurring accelerations. The paper presents a study to compare a teleoperation concept with a conventional forklift control concept. In the present study, no difference in perceived usability, stress load and intuitiveness could be determined. Both control concepts were rated with an SUS value of 80 for the conventional control and 74 for the teleoperated control in the range of "good" to "excellent". The raw NASA TLX and QUESI values showed promising results for both concepts.

INTRODUCTION

Because of their flexibility, forklifts are one of the most popular transport vehicles in intralogistics. However, the number of inexperienced employees increases due to the need for more skilled and flexible allocated personnel in the logistics sector. In a survey of logistics companies, 75% of respondents said they struggle filling vacancies adequately (Voss, 2015). In practice, the lack of knowledge in handling forklifts raises a high safety risk and leads to lower handling performance (Norman, 1983). Due to innovations in tech- nologies like steer-by-wire and data transfer via Wi-Fi or 5G, teleoperated machines are constantly evolving (Fernride, 2024; Hagen et al., 2014). This possibility can solve a multitude of challenges in intralogistics. On the one hand, using teleoperated forklifts can promote working remotely and open a new labour market. A more attractive workplace can be offered by sep- arating the workplace from the original environment, for example, deep freeze warehouses. On the other hand, freedom in the workplace design can increase intuitiveness due to the possibility of adapting the operating concept to the individual previous experience. It can thus improve the learnability of the system. But there is also a risk that teleoperation reduces intuitive- ness due to the lack of feedback from the machine’s status, such as occurring

© 2024. Published by AHFE Open Access. All rights reserved. 78

Evaluation of the Usability and Intuitiveness of a Teleoperated Forklift 79

accelerations. However, studies currently need to investigate the design of teleoperated forklifts compared with conventional forklifts.

BACKGROUND

State of the Art in Forklift Controls The current state-of-the-art forklift controls consist of manufacturer-specific concepts. In addition to the conventional controls through separate hand or finger levers for each movement of the mast (Figure 1 – two leftmost photos), new controls are being developed and combine several movements of the mast via one or two joysticks (Figure 1 – two rightmost photos).

Figure 1: State-of-the-art controls for forklift masts (Jungheinrich 2024, Crown 2024).
Original PDF Figure 1:

The forklift’s control system for driving consists in most cases of a steering wheel, an accelerator pedal, and a brake pedal. The forward-reverse function can be operated either through a hand-controlled lever or with two pedals, each for forward or reverse movement. The development of intuitive control systems is currently part of research (Feiner and Fottner, 2020; Feiner et al., 2022).

Teleoperated Forklift Advanced technological capabilities in the fields of cameras, communica- tion technologies, and AR solutions enabled the teleoperation of industrial machines. Teleoperation solutions are already in practice, including applica- tions in the medical field (Hagen et al., 2014) and implementations in inner company factory truck transportation (Fernride, 2024). However, challenges persist. One significant problem is the need for more spatial awareness for the machine. Additionally, teleoperation solutions relying on cameras may lead to a loss of depth perception (Maruhn et al., 2019). There’s also a lack of a tactile connection to the load and speed (Chew et al., 2021). Another aspect is mode awareness and situation awareness, which determines the necessary information to understand the teleoperated system’s operational mode and monitor the machine situation consistently and accurately. The challenge is finding the most effective way to present this information (Nitsch et al., 2013; Johansson and Vin, 2009).

One possible technical solution is presented by enabl Technologies GmbH (Karlsruhe, Germany, Figure 2). They present a teleoperated control sys- tem for a forklift containing seven cameras (1920 × 1200 pixels) posi- tioned to enable a panoramic view, ensuring situational awareness (Figure 3).

Positioned at the top and center of the forklift, the front-facing camera posi- tion is at 300 cm height. Another two front-facing cameras (fork top left and right) are mounted on the fork carriage, which is 120 cm above the fork tines and has a lateral offset of 50 cm from the center. This cameras move in tandem with the lifting mechanism.

Figure 2: Teleoperated forklift (left) and the teleoperations workplace (right).
Original PDF Figure 2:

Additionally, the system includes a camera and a red laser positioned in the center of the forklift at the height of the forks (fork laser and camera). The laser projects a line to visualize the height of the forks. For lateral visibility, side-facing cameras are installed at a height of 185 cm above the ground, corresponding to the driver’s position. Completing the setup, a rear-facing camera stands at the height of 175 cm above the ground, providing a centered view along the vehicle axis.

Figure 3: Positions of the seven cameras and the LASER on the teleoperated forklift.
Original PDF Figure 3:

The teleoperation workstation has a Logitech G920 Driving Force Gaming steering wheel, a Logitech Extreme 3D Pro Joystick, and a 49-inch monitor with a 3,840 × 1,080 pixels resolution and a 32:9 aspect ratio. The Log- itech G920 Driving Force Gaming steering wheel offers control with haptic feedback that increases resistance as speed rises, requiring the operator to exert more force. This dynamic feature enhances realism, providing intuitive signals for speed changes during teleoperation. The connection between the forklift and the teleoperation workstation is established using Wi-Fi.

Evaluation of the Usability and Intuitiveness of a Teleoperated Forklift 81

During operation, the transition between the “forward” and “reverse” driving modes can be executed through a switch on the joystick. (Figure 4) The “handling” mode, activated by pressing a button on the joystick, over- lays the front camera with the image from the camera positioned between the fork tines (Figure 6). In the “forward” mode, the camera stream show left and right like sitting in the forklift. Upon switching to the “reverse” mode, the images are inverted, facilitating seamless backward navigation without the driver needing reorientation. Further, the image from the top camera is replaced by the image from the back camera. (Figure 5) Addition- ally, augmented reality (AR) elements are superimposed to enhance situation awareness. These consist of distance lines at intervals of 0.5 m, 1.0 m, and 2.0 m from the forklift, aiding in a more accurate assessment of distances to surrounding objects.

Figure 4: Screenshots of the teleoperation workplace in “forward” driving mode.
Original PDF Figure 4:
Figure 5: Screenshots of the teleoperation workplace in “reverse” driving mode.
Original PDF Figure 5:
Figure 6: Screenshots of the teleoperation workplace in “handling” mode.
Original PDF Figure 6:

STUDY DESIGN Driving and Lifting Task

All participants carried out the following task after an introduction to the test procedure, safety regulations, and the function of the operating elements. The task consisted of driving and load-handling parts. A sketch and a process description of the task are shown in Figure 7. After performing the task with an operating concept, the test subjects completed questionnaires and then repeated the task with the other operating concept. The order of the operating concepts between the subjects was permuted. During the task, the execution times were measured by the test supervisor.

Figure 7: Sketch of the experimental setup including the task description.
Original PDF Figure 7:

In addition, questionnaires were used to record the perceived usability (System Usability Scale, SUS) and to record the intuitiveness of use (NASA Task Load Index, NASA-TLX, and Questionnaire for the Subjective Conse- quences of Intuitive Use, QUESI). The SUS consists of ten statements to which users express their level of agreement on a five-point scale. The evaluation of the SUS results in a numerical value between 0 and 100, with 100 being the best possible score (Brooke 1996; Bangor et al., 2009). The NASA-TLX comprises six dimensions: mental demand, physical demand, time demand, performance, effort, and frustration. The dimensions were not weighted in the present study, and the subscale values from 1 to 10 were averaged to form the total value. A higher value corresponds to a higher stress level and, there- fore, a poorer rating (Hart and Staveland, 1988). The QUESI determines the subjective consequences of intuitive use. These include the subjective men- tal strain, the perceived goal achievement, the perceived learning effort, the familiarity, and the perceived error rate. An overall value from 1 to 5 can be formed for intuitiveness by averaging the subscale values. The higher the QUESI value obtained, the higher the system’s intuitiveness (Naumann and Hurtienne, 2010).

Evaluation of the Usability and Intuitiveness of a Teleoperated Forklift 83

Hypothesis and Survey It was hypothesized that interaction with the newly developed teleoperating concept would be more usable and intuitive than the conventional operat- ing concept. This was initially examined across all test subjects, regardless of expertise. Furthermore, the differences in the evaluation of the groups according to previous experience and within the group of novices and experts were examined. The usability evaluation consisted of the dimensions of the required task time and the SUS value. The evaluation of intuitiveness was derived from the NASA-TLX and the QUESI value.

The study was conducted using a Jungheinrich EFG220. It features con- ventional controls, including a joystick for mast control, a steering wheel, and pedals for the accelerator and the brakes. A switch on the joystick controls forward and reverse movements.

A total of 34 test subjects took part, primarily conducted among employees and students at the Technical University of Munich who were at least 18 years old. The group of test subjects consists of 27 men and 7 women. On average, the subjects are 29.6 years old (SD 6.78 years). Among the test subjects, 12 participants possessed a forklift driver’s license and were therefore classified as experts. 22 test subjects had no experience operating a forklift before and, therefore, formed the group of novices.

RESULTS AND DISCUSSION

Comparison of the Usability The evaluation of the mean values of the subjectively perceived usability of the tested controls resulted in a SUS value of 74 for the teleoperated control, corresponding to a rating in the “good” to “excellent” range (Bangor et al., 2009) (Figure 8). The conventional control system achieved an average of 80 in the “excellent” range. The planned contrasts showed no significant dif- ference in SUS values between the conventional and the teleoperated control across all test subjects. Furthermore, the ANOVA showed no significant main effect of expertise. Novices and experts, therefore, remained similar in their SUS values regardless of the operating concept.

Figure 8: Results of SUS, raw NASA TLX and QUESI questionnaire as clustered boxplots.
Original PDF Figure 8:

The task execution times for the conventional system averaged 431 sec- onds, while the teleoperated system showed an average execution time of 717 seconds. For the group of novices, there was less increase in execution times, from 504 seconds to 750 seconds, compared to the expert group, where the time increased from 298 seconds to 658 seconds. This difference can be attributed to the pre-existing experience of the expert group with the con- ventional system. The overall increase in execution time for the teleoperated system is primarily attributed to the prototypical implementation of the tele- operation system. Consequently, further analysis of the execution time is not pursued in this context. In summary, the original hypothesis that the teleop- erated system is more suitable for use than the conventional system could not be confirmed and must be rejected. Conversely, however, it can be shown that the newly developed concept has a similar usability to a system that has already been tried and tested over decades.

Comparison of the Intuitiveness In the evaluation of the intuitiveness of the system, both the NASA-TLX and the QUESI values show comparable results. There was no significant difference in the NASA-TLX values in the planned contrasts between the conventional and the teleoperated concepts. On average, the conventional concept achieved a rating of 3.4 in the Raw NASA TLX and, thus, a slightly better rating than the teleoperated concept with 4.2. The pairwise post-hoc comparisons revealed no significant difference between the concepts in the NASA TLX values for the novices and the experts. This trend was confirmed in the evaluations of the QUESI questionnaires. The conventional concept was rated slightly better, with a mean score of 3.8, than the teleoperated con- cept, with 3.6. The hypothesis that the teleoperated operation of the forklift increases intuitiveness could not be confirmed.

Examining the subscales of the Raw NASA TLX for both systems across all participants, it is noticeable that the frustration subscale has increased the most by approximately 35%. One reason could be technical difficulties during the test, which are delays in camera images and commands from the controls to the forklift. Furthermore, a lower assessment of intuitiveness by experts does not appear unexpected. This is because the evaluation of intu- itiveness heavily depends on previous experience, which, for the experts, is characterised by the conventional system (Blacker, 2019).

CONCLUSION

A teleoperated operating concept was designed to provide high intuitiveness for inexperienced and experienced forklift operators and the paper introduces a study comparing teleoperated and conventional control of a forklift. In this study, 34 participants performed a representative task and completed the SUS, raw NASA TLX and QUESI questionnaires. The results demonstrated a high perceived usability and intuitiveness of the interaction, with no signif- icant difference found between the two control methods. Experts rated the usability and intuitiveness of the teleoperated concept slightly higher than the

Evaluation of the Usability and Intuitiveness of a Teleoperated Forklift 85

conventional operating concepts. The results imply that the experts’ subjec- tive evaluation was influenced by their familiarity with traditional concepts. Challenges in learning the teleoperated system were identified due to the lack of multimodal driving experiences. Moreover, the study involved only one execution of the driving task, making it challenging to capture learning effects accurately. A long-term study involving full-time forklift drivers is suggested to address these issues.

Further system development could involve expanding AR elements or achieving a panoramic view by seamlessly merging camera images into a bird’s-eye perspective, enhancing situational awareness. Additionally, con- veying more vehicle conditions, such as vibrations or load weights, haptically to the driver should be considered. In addition, an in-depth investigation into ergonomic design and long-term effects is necessary.

REFERENCES

Bangor, A.; Kortum, P.; Miller, J. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. In: Journal of Usability Studies, Jg. 4, Nr. 3, page 114–123. Blackler, A. (2019). Intuitive Interaction. CRC Press, Miami. Brooke, J. (1996). SUS: A ‘Quick and Dirty’ Usability Scale. In: Jordan, P. W.,

et al. (Hrsg.): Usability Evaluation in Industry. Taylor & Francis, London, page 189–194. Chew, J. Y., M. Kawamoto, T. Okuma, E. Yoshida and N. Kato (2021). Adaptive attention-based human machine interface system for teleoperation of industrial vehicle. Scientific reports 11 (1), page 17284. Crown Gabelstapler GmbH & Co. KG (2024). Stapler der FC Serie, Die FC Stapler

bieten Produktivität mit jeder Bewegung; https://www.crown.com/de-de/gabelsta pler/ elektro-gabelstapler-fc.html, Access 23.01.2024. Feiner, L.; Fottner, J. (2020). Design of an Intuitive Control Concept for Lifting Operations Using the Example of Forklifts. In: Rebelo, F.; Soares, M. (Hrsg.): Advances in Ergonomics in Design. Springer International Publishing, Cham, page 309–316. Feiner, L.; Geisel, V.; Fottner, J. (2022). Intuitives Bedienkonzept für Gabelstapler.

Zeitschrift für wirtschaftlichen Fabrikbetrieb (zwf), 117 (6), page 390–394. Fernride (2024). Autonomous, electric yard-trucking, https://www.fernride.com/, accessed 23.01.2024. Hagen, M. E. and Curet, M. J.: The da Vinci Surgical System (2014). In Robotic

Surgery, Wantabe Go (Eds.), page 9–19, Springer Japan. Hart, S. G.; Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock, P. A.; Meshkati, N. (Hrsg.): Human Mental Workload. Elsevier, page 139–183. Johansson, D., and de Vin, L. J. (2009). Omnidirectional robotic telepresence through

augmented virtuality for increased situation awareness in hazardous environ- ments. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, page 6–11. Jungheinrich AG (2024). Elektro-Gabelstapler, Die perfekte Verbindung von Leistung und Effizienz: https://media-live2.prod.scw.jungheinrichcloud.com/r esource/blob/112615 6/67720641cc541aed48e584ed08ee21a0/efg-brochure-de- elektrostapler-pdf-data.pdf, Access 23.01.2024.

Maruhn P, Schneider S, Bengler K (2019). Measuring egocentric distance perception in virtual reality: Influence of methodologies, locomotion and translation gains. PLOS ONE 14(10): e0224651. https://doi.org/10.1371/journal.pone.0224651 Naumann, A.; Hurtienne, J. (2010). Benchmarks for Intuitive Interaction with

Mobile Devices. In: Association for Computing Machinery (Hrsg.): Mobile HCI ‘10: Proceedings of the 12th international conference on Human computer interaction with mobile devices and services, page 401–402. Nitsch, Verena & Faerber, Berthold (2013). A Meta-Analysis of the Effects of Haptic Interfaces on Task Performance with Teleoperation Systems. IEEE Transactions on Haptics. 6, page 387–398. 10.1109/TOH.2012.62. Norman, D. A. (1983). Design rules based on analyses of human error. Communica-

tions of the ACM. 4, page 254–258. Voss, P. (2015). Logistik – eine Industrie, die (sich) bewegt. Springer, Wiesbaden.

Cover of the proceedings volume Usability and User Experience

Explore the complete digital volume

Usability and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-32-8

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →