Felix Dreger, Lewis L Chuang, Gerhard Rinkenauer (2022). Learning to Master Robotic Arm Movements with Bimanual Joystick Control: Indicators for Evaluating the Difficulty of Movement Tasks. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 57, pp. 114–121. https://doi.org/10.54941/ahfe1002318
Manual control of robotic arms is challenging and productive operators require extensive prior training. Effective training should systematically vary the difficulty level of the robot arm motions. This study investigates the extent to which Fitts’ law could define movement difficulty for bimanual controlled movements of robotic arms. Inspired by forestry work-methods, we designed Fitts’ tapping Task to assess the movement time and throughput of ten unskilled participants over nine training sessions. We found that robotic arm movements observe Fitts’ law for reaching in depth but deviate for lateral and concentric movements. In other words, training can utilize Fitts’ law to vary the difficulty of forward robot arm movements. Further studies on the difficulty of lateral and concentric movements are necessary to refine work methods and improve training.
To control articulated robotic arms, such as forest harvesters, operators must learn how to simultaneously transform multiple independent degrees-of- freedom. Extensive simulator training and field training is currently necessary before operators are sufficiently skilled to operate safely and independently. Such training is costly and could be more efficient.
To begin, not all movements are equally difficult not likely to be required. Although operators are often expected to exert nimble and flexible control of robotic arms to actuate a diverse repertoire of manoeuvres and cope with all exigencies. On the other hand, some movements are typically more frequent than others, especially when the operational context is taken into conside- ration (Ranta, 2009). In forestry, the frequency of the robot arm (crane) movements result from well-specified work requirements. For instance, in clear felling, trees are required to be felled in the direction of travel (i.e., machine trail) and produced logs should be placed alongside the machine
Learning to Master Robotic Arm Movements with Bimanual Joystick Control: Indicators 115
trail. The most productive methods are ‘Two-Sided Piling’ and Forward Fel- ling’ in clear cuts (Ovaskainen et al., 2011). Further, a narrow fell swath in the direction of travel increases productivity (Nordberg, 1987). Superfluous movements and trajectories, though possible, render operations inefficient (Ovaskainen et al., 2004). Currently, machine operators in training are only exposed to a variety of situations that place demands on the complexity of crane movements (e.g., number of segments served simultaneously, Wickens et al., 2013). Instead, tasks should be presented that scale with the growing aptitude of the trainee. However, it is unclear how movement difficulty ought to be defined and operationalized.
In human aiming movements, movement difficulty can be systematically varied using Fitts’ tasks. The current work is motivated to situate the dif- ficulty of moving robotic arms within a such a framework or one that is comparable. The movement difficulty of Fitts tasks are operationalized in terms of the information capacity of the human motor system (Fitts, 1954). Typically, subjects perform repetitive aiming movements to tap alternatin- gly between two targets. The information capacity required to perform this repetitive movement can be defined in terms of the size of the targets and the distance between them (Soukoreff and MacKenzie, 2004). The task is frequently amended for specific applications i.e., to evaluate performance of varying (tele-operated) robotic arm movements (Draper et al., 1990; Jung et al., 2013; Suzuki and Harashima, 2008). However, Fitts’ task was origi- nally intended to study human aiming movements and not the movement of robot arms.
In summary, the extent that Fitts’ law can be applied to human controlled robotic arm motions in the forest context is unclear. Operator must move their arms to operate bimanual controls that actuate robotic arm movement. However, the amplitude of these manual movements do not correspond dire- ctly to the actuated robot arm movement—robot arm movement is indirectly controlled by the angular velocities of the robot joints. Thus, operators must learn a nonlinear transformation to map their own motor actions onto the robot arm. In this paper, we use Fitts’ law as a theoretical framework to manipulate the difficulty of crane motions. If Fitts’ law applies, manipulati- ons of target distance and target size according to the relationships derived by Fitts should lawfully govern performance indicators such as movement time and information throughput (Fitts, 1954). Furthermore, we are interested in the extent that learning the robot arm movements affects Fitts’ performance indicators.
METHODS
Participants and Apparatus Ten (6f, 4m) volunteers with an average age of Mage = 28 years (SDage = 9.29) learned to operate a robotic arm simulator across nine successive sessions. All participants had no prior experience in the task and robotic arm control. The robotic arm simulator consisted of a Grammar seat (Chicago 1040673-C)
with two mounted joysticks (Thrustmaster T.16000M FCS) and a Sam- sung 55” TV Screen (Samsung LE40C750R2Z). The simulation software comprised C++, ROS, and GAZEBO.
Robotic Arm Simulation and Stimuli The simulation provided a manipulator (open manipulator) with four arti- culated joints and an end effector (gripper) (ROBOTIS Inc., Korea). The manipulator had 4 degrees of freedom and was simulated in GAZEBO. Par- ticipants were presented with a tilted view, centred on the manipulator on a gridded ground plane (Fig. 1).
Figure 1: Displayed is the simulator setup (a), the Targets in purple (b) and blue (c) colour are shown from the perspective of the participants.
Movement targets were displayed as blue and purple circle pairs (W = 0.5cm) on the ground plane. Blue indicated that tapping should begin on the left side and purple, the right side. There was a visible circle on each side of the robotic arm. The distances between them and their locations were selected such that the movement patterns (e.g. movement direction) were comparable with contemporary working methods (i.e., two-sided and forw- ard felling; c.f. Ovaskainen et al., 2011). These diagonal patterns (Fig. 2) varied in amplitude (D, spacing) and required participants to exploit all possible DOFs of the robotic arm.
Figure 2: Shown are the predefined movement trajectories (a, start left; b, start right).
Learning to Master Robotic Arm Movements with Bimanual Joystick Control: Indicators 117
Fitts’ law assumes that motor performance can be described by the math- ematical concept of “information”. The amplitudes of aiming movements are considered analogous to the information of transmitted signals, and the spatial accuracy (variability) of movements is considered analogous to system noise. Since Fitts’ law treats the human motor system as a commu- nication channel, the bandwidth (or information throughput) of movement can be expressed in bits/s. Thus, the difficulty of a motor task can be quan- tified by the difficulty index (ID in bits), which is the logarithmic ratio of target distance (D) and target width (W), ID = log2 (D/W + 1). Fitts’ law predicts that movement time (MT) increases linearly with the difficulty index ID, MT = a + b ID. The effective difficulty index IDe = log2 (D/We + 1) achieved is used to calculate information throu- ghput TP = (IDe/MT) in [bits/s]. Here, the effective distance We, defined by the variability of the executed aiming movements, is taken into account ( e.g., MacKenzie, 1992). The current study kept target width constant (W = 0,5cm) and target distances between circles (D) where selective to derive a effective ID range (ID =2.49, 3.09, 3.35, 3.92). Balanced for star- ting position, this resulted in eight target combinations (for further details see Soukoreff & MacKenzie, 2004).
Procedure Participants first provided signed consent to written instructions. Next, the participants sat in the simulator and received task instructions, followed by a short training of four movements between targets (W = 1.5cm). Next, the task began, and participants tapped each target ten times, returning nine recorded movements for each randomized target within one experimental Block. A session consisted of six Blocks. Altogether, participants comple- ted nine sessions and generated 4320 movements in total. A Session lasted from 3.5h to 1.5h depending on the learner’s skill level and progression in the experimental series. After the task, participants filled a questionnaire on demographics and co-founding variables (i.e., joysticks use).
Design The study was conducted as 2 (left, right) x 4 (index of difficulty) x 6 (Block) x 9 (Session) repeated-measurement design.
RESULTS
The data was pre-processed and analyzed in MATLAB (v.2021a) and R (v.4.1.1).
Performance and Learning Task performance was operationalized as MT and throughput. According to Fitts’ law, both are expected to increase with the index of difficulty of the aiming movements. An repeated-measures Analysis of Variance (ANOVA), with main factors of ID (4 levels) and Sessions (9 levels), showed a signifi- cant effect of IDs on movement times (F(1.15, 10.39) = 38.10, p < .001, ω2p= 0.57). Post-hoc pairwise Tukey adjusted comparisons revealed that
MTs increased with difficulty from IDs 2.49 to 3.35 bits and abruptly decre- ased for ID = 3.92 bits—MTs did not significantly differ between ID = 3.92 bits (M = 8.51s, SD = 2.51s) and ID = 2.49 bits (M = 8.21s, SD = 2.89s, p = .212; see Fig. 3).
Figure 3: Movement time in seconds as a function of Fitts Index of Difficulty and experimental sessions. Dots indicate observed data and straight lines indicate linear prediction of movement time according to Fitts Law. Session is represented by color.
Two separate one-way repeated measures ANOVA were conducted to analyze the throughput for the factors of ID and sessions and revealed that throughput increases with higher IDs, (F(1.04, 9.37) = 58.39 p < .001, ω2p = 0.66) and later session (F(8, 72) = 84.70 p < .001, ω2p = 0.007) (c.f. Fig. 4).
Figure 4: Throughput in bits/s per index of difficulty shown for each session, separately.
Learning Maximum Throughput should increase with motor skill development but can be expe- cted to be limited by the maximum achievable performance level. This could provide insights into remaining gains and the efficiency of the given training
Learning to Master Robotic Arm Movements with Bimanual Joystick Control: Indicators 119
protocols. Thus, we expected throughput to increase with Sessions to the asymptotic maximum. We fitted an exponential function Vinf+αse-γ sn, where Vinf denotes performance that is the estimated maximum achievable TP, αs is the start value, γ is the rate in increase of throughput and n the num- ber of experimental sessions. The fitted exponential curves (Fig. 5) show that ID = 3.92 bits shows the highest maximum throughput with TP3.92 = 0.032 bits/s. This contrasts to the other IDs TP2.49 = 0.02 bits/s, TP3.09 = 0.021 bits/s, TP3.35 = 0.014 bits/s. Further, there was a significant interaction of ID and session (F(24, 216) = 9.13, p = .001, ω2p = 0.07)). Learning pla- teaus with more training sessions and, hence, lower gains are achieved the closer performance gets to this asymptote. Mostly, skill development is ach- ieved before expert level thus it is useful to have an approximation of when improvements are no longer significant. We performed planned Tukey adju- sted contrasts for all consecutive sessions there were no significant differences after session four (p1-2; p2-3; p3-4 < .001, p4-5 = 0.12, p5-6 = 0.84, p6-7 = 0.57, p8-9 = 1.00)).
Figure 5: Information capacity (throughput in bits/s) plotted across experimental sessions by each index of difficulty. Solid lines show fitted function.
DISCUSSION
This study sought to clarify if Fitts’ law could be applied to pre-determine the movement difficulty of robotic arm movements in forestry work methods. If true, it could inform the design of how tasks are scheduled for operator training. To this end, we evaluated how useful Fitts’ index of difficulty is in defining the progression of acquired skill.
We found that movement time and throughput of the 4 DOF robo- tic arm followed the prediction of Fitts’ law for three (ID = 2.49, 3.09, 3.35) out of four IDs independent of skill development over training. As expected, learning causes movement times to decrease and information throu- ghput to increase with sessions. Both, movement time and information throughput showed saturation effects, that allowed to estimate the required maximum training effort and revealed that the ID = 3.92 bits appeared to stand out. Interestingly, movement time and throughput of the ID = 3.92
bits, do not agree with Fitts’ predictions. Here, performance is surprisin- gly similar to ID = 2.49 bits. In other words, motor control for a task that was predicted to be difficult, was in fact, simpler to perform. Pre- liminary analyses suggest that our participants adopted a different motor control strategy for ID = 3.92 bits, which could have accounted for this non-linearity. The distance from left to right is mainly bridged by a slew- ing motion that can be realized by a single joystick axis input. In addition, the concentric movement allows for targets close to the orbit to reduce the invested joint range (overall or specific i.e., by not using the gripper). Such strategy facilitates control demand and would lead to an optimized speed compared to the lower presented IDs where movements are not close to the slewing orbit. Follow-up studies could verify this by assessing the effect of lateralization in IDs and distance to the orbit of the concentric movement.
Our study revealed that robotic arm movements’ difficulty can be descri- bed for movements in front/in depth of the manipulator by Fitts’ law. Selectively, training difficulty can thus be tailored in accordance with Fitts’ ID. Concentric movements ease motions where the actual difficulty is yet to be systematically operationalized. In regard to forestry work methods, we can conclude for working close to the front midline that short tree hand- ling distances are favorable over longer distances and larger tolerances over small tolerances in harvester head positioning. This is in line with work meth- ods that recommend working close and in front of the machine. However, long movements that consist mostly of concentric, slewing motions keep up with short tree handling times and distances. Still lateral and concentric movements need further research on their actual demand and the applica- tion of Fitts should therefore be used with caution in movement difficulty description.
ACKNOWLEDGMENT
We would like to thank Annika Schütz, Luisa Lau, Sarah Kuhlmann, Adrian Böckenmap, and Hanno Mussmann for their support. This work was funded by the ERA-NET Cofund Action “ForestValue—Innovating the forest-based bioeconomy”, by Fachagentur Nachwachsende Rohstoffe, Forskningsradet and VINNOVA. ForestValue received funding from the European Union’s Horizon 2020 research and innovation programme: agreement N° 773324. Addional funding was provided by “Instant Teaming between Humans and Production Systems co-financed by tax funds of the Saxony State Ministry of Science and Art (SMWK3-7304/35/3-2021/4819) on the basis of the budget passed by the deputies of the Saxony state parliament.”
REFERENCES
Draper, J.V., Handel, S., Hood, C.C., 1990. Fitts’ Task by Teleoperator: Movement Time, Velocity, and Acceleration. PROCEEDINGS of the HUMAN FACTORS SOCIETY 34th ANNUAL MEETING- 5. Fitts, P.M., 1954. The Information Capacity of the Human Motor System in
Controlling the Amplitude of Movement 8.
Learning to Master Robotic Arm Movements with Bimanual Joystick Control: Indicators 121
Jung, K., Chu, B., Park, S., Hong, D., 2013. An implementation of a teleoperation system for robotic beam assembly in construction. Int. J. Precis. Eng. Manuf. 14, 351–358. https://doi.org/10.1007/s12541-013-0049-3 MacKenzie, I.S., 1992. Fitts’ Law as a Research and Design Tool in Human-
Computer Interaction. Human–Computer Interaction 7, 91–139. https://doi.or g/10.1207/s15327051hci0701_3 Nordberg, M. 1987. Feasibility of strip-road spacing less than 25m in thinnings - areport on trials in first thinnings. The Forest Operations Institute of Sweden, Kista, Report No. 3:33. ISSN 0346–6671. Ovaskainen, H., Palander, T., Tikkanen, L., Hirvonen, H., Ronkainen, P., 2011.
Productivity of Different Working Techniques in Thinning and Clear Cutting in a Harvester Simulator. BALTIC FORESTRY 17, 11. Ovaskainen, H., Uusitalo, J., Väätäinen, K., 2004. Characteristics and Significance of a Harvester Operators’ Working Technique in Thinnings. International Jour- nal of Forest Engineering 15, 67–77. https://doi.org/10.1080/14942119.2004. 10702498 Ranta, P., 2009. Added values of forestry machine simulator based training, in:
Proceedings of the International Conference on Multimedia and ICT Education. Citeseer, Lisbon, p. 6. Soukoreff, R.W., MacKenzie, I.S., 2004. Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts’ law research in HCI. International Journal of Human-Computer Studies 61, 751–789. https://doi.org/10.1016/j.ijhc s.2004.09.001 Suzuki, S., Harashima, F., 2008. Analysis of machine operation skills using hand
discrete movement. Presented at the 2008 International Conference on Control, Automation and Systems, pp. 2824–2829. https://doi.org/10.1109/ICCAS.2008. 4694240 Wickens, C.D., Hutchins, S., Carolan, T., Cumming, J., 2013. Effective- ness of Part-Task Training and Increasing-Difficulty Training Strategies: A Meta-Analysis Approach. Hum Factors 55, 461–470. https://doi.org/10.1177/ 0018720812451994
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-33-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional