Jonas Andersson, Daban Rizgary, Mikael Söderman, Johanna Vännström (2023). Exploring remote operation of heavy vehicles – findings from a simulator study. In: Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems. AHFE Open Access, vol. 69, pp. 46–55. https://doi.org/10.54941/ahfe1002820
Fully automated driving has posed more challenges than expected, and remote operation of heavy vehicles is increasingly getting attention. Therefore, human remote operators may have an essential role in compensating for the technological shortcomings in vehicle automation. This poses challenges in designing the work of human remote operators of automated heavy vehicles. This paper presents findings from a research project performed in collaboration between the RISE Research Institutes of Sweden and Scania. In the project, human-automation interaction requirements and challenges for remote operator work were explored through a simulator study. Before the study, three main operator tasks were defined: assessment, assistance, and remote driving. The simulation occurred in a transportation scenario where operators handled ten trucks driving on a public road and confined areas (transportation hub). Fifteen participants completed the study. The results provide examples and insights into classical automation-related challenges in a new context – the remote operation of heavy vehicles. Instances of challenges with situational awareness, out-of-the-loop, trust, and attention management were found and are discussed in relation to HMI design and requirements.
As it stands today, autonomous vehicles require human supervision and assi- stance to safely roam the roads, be it in a confined area or on the highway (Habibovic and Chen, 2021). With the help of stable network communi- cation and a remote operator interface humans can monitor and command the vehicle from virtually anywhere. The benefits of remote operation super- sedes having a safety driver on-board the vehicle in the driver’s seat from a cost perspective. However, simultaneously, many new challenges emerge from replacing the safety driver to a remote location and broadening the task of managing one vehicle to potentially managing multiple vehicles.
Remote control refers to operating a system at a distance which SAE Recommended Practice J3016 (SAE International, 2018) defines as: “A driver
Exploring Remote Operation of Heavy Vehicles – Findings From a Simulator Study 47
who is not seated in a position to manually exercise in-vehicle braking, acce- lerating, steering, and transmission gear selection input devices (if any) but is able to operate the vehicle.”
A Systemic View of Remote Control A recently completed prestudy on human factors in the remote operation of automated heavy vehicles shows that the success of the remote operation will be affected by many interdependent factors (Habibovic et al., 2020). For example, the situational awareness (Hosseini and Lienkamp, 2016), tele- presence (Bout et al., 2017) and workload (Chucholowski, 2016; Neumeier et al., 2019) are factors that need to be addressed. Moreover, from an orga- nisational standpoint, human operators in remote driving might need other skills and training compared to drivers of manually operated trucks. It is also essential to consider when transitions will occur between different control modes in the remote operation task (Michon, 1985). The cognitive wor- kload can be expected to vary when switching between tasks and control modes (Squire and Parasuraman, 2010) if the operator environment is not adequately designed from a human factors standpoint.
Remote Control Applications Remote operation can be done on operational, tactical or strategic control modes (Michon, 1985), which are often intertwined, and carried out in combination with automated driving functions. In tactical and strategic ope- rations, it is likely that the remote operator will instead monitor and plan for several vehicles at a time and intervene only when something does not go according to plan. In the project, the remote operation was envisioned to be utilised for the following applications:
Remote assessment: Enables the remote operator to investigate (debug) issues. In remote assessment, the information flow is one-way, i.e., the vehicle sends error messages and system state information to the human operator, but the operator does not directly control the vehicle. This supe- rvisory work is always relevant and can be seen as a base case for remote operations.
Remote assistance: Enables the remote operator to help the vehicle understand and handle a given situation. This is sometimes relevant.
Remote driving: Enables the remote operator to “drive” or evacuate the vehicle in an emergency (e.g., at roadworks or when the vehicle is stuck in a complex situation). This is rarely relevant but presumably very critical when it is needed.
In the present study, we explored the remote operation task in the menti- oned applications: remote assessment, remote assistance and remote driving by designing a remote operator interface and evaluating it in a simulator. The purpose of the study was to explore requirements for remote operator work and human-machine interaction in a hub-to-hub transportation setting.
Figure 1: Workstation for assessment and assisting operations (left) and remote driving (right).
METHODOLOGY
In the exploratory simulator study, 15 test participants (ten male, five female) were recruited within Scania. Eight participants worked with R&D of auto- mated vehicle functions, three worked with R&D of remote operation of heavy vehicles, one worked as a control room operator in a testing faci- lity, and one worked as a fleet manager of manual trucks. Six participants had previous experiences working in a control centre or working with the development of control centres. Four participants stated no experience wor- king with autonomous trucks. Their age spanned from 25 years to 55 years (Mean = 38.4 y, SD = 9.03 y).
Simulator Study Setup and Procedure A control room simulator was designed for human operators to assess, assist and actively drive automated vehicles remotely. The control room simulator was divided into two workstations: one with a mouse, keyboard and speakers for the assessment and assistance tasks and one with a steering wheel and pedals for the driving task. The two stations were set up on a regular office desk allowing the test participants to adjust the height according to their preferences. For seating, a regular office chair was used with wheels allowing the test participant to move between the stations (Figure 1).
The information provided on the screens to the participants included:
A list of vehicles in operation, the name of each vehicle, its current activity destination, speed, estimated time to arrival, and deviations from each vehicle’s schedule.
Vehicle Information: Specific information about vehicles selected by the participant, such as planned mission, speed and direction. This informa- tion also included vehicle camera views and CCTV cameras from the hubs, a safe stop function, and an offboard software parking brake. It was also possible to see the departure time after loading.
A schematic view, i.e., a visualisation of the vehicles’ positions in time in relation to each other. The view also showed the vehicles’ names and if the vehicles were driving between the hubs or were operating in the hubs.
A geographical map showing the vehicles’ geographical positions, directi- ons and activities. It was possible to zoom in and out and to pan the view.
Exploring Remote Operation of Heavy Vehicles – Findings From a Simulator Study 49
Figure 2: Remote operation user interface with a geographical map and alerts (red areas on the screen).
Alerts were displayed in three places on the screen with a sound: (I) inco- ming alerts on top of the screen with a red border to draw the participants’ attention. These alerts disappeared after a few seconds not to distract the operator while being occupied with other tasks, (II) incoming alerts are displayed in the vehicle list next to the vehicle it concerned, and (III) a list of all the incoming alerts (old and new) in the right top corner of the screen. Red dots indicated new alerts (Figure 2). These incoming alerts did not disappear until they were acknowledged or cleared by the participants.
Experiment Procedure The test leader introduced the test participants to the study (background, aim and purpose), who also described their role as remote operators and the tasks they were asked to accomplish. The test leader also explained the control room simulator’s different functions and how to use the two workstations. The participants were also given time to practice with the two workstations to perform given tasks in the three control modes (assessment, assistance and remote driving).
The main tasks the participants were given in the study were to assess ten automated trucks driving between two hubs, maintain an even flow, and act accordingly on messages and alerts from the system. Vehicles had different speeds in different parts of the map (highway and hub). The operators were asked to act on events that could occur between the two hubs.
Five events were defined along the route. The events were introduced by the second test leader sitting at another desk behind the participant and the prominent test leader. The events consisted of different types of obstacles and problems.
Road works event: The delays for the specific vehicles were shown in the vehicle list. Action: No specific actions are required by the operator
Figure 3: The bathtub event. Vehicle information is to the left, and camera views are to the right. The vehicle has requested permission from the remote operator to cross the side lane marking to pass the bathtub blocking the way. Blue lines indicate the planned route.
Water puddle event: The water puddle made it impossible for the vehicle sensors to confirm that the route was safe to drive. The system requested assistance from the operator. Action: the operator takes control and drives the vehicle through the water puddle.
Bathtub event: A bathtub blocked parts of the drive lanes in both directi- ons. The vehicle cannot pass unless it crosses the road’s side lines. The system requests assistance from the operator. Action: The operator temporarily gives permission to the vehicle to cross the sidelines to pass the obstacle (Figure 3).
Loading dock event: The vehicle stopped at a parking pocket and sent an alert to the operator requesting help to drive to the specific loading bay. Action: the operator takes control and drives the vehicle to the designated area for the vehicle.
Sensor degradation event: The automation system is not working due to sensor malfunction. The vehicle lowers the speed to 30km/h. The system alerts the operator to initiate a safe stop manoeuvre. Action: The operator initiates a safe stop manoeuvre, and the vehicle drives to a safe stop position.
The test session took around 1,5 h to complete. During the test, the par- ticipants were also asked to think aloud, i.e., to orally communicate to the test leaders about their uncertainties, questions and thoughts, and to describe what they were thinking and how they were acting on the different messages and alerts.
After the sessions, the test participants were asked about their experi- ences, perceived challenges and problems, missing information and other areas of improvement regarding the human-machine interfaces. Finally, the
Exploring Remote Operation of Heavy Vehicles – Findings From a Simulator Study 51
participants were asked if they had additional comments and thoughts to share about the system, the HMI and their experiences as a remote operator.
RESULTS
First, the participants’ feedback from the think-aloud protocol and post- interviews on the HMI are presented. This is followed by an analysis of the assessment, assistance and driving tasks and subjective workload assessments.
Evaluation of HMI Components The usefulness of the vehicle list received mixed feedback, where some infor- mation was seen as redundant. In the current study (only) ten vehicles were monitored, and sorting, filtering and prioritisation were not necessary to see all vehicles. In a scenario where the number of vehicles is scaled up, functions to filter out to see, e.g., a list of all vehicles with deviations or in a defined area, will probably be of higher importance. Concerning individual vehicle information, test participants sought more assurance when they needed to make changes to the vehicle’s operation. More off-board support in terms of recommendations and what the changes mean in terms of impact on opera- tions (e.g., loss of deliveries) were requested. In the simulated environment, the operators (naturally) optimised operations from the available informa- tion on system performance. In general, most of the test persons appreciated the task-based schematic view, which was given higher priority than the map. There were, however, questions regarding how to interpret the visualisation. The test persons liked the simplicity of the visualisation and how it supported keeping the vehicles at an even time distance. However, manual calculations on adjusting time distances could have been automated. Several participants mentioned that they did not use the geographical map to any great extent. The schematic view was perceived as more helpful to do the task of kee- ping an even flow. Apparently, the map feature was used more as a general overview and did not contribute much to the actual job in the simulated envi- ronment. For the alerts, several operators expressed the need to prioritise alarms to know when to drop a task to act upon another incoming alarm. In the hub-to-hub simulation, sensitive bottlenecks in the system could be, e.g., entrances and exits to the hub, blocking of loading docks etc, where several vehicles are quickly affected by a failing vehicle. Hence, alarm prioritisation must be done not only based on vehicle parameters but also considering the working context and overall system impact of a failure. Several operators missed incoming alarms indicating the need for salient alarms where the user needs to confirm incoming alerts.
Evaluation of Remote Operator Tasks – Assessment, Assistance and Driving Remote Assessment The remote assessment task was characterised by keeping track of the ten veh- icles, adjusting and maintaining an even flow of vehicles between the hubs,
and responding to alarms by transitioning to remote assistance or remote dri- ving. The HMI allowed for the delay of vehicles at each hub, respectively, as a tool to even out the flow. The participants expressed the expectation and need to be able to communicate with roadside assistance to remove obstacles and solve problems along the route. Several operators express that the job would probably become boring in the long run. According to several partici- pants, ten vehicles could be monitored without problems from the assessment perspective if there are no other issues in parallel. Participants expressed wor- ries about handling the situation in case of problems with several vehicles at once, or cascading events, pointing out the need to consider the organisation and teamwork in the job design (Habibovic et al., 2020). The experiment illustrated how the role and tasks of the operators and factors such as vehicle capability, deviations in the environment, task design, available information, organisational support etc., are highly interdependent.
Remote Assistance The current study shows examples of how, to some extent, the tasks of asses- sing and assisting automated vehicles overlap. In our experiment, the remote assessment task was mainly a cognitive task with a high-level assistance con- trol component to even out vehicle time distances. Also, it is mainly the control system that drives the vehicles, and the operator waits for alarms or messages to act on. In the assistance mode, the operator transitions to a more detailed decision-making situation with a single vehicle where understanding the situation and the surrounding context becomes critical. Several parti- cipants expected to handle problems (both momentarily and removing the cause) either by solving them themselves or handing over the issue to another instance, as well as warning other road users of dangerous traffic situations. This means that when designing an interface for a remote operator, the need to communicate with others safely and efficiently must be considered either by automatic messages through the system or by direct communication (e.g., roadside assistance, service units, terminal workers and infrastructure). The teamwork aspect was not part of the test setup in this experiment but invites future research.
Remote Driving The remote driving task was performed as the system prompted participants to take over the vehicle to solve the operational problems in the water puddle and loading dock events. The simulator setup for driving remotely in the experiment had some noticeable flaws. The limited field of view made it dif- ficult to get an overview. The operator did not use the available camera views to any greater extent since they were placed in the assistance/assessment sta- tion and not readily available in the driving station. The operators reflected on this difficulty and suggested better fields-of-view and other sensor capabi- lities, e.g., remote operator driver assistance systems, to improve situational awareness. Despite the shortcomings of the simplified simulator setup, the results point out findings of importance to future remote driving implemen- tations. The scarcity of information due to limited field-of-view also impacted
Exploring Remote Operation of Heavy Vehicles – Findings From a Simulator Study 53
the feeling of driving safely. Questions of liability also arose – “Who is respon- sible if I hit something, and the information was not sufficiently available?” One participant experienced that the only way to solve the situation is to trust the system, even though the participant would have preferred more information to feel comfortable. This could also occur outside the simula- ted environment and impose needs, e.g., communication and reassurance of situations. Another question posed was how liability would be handled during remote operations. Can the remote operator’s driving license be with- drawn if a “wrong” decision is made, even if the appropriate information is scarce? A challenge is that this scarcity of information may only become evident in hindsight when, e.g., when accident analysis has established the facts.
Further, we expected task-switching between assessment, assistance and driving to be effortful or at least time-consuming as operators needed to shift focus back and forth. However, the participants perceived these shifts as a natural part of the work and did not bring forward these switches as anything out of the ordinary. It was the tasks and problem-solving that imposed the operator workload rather than the task-switching.
DISCUSSION AND CONCLUSION The present study explored the requirements for remote operator work in supervisory control of ten heavy vehicles in hub-to-hub transportation by means of a simulator study. Despite a very short introduction and learning period before the experiment (15 min.), the participants could perform the assigned tasks indicating that the simulator had a basic level of usability. However, there are also several points for improvement. Participants asked for prognostic tools when making vehicle adjustments to be able to under- stand the effect on operations of a specific change on the fleet level. This need resonates with the concept of situation awareness and the ability to project future status (Endsley, Jones and Bolte, 2003), which should be sup- ported through design. The schematic view of time distances was appreciated since it facilitated the task of delaying vehicles to achieve an even flow. It points out the relevance of task-based displays (see e.g., Jamieson et al., 2007) for an example from process control) that are tightly connected to the goals or KPI:s of the operator work. In a real system, this could be more complex tasks such as keeping delivery times by re-planning and optimisa- tion tasks that are facilitated by task-based visualisations. Interestingly, the geographical map was not perceived as a key feature by the participants since the physical distance on the map did not provide sufficient information about the flow of vehicles when speeds varied between the hubs (a relatively short distance on the map could still be a long time due to low speed). The expressed need for support in task prioritisation in relation to alerts from the vehicles also highlights the need for systems analysis, systems modelling and risk analyses for remote operation. If contextual factors are not consi- dered, appropriate decision support will be very difficult to provide. Here, scenario-based methodologies could provide useful input (Kettwich et al., 2022). Although only working for 1.5h, some participants mentioned being
bored from working as a remote operator. Boredom highlights the irony of automation (Bainbridge, 1983) - when everything works, very little needs to be done, and staying vigilant is challenging for humans in general. Boredom is a known caveat in monitoring highly automated systems and seems valid also in this context. Since an operator cannot be expected to be vigilant over longer periods, appropriate alarm systems (Thunberg and Osvalder, 2007) must also be designed and implemented for the remote vehicle operation type of systems. Designing the remote operator job in context, the worker should be expected to be part-time out-of-the-loop. Therefore, we argue that design effort should be focused on how operators can work themselves back into the loop at the appropriate abstraction level (from fleet to single vehi- cle and vice versa) as needed. In comparison to driving automation, remote operation of vehicles will also benefit from further advancing the understan- ding of the out-of-the-loop concept for this particular domain (Merat et al., 2019). The results from the simulator study show the importance of taking a systems perspective in the development and implementation of remote opera- tion control centres. Aspects such as the operational context, control modes, vehicle capabilities, operator tasks, HMI, and organization of work have to be considered in parallel to ensure operational safety and performance (Habibovic et al., 2020). The study also showed how classical automation- related challenges such as situation awareness, boredom, vigilance, out- of-the-loop performance problems and the need for appropriate alarm systems can be expected also in the domain of remote operation of heavy vehicles.
ACKNOWLEDGEMENT
The authors would like to acknowledge the Swedish Innovation Agency Vin- nova for making this research possible through the research grant project “Heavy Vehicle Operation Center” (2020-02953).
REFERENCES
Bainbridge, L. (1983) ‘Ironies of automation’, Automatica, 19(6), pp. 775–779.
Available at: https://doi.org/10.1016/0005-1098(83)90046-8. Bout, M. et al. (2017) ‘A Head-Mounted Display to Support Teleoperations of Shared Automated Vehicles’, in Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct. Auto- motiveUI ‘17: ACM 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Oldenburg Germany: ACM, pp. 62–66. Available at: https://doi.org/10.1145/3131726.3131758. Chucholowski, F.E. (2016) ‘Evaluation of Display Methods for Teleoperation of
Road Vehicles’, Journal of Unmanned System Technology, 3(3), pp. 80–85. Available at: https://doi.org/10.21535/just.v3i3.38. Endsley, M.R., Jones, D.G. and Bolte, B. (2003) Designing for situation awareness.
London: Taylor & Francis. Habibovic, A. et al. (2020) Human factors related to remote control of automated heavy vehicles. Final prestudy report FP04. SAFER. Availa- ble at: https://www.saferresearch.com/library/final-report-human-factors-related- remote-control-automated-heavy (Accessed: 28 April 2021).
Exploring Remote Operation of Heavy Vehicles – Findings From a Simulator Study 55
Habibovic, A. and Chen, L. (2021) ‘Connected Automated Vehicles: Tech- nologies, Developments, and Trends’, in International Encyclopedia of Transportation. Roger Vickerman. Elsevier Ltd (1st Edition), p. 10. Available at: https://www.elsevier.com/books/international-encyclopedia-of- transportation/vickerman/978-0-08-102671-7. Hosseini, A. and Lienkamp, M. (2016) ‘Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality’, in 2016 IEEE Intelligent Vehicles Symposium (IV). 2016 IEEE Intelligent Vehicles Symposium (IV), Gotenburg, Sweden: IEEE, pp. 1366–1373. Available at: https://doi.org/10.1109/IVS.2016.7535568. Jamieson, G.A. et al. (2007) ‘Integrating Task- and Work Domain-Based Work Analyses in Ecological Interface Design: A Process Control Case Study’, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 37(6), pp. 887–905. Available at: https://doi.org/10.1109/TSMCA.2007.904736. Kettwich, C. et al. (2022) ‘A Helping Human Hand: Relevant Scenarios for the
Remote Operation of Highly Automated Vehicles in Public Transport’, Applied Sciences, 12(9), p. 4350. Available at: https://doi.org/10.3390/app12094350. Merat, N. et al. (2019) ‘The “Out-of-the-Loop” concept in automated driving: proposed definition, measures and implications’, Cognition, Technology & Work, 21(1), pp. 87–98. Available at: https://doi.org/10.1007/s10111-018-0525-8. Michon, J.A. (1985) ‘A Critical View of Driver Behavior Models: What Do We
Know, What Should We Do?’, in L. Evans and R.C. Schwing (eds) Human Beh- avior and Traffic Safety. Boston, MA: Springer US, pp. 485–524. Available at: https://doi.org/10.1007/978-1-4613-2173-6_19. Neumeier, S. et al. (2019) ‘Teleoperation: The Holy Grail to Solve Problems of Automated Driving? Sure, but Latency Matters’, in Proceedings of the 11th Internatio- nal Conference on Automotive User Interfaces and Interactive Vehicular Applica- tions. New York, NY, USA: Association for Computing Machinery (AutomotiveUI ‘19), pp. 186–197. Available at: https://doi.org/10.1145/3342197.3344534. SAE International (2018) Surface Vehicle Recommended Practice (J3016): Taxo-
nomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. Recommended Practice. Squire, P.N. and Parasuraman, R. (2010) ‘Effects of automation and task load on task switching during human supervision of multiple semi-autonomous robots in a dynamic environment’, Ergonomics, 53(8), pp. 951–961. Available at: https://doi.org/10.1080/00140139.2010.489969.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-45-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional