Hironao Yamada, Satoshi Ueki, Takahiro Ikeda (2025). Construction machinery work support system using mixed reality. In: Human Interaction and Emerging Technologies (IHIET 2025). AHFE Open Access, vol. 197, pp. 213–219. https://doi.org/10.54941/ahfe1006714
Although the number of fatal and injured accidents in the construction industry has decreased in recent years, they are still high compared to other industries, and ensuring safety at construction sites is an important issue. Construction sites are complex and dangerous working environments, and workers from different industries often have to work together. In addition, changes in work methods and procedures may occur due to design changes, bad weather, delays in work in other trades, and shortages of equipment and labor. In order to ensure safety and resume work, it is necessary to report the change in work method to the construction supervisor and adjust and discuss the changed work method. Meanwhile, the construction industry is also promoting the digitalization of work and the use of smart devices as DX (digital transformation). One of these is to provide rapid and accurate technical support even in the event of a wide-area disaster by grasping the local situation from a remote location without having to go directly to the disaster site. In addition, in order to improve work efficiency, attempts are being made to apply remote presence to work that previously required attendance. In this study, we developed a human-machine system that visually displays instructions from a site supervisor to workers using mixed reality (MR), that is, an MR work instruction presentation system, with the aim of quickly and accurately conveying complex instructions to workers while working with construction machinery, which has been difficult to convey while working. By using this system, arrows and construction drawings can be superimposed on the worker's field of vision, making it possible to easily and accurately convey detailed instructions such as the location to crush with construction machinery, the position to grab, and the transport route of the transported items. In this study, we built a construction machinery work support system using Microsoft's HoloLens and conducted subject experiments using an actual machine.As a method of presenting work instructions using MR for the transport and stacking work of blocks by construction machinery using this system, we proposed four methods: A) work instructions using only drawing processing, B) work instructions using drawing processing and text, C) work instructions using drawing processing and voice, and D) work instructions using animation.In the experiment, block transport and assembly work was performed using the four proposed work instruction presentation methods, and the optimal instruction presentation method was examined based on the work time, mental burden using NASA-TLX, and a questionnaire survey. As a result, it was confirmed that using this system to give work instructions shortens the time required to operate construction machinery and is also effective in reducing the mental burden on NASA-TLX. In addition, it was found that of the four proposed methods of presenting work instructions, the most effective methods were the method using drawing processing and voice, and the method using animation. These results are believed to be useful for the development of a human-machine system that visually displays instructions to workers.
Construction sites are complex and inherently dangerous working environments where workers from various industries frequently collaborate. Concurrently, the construction industry is promoting the digitalization of work and the adoption of smart devices as part of digital transformation (DX). One such application is the provision of rapid and accurate technical support during wide-area disasters by remotely assessing the local situation, thus eliminating the need for direct on-site access. Furthermore, to enhance work efficiency, efforts are being directed towards applying remote presence technologies to tasks that traditionally required on-site attendance.
Xiao et al. (2018) conducted a review of virtual reality and augmented reality (VR/AR) applications in construction safety, which included a classification based on technology characteristics, application areas, safety enhancement mechanisms, and evaluation. They also have noted that challenges for safety inspection and instruction contain low interoperability of VR/AR-CS information and unskilful visual literacy of workers. Yang et al. (2021) conducted a literature review on augmented reality for digital fabrication in architecture, analysing human-robot interaction, data sharing, and AR3D holographic instructions. They indicated that AR seems the have indeed the potential capabilities to solve the
traditional construction problems and fabrication difficulties, by overlapping holographic instructions to on-site projects in the real world. Alizadehsalehi et al. (2020) presented a review containing relevant articles on BIM (Building Information Modeling) and XR (Extended Reality: VR, AR, MR) within the AEC (Architecture, Engineering, Construction) industry. They indicated that various XR technologies have shown great potential to truly revolutionize how to design, build, operate, and monitor in the AEC industry. Consequently, there are high expectations regarding the potential of AR, MR, and XR within the construction field. However, limited research has been reported on the application of MR to construction machinery, such as backhoes, and the verification of its effectiveness. In a previous study (Hironao et al., 2024), the authors compared the effectiveness of the proposed MR work support system against presenting the completed figure after stacking blocks and instructions from the block-stacking manual. The results confirmed that the working time tended to decrease with the use of this system, and a similar trend was observed for the mental burden, as measured by NASA-TLX. Therefore, this study proposes four variations of MR work support systems and compares their performance.
To address the challenge of quickly and accurately conveying complex instructions to workers operating construction machinery, a human-machine system has been developed. This system utilizes mixed reality (MR), functioning as an MR work instruction presentation system, to visually display instructions from the site supervisor to the workers. This system superimposes arrows and construction drawings onto the worker’s field of vision, enabling the easy and accurate conveyance of detailed instructions, such as the target crushing location for construction machinery, the grasping point, and the transportation route of materials. In this study, a construction machinery work support system was constructed utilizing Microsoft’s HoloLens, and subject experiments were conducted using an actual machine. For the presentation of work instructions using MR for the transport and stacking of blocks by construction machinery, four methods were proposed in this study. During the experiment, block transport and assembly work was performed using the four proposed work instruction presentation methods. The optimal instruction presentation method was then examined based on the recorded work time and the mental burden assessed using NASA-TLX.
A Work Order System Using MR
Figure 1 illustrates an overview of the proposed system, which comprises a construction robot, a see-through head-mounted display (HMD) worn by the construction robot operator, and a tablet PC operated by the work leader. In this research, Microsoft HoloLens was selected as the see-through HMD due to its established presence in the construction field. The tablet PC serves as the interface for work instructions. The work instructor uses it to acquire the image perspective of the construction robot operator transmitted from the HoloLens in real time. Subsequently, the work instructor can draw instructions onto the acquired video image based on the work situation and present them to the construction robot operator. The MR system has the
Construction Machinery Work Support System Using Mixed Reality 215
potential to significantly enhance efficiency on large construction sites by enabling skilled workers to instruct multiple less-skilled workers on work procedures.
Figure 1: Overview of the proposed system.
Four methods for presenting work instructions using MR for the transport and stacking of blocks by construction machinery were proposed in this study: A) work instructions using only drawing processing, B) work instructions using drawing processing and text, C) work instructions using drawing processing and voice, and D) work instructions using animation. (see Figure 2).
Figure 2: Instruction presentation method.
A) Work instructions are provided through drawing processing, where the work instructor utilizes the following symbols:
(block gripping position), L (installation alignment mark), and arrows to specify the block’s transport number and destination.
Figure 3 illustrates the provision of work instructions. The figure depicts lines and symbols indicating the gripping, installation, and stacking positions as displayed by the MR system from the operator’s perspective. Based on the instruction, the operator moves the corner of the block (Fig. 3, left), indicated by the green L-shaped line, to the position of the red L-shaped line (Fig. 3, right). The alignment of the red L-shaped line, positioned within the right- side black dashed line, with the corner of block 1, indicated by the green L-shaped line within the left-side dashed line, is observed.
Figure 3: Work instructions using drawing processing.
B) In addition to method A), text instructions provide specific guidance on how to proceed with the work. The text is displayed within the field of view of the construction robot operator at the work site.
C) In addition to method A), specific instructions on how to perform the work are provided through audio.
D) Instructions for stacking blocks are presented in an animated video. When providing instructions, a PowerPoint slideshow is initiated, and the slides progress according to the work situation.
Evaluation of the Effectiveness of MR-Based Work Instructions The experimental evaluation process utilized the following indices: work time and mental strain as measured by NASA-TLX.
a) Working time The time [s] required to complete the designated work was recorded as an index of work time. To standardize measurement, timing commenced when work instructions were presented to the subject and concluded upon the successful stacking of all specified blocks.
b) Mental Workload Evaluation (NASA-TLX) The workload was evaluated using the NASA Task Load Index (TLX) (Hart and Staveland, 1988). The NASA-TLX subjective workload is assessed based on six factors: mental demand (MD), physical demand (PD), temporal demand (TD), own performance (OP), effort (EF), and frustration level (FR).
Construction Machinery Work Support System Using Mixed Reality 217
The assessment using NASA-TLX involved the following steps: (1) Paired comparison of each subscale, (2) Workload assessment tasks (block stacking in this study), (3) Workload assessment for each subscale.
Following these steps, the overall workload (WWL score: Mean weighted workload score), as well as the workload for each subscale, can be assessed.
To investigate the optimal instruction presentation method, a subject experiment was conducted utilizing the four proposed methods. The subjects operated a construction robot to stack four designated blocks out of seven on a platform in a specified order. The blocks to be used and the stacking procedure were specified for each of the four work instruction presentation methods. Six subjects participated in the experiment, and the optimal work instruction presentation method was investigated based on the recorded work time and the mental burden assessed using NASA-TLX.
Figure 4 presents the average task times for all subjects. The results indicate that methods C and D yielded shorter average task times compared to methods A and B. This is likely attributed to the fact that methods C and D do not necessitate the checking of text or arrows, unlike methods A and B, and facilitate an intuitive understanding of instructions.
Figure 4: Average of work time.
Figure 5 presents the average results of the six NASA-TLX evaluation indices obtained from six subjects across the four different operation methods. The vertical axis represents the average evaluation value for each index, with method A depicted in blue, B in yellow, C in green, and D in red. The results indicate that method B exhibited the highest values in the MD (mental demand) and EF (effort) categories compared to methods A, C, and D, with a substantial difference observed.
This result can be attributed to the time and effort required to process textual instructions within the workspace during the implementation of drawing processing and text (method B), as well as to the comparatively lower intuitiveness of these instructions. Methods B and A showed higher FR (frustration) scores. For method B, the effort involved in reading text during the task, coupled with the potential for the text display screen to be outside the HoloLens viewing angle during block transport and assembly, likely contributed to these outcomes.
Figure 5: Results of NASA-TLX.
Figure 6 presents the average WWL scores obtained from NASA-TLX. The figure indicates that methods C and D resulted in low WWL scores, which may contribute to a reduction in mental stress. Conversely, method B yielded the highest WWL score. This is likely due to the cognitive effort required to process instructions, as the text necessitates reading during task execution.
Figure 6: Average of WWL score.
CONCLUSION
This study developed and evaluated a construction machinery work support system utilizing Microsoft’s HoloLens through subject experiments with an
Construction Machinery Work Support System Using Mixed Reality 219
actual machine. The results confirmed that the developed system, when employed for delivering work instructions, reduces construction machinery operation time and significantly lowers mental burden as measured by NASA-TLX. Furthermore, among the four proposed methods for presenting work instructions, the most effective were the method employing drawing processing and voice, and the method utilizing animation. These findings are considered valuable for the further development of human-machine systems that visually display instructions to workers.
ACKNOWLEDGMENT
The authors gratefully acknowledge Mr. Hiroki TACHIBANA, a graduate student (currently affiliated with Mitsubishi Electric Corporation), for his assistance in conducting the experiment.
REFERENCES
Hart S., Staveland L. (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research in Advances in Psychology, Human mental workload, 52, North Holland, pp. 139–183. Hironao Yamada, Satoshi Ueki, Takahiro Ikeda (2024), Effects of Work Support
System for Construction Machine Work, Frontiers in Artificial Intelligence and Applications, Volume 396: Electronics, Communications and Networks, pp. 179–184, doi: 10.3233/FAIA241339. Xiao Li, Wen Yi, Hung-Lin Chi, Xiangyu Wang, Albert P. C. Chana (2018) “A critical review of virtual and augmented reality (VR/AR) applications in construction safety”, Automation in Construction, Volume 86, Pages 150–162. Yang Song, Richard Koeck, Shan Luo (2021) Review and analysis of augmented
reality (AR) literature for digital fabrication in architecture, Automation in Construction, Volume 128, 103762. Sepehr Alizadehsalehi, Ahmad Hadavi, Joseph Chuenhuei Huangc (2020) From BIM to extended reality in AEC industry, Automation in Construction, Volume 116, 103254.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2025)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-73-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional