Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation

The Hong Kong Polytechnic University ↗
Publication year
Digital Object Identifier10.54941/ahfe1005672

Abstract

Mixed reality (MR) technology combines digital elements with the real world, offering immersive experiences in various fields. However, content creation in MR remains challenging for average users due to complex 3D modelling techniques. This paper introduces Light-Tracing, a novel system integrating light painting with MR environments using a robotic arm. Light-Tracing allows users to intuitively create 3D models through path light painting, which AI tools convert into digital assets. The system simplifies spatial content input, making MR content creation more accessible and enhancing user experience. Evaluations demonstrate its positive impact on usability and immersion. Importantly, Light-Tracing lowers the barrier for MR content creation and opens new possibilities for participatory design in urban planning and vehicle customization, highlighting its potential to revolutionize MR content creation.

INTRODUCTION

Mixed reality (MR) provides users with an immersive and interactive experience by combining digital objects with a realistic environment in the user’s field of view. This technology is already widely used in fields such as healthcare, education, and design (Carrasco & Chen, 2021; Morimoto et al., 2022; Tang et al., 2020), and is expected to become even more prevalent with the advancement of Industry 4.0 and the third wave of the Internet (Ziker et al., 2021). However, the average user still faces significant challenges in creating and entering MR content. For example, in urban planning, citizens may have difficulty visualising and engaging with 3D models of proposed infrastructure due to the complexity of current 3D modelling software (Hanzl, 2007). Similarly, in educational environments, teachers and students struggle to create interactive 3D content without specialised training (Murray & Johnson, 2021). Current technologies such as 3D scanning and photogrammetry, while allowing for the creation of accurate virtual models, do not make the process easy for untrained users (Darzentas et al., 2018). Light-Tracing creates 3D models by allowing users to light- draw through intuitive paths and have them converted into digital models by AI tools to convert them into digital assets, addressing these challenges. This

© 2024. Published by AHFE Open Access. All rights reserved. 1058

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation 1059

approach not only simplifies spatial content input, but also enhances the user experience, making MR content creation more accessible and engaging.

This study proposes a content creation system focusing on user input, dithering noise reduction, size expansion, and path light painting, all under the intelligent control of a robotic arm. The system ultimately restores long exposure photographs containing light painting information in an MR environment. It aims to lower the threshold for users to input spatial content and enhance their experience. This paper introduces two content input methods to meet different scenario requirements, creating content with varying stability and degrees of freedom.

The system does not require the user to have 3D modelling skills, and the user can directly control the robotic arm to complete the creation of stereoscopic trajectories. We conducted a user test with four users without modelling skills, introducing them to common 3D modelling tools and allowing them to experience the system we designed. The user tests showed that the system has improved in usability, satisfaction, and immersion compared to traditional 3D information input methods, and the user feedback indicated that the mixed reality light painting system inspires creativity in 3D space.

The scope of this study includes: 1) developing and evaluating an MR content creation system; 2) exploring the effectiveness of this system in different application scenarios; and 3) demonstrating how simplifying the user input process can enhance user experience. This research primarily addresses the challenges of user input in MR content creation, specifically through intuitive 3D model creation using intelligently controlled light painting and long-exposure photographs.

RELATED WORKS MR Spatial Content Creation

Creating a comprehensive mixed reality (MR) environment requires significant time and financial resources. Current 3D modelling techniques, while effective, often demand advanced expertise and high-end equipment. For example, 3D scanning and photogrammetry provide detailed captures of real-world objects but require expensive hardware and technical skills (Ratican et al., 2023). Procedural content generation (PCG) uses algorithms to create content automatically but requires complex computational resources (Smelik et al., 2014).

Gesture-based interfaces and haptic feedback devices allow intuitive interactions with virtual elements but depend on sophisticated hardware and calibration (Schraffenberger & van der Heide, 2016). Collaborative MR platforms enable remote teamwork but rely on stable, high-speed internet and compatible devices. AI and machine learning assist in tasks like object recognition and scene understanding but require specialized knowledge and substantial computational power (Tung & Cheng, 2018).

In contrast, light painting offers a simpler and more accessible method. It involves using a light source to ‘draw’ in the air, captured by long exposure

photography, requiring minimal equipment – just a light source and a camera. This technique is highly intuitive, making it suitable for non-experts.

Light-Tracing builds on this by integrating light painting with robotic arm control and AI tools, enabling detailed digital asset creation through simple user inputs. This approach simplifies the creation process and enhances user engagement and creativity in MR environments, addressing the accessibility challenges of other methods.

Light Painting Light painting is an artistic form that captures the movement of light in space using long exposure photography (De Benedetti et al., 2019). It has been observed that by creating directly within space using light painting, users can more quickly understand and create paths that incorporate three- dimensional spatial information. However, the inherent limitations of long exposure photography prevent users from seeing their creations in real-time, which can diminish the quality of the content and the user experience.

Wang and others have combined light painting with augmented reality (AR) technology. Through moving a light source along these AR trajectories, users can create accurate traces on the photograph, thereby bridging the gap between actual output and planned trajectories, and enhancing the quality of the creative content (Wang et al., 2021). Nestor Z. Salamon and others have proposed using light painting videos combined with computation to optimise the photo environment and light painting patterns. By eliminating the motion artefacts of people and optimising light painting paths, the quality of the resulting photographs is improved (Salamon et al., 2017). Although the light painting patterns obtained through this method are smoother and more aesthetically pleasing, they correspondingly lack a human touch.

Based on the analysis and reference of the above technologies, the system proposed in this paper supports users in quickly creating content through directly spatial light painting in MR environment according to their own ideas. At the same time, long exposure photographs containing light painting trajectories are still captured and preserved.

Robotic Arm Control Robots and multi-degree-of-freedom robotic arms have been widely applied in sectors such as manufacturing and healthcare (Campeau-Lecours et al., 2019). In the system proposed in this paper, the significance of employing a robotic arm lies in its ability to act as a flexible extension of human physiological functions and its high controllability when integrated with algorithms.

Robotic arms demonstrate the stability of machines and the capability to perform actions over a broader range than humans, breaking through the physiological limitations of human capabilities (Gawli et al., 2017). When paired with intelligent algorithms, these arms transcend their roles as mere tools. For instance, the development of visual capabilities allows them to ‘recognize’ and ‘judge’ objects to be grasped (Lee, 2020). Combined with

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation 1061

intelligent algorithms, they are poised to become ‘smarter’ collaborators capable of learning and adapting, offering superior cooperative behaviours.

SYSTEM The new 3D virtual content creation system proposed in this paper (hereinafter referred to as ‘the system’) consists of three main components, including Input, Record, and Outcome (see Figure 1).

The basic workflow is as follows: a) the user selects tangible or intangible control of the robotic arm; b) the user previews the trace of the manipulated robotic arm movement in real time in the MR world; c) the camera records the track of the attached light during the movement of the robotic arm by taking long exposures; and d) the AI converts the multiple photographs recorded by the camera into a digital asset to be bound in the real world.

Figure 1: New 3D virtual content creation system.
Original PDF Figure 1:

Input The components of the first part, ‘Input’, include the user, the controller, the robotic arm and the MR headset.

This research is based on HRC methodology, where human is an indispensable subject in the system. Therefore, in the system proposed in this paper, the robotic arm is not working fully automated and the user will act as the main content creator providing content input for the operation of the robotic arm. In this system, users can input commands through two methods: physical operations and position tracking. After the robotic arm receives movement commands from the user via its independent local area network (LAN), the AI algorithm (OpenAI, 2024) optimizes these commands to create smoother and more stable movement trajectories.

Because the light trace can only be observed after a long camera exposure (detailed in the ‘Record’ section), users cannot see what is happening in real- time as they type. This is similar to drawing on a piece of paper without being able to see if the lines are accurate, which can affect the quality of the final output for untrained amateurs. To address this issue, we integrate a position sensor into the robotic arm’s light source to track its spatial coordinates. This

sensor continuously sends the light source’s spatial position information to the MR headset, creating visualized movement paths within the headset’s field of view, thus providing a real-time preview of the input content.

Record The components of the second part, ‘Record’, consist of the robotic arm and cameras. It’s important to note that during recording, the robotic arm is equipped with an LED of a user-specified colour. This allows the camera to capture the light trace through long exposure.

Each long exposure image serves as a prompt for the AI model to generate 3D content. To achieve higher quality 3D content, the system uses a panoramic surround shot method with 16 cameras arranged around the robotic arm to capture detailed light traces from all angles. However, in practice, 4 angles could not capture the complete shape of the trajectory due to the occlusion of the robotic arm base as well as the robotic arm itself. So finally, the effective shooting angles used in this system are 12 (detailed in the ‘Design and implementation’ section).

Outcome The components of the third part, ‘Outcome’, include 3D Model Training and Application.

Some open-source AI model generation algorithms are used in this system to assist in building the system prototype. The quality of the final output 3D model relies heavily on the quality of the image captured by the camera through long exposure. The final outcome of this system is a virtual content tied to the real world. It can be directly observed in MR environments and can be used in the future in a variety of fields such as creative communication, urban planning, large-scale decoration, or medical education (detailed in the ‘Discussion’ section).

Figure 2: Two methods to control the robotic arm.
Original PDF Figure 2:

METHODOLOGY

We clarified the design and implementation methodology of the above system and verified the effectiveness through testing with target users.

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation 1063

Figure 3: Results of testing the factors affecting image quality.
Original PDF Figure 3:

Design and Implementation Initially, to ensure safety, we defined the safe operating range for the robotic arm as follows: in the x direction, from –600 to 600; in the y direction, from –600 to 600; and in the z direction, from 100 to 600. Based on this range, we designed two control methods.

The first one is physical operation. Since the system begins with human input, how users control the robotic arm for pattern drawing is crucial. We prioritized tangible interaction because the interaction through the physical control of mechanical movement is more helpful for the user to understand the logic of the mechanical operation and can increase the user’s engagement (Horn, Crouser & Bers, 2012). Therefore, we chose a 2-axis joystick to control the robotic arm’s movement in the x and y directions, while using two buttons to control the robotic arm’s movement in the z direction. The system reads the joystick’s position through Arduino analogue pins using an ADC to detect changes in the potentiometer’s resistance, converting these signals to control the arm’s movement. This kind of physical operation method simplifies input and improves output stability and accuracy.

The other one is position tracking. Based on Python 3.9, opencv-python library and xARM-Python-SDK library, we design a method that tracks the position of the user’s gesture through a camera. To exclude the effect of ambient light, we achieve more stable position tracing by recognising objects with specific shapes and colours (see Figure 2). With OpenCV, the in-built camera of the computer can recognize the 2D (‘X’ & ‘Y’) coordinates of objects. To get ‘Z’, this system measures the height of the controller from the camera through recognizing the size of it. Then the system can capture the complete 3D coordinates of the controller.

Based on the above two robotic arm control methods, we capture user- created light-painted content through long exposures to transform it into 3D digital assets in mixed reality. Different exposure times and light movement speeds significantly impact photo quality. To find the optimal shooting parameters, we designed a robotic arm test based on the control variable method. The test involved drawing a 240mm square in the XZ plane with a 240mm pace as the benchmark. We respectively tested the imaging results and 3D models generated by different step sizes at the same exposure time and different exposure times at the same step size.

Comparisons were judged based on imaging graphical results, spatial computation accuracy, and model completeness. The imaging graphical results reflect model similarity to the exposure images. Spatial computation accuracy refers to how accurately the model’s position matches the original space. Model completeness measures the error from broken or missing surfaces.

The results show that exposure times below 15 seconds result in dark, incomplete images due to insufficient light capture. While too long exposure time leads to loss of light information and affects the image quality. When the step size is constant, an exposure time of 20 seconds produces the best image. Additionally, the arm’s 30 mm step size allows for lower imaging graphic errors, higher spatial accuracy, and better model integrity. Therefore, when creating content, we should choose a 30 mm step size and correspondingly set an exposure time of 20 seconds for the camera (see Figure 3).

In the experiments, it was found that using a single image to generate a model will always have errors, especially for more complex spatial shapes. In order to optimize the quality of the model, we took long exposure photos from 12 unobstructed angles to create the only one model.

The model effect generated by superimposing these 12 photos is the closest to the light-painted pattern in a real scene (see Figure 4).

Figure 4: Generating a better model with multi-angle photos.
Original PDF Figure 4:

Validation In the user testing phase, we recruited four target users who had no experience in 3D modelling. The objective of the test was to evaluate the combined impact of light tracking and robotic arms in this mixed reality input method on users’ ability to input 3D information, across three dimensions: usability, satisfaction, and immersion. We also aimed to identify any issues or shortcomings for further improvement and optimization. After experiencing the new 3D information input method, users were asked to rate their experience on a feedback form, with scores ranging from 1 to 5, where higher scores indicate better evaluations (see Table 1).

Based on the test results, users rated Immersion the highest, followed by Satisfaction and Usability. This indicates that the new three-dimensional input method can better help users establish an immersive environment, thereby enhancing their interest in creating forms in a three-dimensional space and positively influencing their spatial imagination. In user feedback,

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation 1065

the main issues with the new system centred around the inability to make modifications after light tracing and accuracy issues with OpenCV recognition.

Table 1. User experience questionnaire using two input methods.

Table available in the original PDF paper.

Types Questions Grades Average Score 1 - 5 3.67 Usability a) I think the method effectively help me to create in 3D. b) I think this approach will help me complete my tasks faster. c) I think based on this method I can create 3D content very easily.

1 - 5 4.19

Satisfaction a) I think the method sparked my interest. b) I am happy with my work of this test. c) I think that the method helps me to be inspired. d) I am willing to use this system again.

1 - 5 5

Immersion Sometimes, I cannot distinguish whether these contents are real or virtual

DISCUSSION

In summary, the “Light-Tracing” system provides a new creative paradigm for content generation by offering users a communication system with the final content creation. It enables users to create content in a more intuitive and immersive manner while lowering the barriers to content generation and enhancing user creativity. It can play a significant role in various fields such as architecture, design, and education.

Public Participation in Urban Planning The research’s content creation method presents new potential for citizen participation in urban planning: it provides a communicative system that eliminates barriers for non-professionals, enables remote access (Rohil & Ashok, 2022), and facilitates participatory process management.

In traditional urban planning, citizen participation often faces challenges and limitations (Hanzl, 2007). Non-professionals often struggle with technical terms and details, hindering their involvement in the planning process. Time and geographical constraints can also prevent their participation. This paper’s approach addresses these issues by creating a communicative urban planning system, enabling non-professionals to take part in urban planning.

Car Skin Creation Virtual development technology has become an important part of the automotive design and manufacturing process nowadays. VR can be a good aid to the process of driving a car (Choi, Jung & Noh, 2015). To address these

challenges, this paper introduces a new MR experience. By simulating real- world sensations and environments, designers and engineers can participate more intuitively in automotive development. This enhances visual effects, making car exterior designs more realistic and precise. Designers can observe and adjust vehicle body shapes, lighting effects, and details in real-time, improving design quality and accuracy.

Educational Creation The new system studied in this paper, utilizing MR, offers immersive skill training and can integrate with existing training solutions (Uhl et al., 2023). In the future, the medical field can utilize this developed system, combining motion capture and AI tools, to aid physicians in reviewing surgical procedures and enhance learning experiences for medical students.

Limitations In terms of limitations, the research of this system is based on virtual reality (VR) and robotic arm technologies, requiring high responsiveness for real-time light-tracing feedback. However, the current hardware technology in the experimental stage cannot fully meet the performance requirements of the system. Therefore, there is a need for further breakthroughs and developments in hardware technology.

Additionally, the portability and comfort of the hardware, specifically the robotic arm and VR headsets, need to be considered. In the current experimental stage, improvements are required to enhance the immersive experience of content creation and reduce user fatigue.

Furthermore, when applying this system to real-world scenarios, challenges related to technology transfer must be taken into account to ensure alignment with the needs of professionals in the respective fields.

CONCLUSION

Mixed reality technology has been extensively researched and applied in the fields of healthcare, education, and design. However, ordinary users still face challenges when it comes to creating and inputting content in MR, which need to be addressed in order to enhance the comprehensibility and convenience of content input methods in the MR environment. To tackle this issue, this paper proposes a user-centric content creation system in MR that utilises mechanical arm-based jitter reduction, size expansion, and intelligent control for path tracing. Ultimately, the system reconstructs the long-exposure photographs containing the traced light patterns in the MR environment. It opens up exciting and unexplored avenues for future research and development in areas such as public participation in urban planning, automotive decoration creation, and medical education.

This paper presents several open-ended questions for future investigation: (1) How can the user’s creative space be expanded throughout the entire content creation system to make it more inspiring and foster creativity, thus increasing user engagement in the creative process?

Light-Tracing: A Novel Approach for Mixed-Reality (MR) Content Creation 1067

(2) In the process of mixed reality creation, how can we better balance the roles of human users and mechanical arms to achieve a more effective creative mode?

(3) How can the system provide a diversified and personalised set of creative tools to meet the needs and artistic styles of different users?

ACKNOWLEDGMENT

This research was funded by the Hong Kong Polytechnic University’s Research Centre for Future (Caring) Mobility (P/ID: P0042701); and the University’s Strategic Importance Project (P/ID: P0036851). The authors would like to express their sincere gratitude to the University’s Master of Design, Intelligent Systems Design (ISD) Programme for providing the experimental equipment and facilities to conduct this research.

REFERENCES

Campeau-Lecours, A., Lamontagne, H., Latour, S., Fauteux, P., Maheu, V., Boucher, F.,... L’Ecuyer, L.-J. C. (2019). Kinova modular robot arms for service robotics applications. In Rapid Automation: Concepts, Methodologies, Tools, and Applications (pp. 693–719). IGI global. Carrasco, M. D. O., & Chen, P.-H. (2021). Application of mixed reality for improving

architectural design comprehension effectiveness. Automation in Construction, 126, 103677. Choi, S., Jung, K., & Noh, S. D. (2015). Virtual reality applications in manufacturing industries: Past research, present findings, and future directions. Concurrent Engineering, 23(1), 40–63. Darzentas, D. et al. (2018) ‘Object-focused mixed reality storytelling: technology-

driven content creation and dissemination for engaging user experiences’, in Proceedings of the 22nd Pan-Hellenic Conference on Informatics. [Online]. 2018 New York, NY, USA: ACM. pp. 278–281. De Benedetti, V., Shadili, G., Morel, A., Loisel, Y., Leculée, C., Smith- DiBasio, A. M., & Corcos, M. (2019). “Light Painting”. Writing of light and conversion of the look. A new mediation in the care of addictions in the adolescence. Neuropsychiatrie de l’enfance et de l’adolescence, 67(3), 158–168. https://doi.org/10.1016/j.neurenf.2019.01.003 Gawli, K., Karande, P., Belose, P., Bhadirke, T., & Bhargava, A. (2017). Internet of things (IoT) based robotic arm. Int. Res. J. Eng. Technol, 4(03). Hanzl, M. (2007). Information technology as a tool for public participation in urban

planning: a review of experiments and potentials. Design studies, 28(3), 289–307. Horn, M. S., Crouser, R. J., & Bers, M. U. (2012). Tangible interaction and learning: the case for a hybrid approach. Personal and Ubiquitous Computing, 16(4), 379–389. https://doi.org/10.1007/s00779-011-0404-2 Lee, H.-W. (2020). The study of mechanical arm and intelligent robot. IEEE Access,

8, 119624–119634. Morimoto, T., Kobayashi, T., Hirata, H., Otani, K., Sugimoto, M., Tsukamoto, M.,...

Mawatari, M. (2022). XR (extended reality: virtual reality, augmented reality, mixed reality) technology in spine medicine: status quo and quo vadis. Journal of Clinical Medicine, 11(2), 470.

Murray, J. T., & Johnson, E. K. (2021). XR content authoring challenges: The creator-developer divide. In Augmented and mixed reality for communities (pp. 245–264). CRC Press. OpenAI, 2024. ChatGPT (May 25 version) [Large language model]. [Online].

Available at: <https://chatgpt.com> [Accessed: 25 May 2024]. Ratican, J., Hutson, J., & Wright, A. (2023). A proposed meta-reality immersive development pipeline: Generative ai models and extended reality (xr) content for the metaverse. Journal of Intelligent Learning Systems and Applications, 15. Rohil, M. K., & Ashok, Y. (2022). Visualization of urban development 3D layout

plans with augmented reality. Results in Engineering, 14, 100447. Salamon, N. Z. et al. (2017) Computational Light Painting Using a Virtual Exposure.

Computer graphics forum. [Online] 36 (2), 1–8. Schraffenberger, H., & van der Heide, E. (2016). Virtual Reality and the leaping motion: Sketched-based 3D-modeling using gesture-based input. International Journal of Creative Interfaces and Computer Graphics (IJCICG), 7(1), 1–14. Smelik, R. M., Tutenel, T., Bidarra, R., & Benes, B. (2014). A survey on procedural

modelling for virtual worlds. Computer Graphics Forum, 33(6), 31–50. Tang, Y.-M., Au, K. M., Lau, H. C., Ho, G. T., & Wu, C.-H. (2020). Evaluating the effectiveness of learning design with mixed reality (MR) in higher education. Virtual Reality, 24(4), 797–807. Tung, F. W., & Cheng, Y. C. (2018). Mixed reality (MR) application in cultural

heritage education. Sustainability, 10(3), 908. Uhl, J. C. et al. (2023) ‘Tangible Immersive Trauma Simulation: Is Mixed Reality the next level of medical skills training?’, in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. [Online]. 2023 New York, NY, USA: ACM. pp. 1–17. Wang, T. et al. (2021) ‘LightPaintAR: Assist Light Painting Photography with

Augmented Reality’, in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. [Online]. 2021 New York, NY, USA: ACM. pp. 1–6. Ziker, C., Truman, B., & Dodds, H. (2021). Cross reality (XR): Challenges and opportunities across the spectrum. Innovative learning environments in STEM higher education: Opportunities, challenges, and looking forward, 55–77.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →