Yate Ge, Yuanda Hu, Xiaohua Sun (2023). Co-Design of Service Robot Applications Using Virtual Reality. In: Human Factors in Virtual Environments and Game Design. AHFE Open Access, vol. 96, pp. 65–73. https://doi.org/10.54941/ahfe1003868
Service robots have been applied in an increasing number of scenarios, including homes, hospitals, offices, schools, hotels, etc. To ensure the usability of the interaction interface and process in the application of service robots, the design of human-robot interaction in the application of service robots involves various interaction modalities, different robot forms, and different physical behaviors of robots in space, etc. This makes it challenging to have low-cost, high-fidelity prototype methods that support design exploration in the early stages of service robot application design.Currently, many rapid prototyping techniques have been applied to the design exploration stage in the design of service robot applications, such as paper prototypes, storyboards, video prototypes, etc. However, these methods have limitations, including low fidelity and being out of the environmental context. Researchers have also been exploring new prototype methods to meet the design exploration and testing requirements for HRI design. Some studies have explored the use of VR test HRI prototypes, and most of them focus on the technical aspect, and the performance of HRI capabilities. Other studies focus on specific HRI design aspects: interactive mechanism, anthropomorphic appearance, social acceptance, and so on. Approaches focusing on these aspects are not suitable for prototyping and testing the overall interaction scenarios of service robot applications.Based on the characteristics of virtual reality technology, this paper aims to explore how virtual reality technology can be used to support multi-user collaborative design of service robot applications in a virtual environment. To address this issue, this paper proposes a system for supporting collaborative design of service robot applications. The system framework and implementation will be described in detail in the paper. Specifically, the system enables multiple users (designers or stakeholders) to enter a virtual environment in real-time using head-mounted VR devices. Users can select appropriate environment model assets based on the target application scenario and perform bodystorming "on site". Using a modular robot building tool, users can add virtual robots to the space and add or delete functional components, as well as adjust the position, size, and orientation of the components. The system's Wizard-of-OZ module allows users to control the robot's movement and component status. The Graphic UI is embedded into the physical display of the robot via WebView and supports the simulation of the Graphic UI interaction process. After completing the initial application concept ideation and virtual robot design, users can use WoZ and Role-Playing techniques to perform the human-robot interaction process to evaluate and optimize the interaction design. In addition, the recorded video of the performance can also support subsequent design discussions.The system is implemented using the Unity game engine, and users interact with the system using the Oculus Quest headsets and controllers. The design activities case based on the system will be evaluated and discussed to analyze the strengths and weaknesses of the system. Further, we discuss the limitations of this work and the future research directions for supporting the design of service robot applications using virtual reality.
Service robots have been implemented in various scenarios, including dome- stic and public service settings such as hospitals, offices, hotels, and restau- rants. For the successful implementation of service robot applications, pro- duct feasibility and user experience are crucial. It is essential not only to consider the functional implementation of the robot, but also to take into account the characteristics of the target users and the service scenarios as service robots are a key service carrier within a service system (Belanche et al., 2020). Hence, it is crucial to carry out design exploration in the initial phases of service robot application design to identify the factors that influence the design of the application, and to explore “point designs” - a possible point in the design space, often in the form of a working prototype that can be tested (Dow et al., 2013). By exploring these point designs, one can establish fur- ther design objectives and development requirements during the early stages of the research and development process.
In service robot design, many rapid prototyping techniques, such as story- boards, cardboard prototypes, and video prototypes, have been commonly used in the design exploration stage. However, these methods have limita- tions, such as low fidelity and a lack of environmental context. Therefore, researchers are currently exploring new methods for designing exploratory prototypes that can address these limitations. Various studies have inve- stigated the utilization of virtual reality (VR) technology for human-robot
interaction (Wijnen et al., 2020; Herzog et al., 2022; Bazzano et al., 2016; Franco dos Reis Alves et al., 2021; Babel et al., 2022). These methods com- bine VR and simulation technologies to validate HRI techniques or to study specific HRI design problems. The potential of VR technology to provide simulations of the environment, robots, human-robot interactions makes it a promising method for concept ideation and prototype development in the design exploration phase of service robot applications.
This study explores how to use VR technology to support the co-design of service robot applications. Co-design involves the participation of vari- ous stakeholders such as designers, engineers, domain experts, and users to collaboratively explore both the problem space and solution space (Eastman and Computing, 2001). The development of service robots requires consi- deration of numerous factors. By using a co-design approach in the design exploration stage, design problems can be viewed from different perspectives, leading to a better understanding of the problem space and effective explora- tion of the design space. In the HRI field, co-design has been widely applied by incorporating users into the design process to inform the development of robots that meet their needs and preferences, ensuring that service robots are responsibly designed to engage in social contexts (Vaziri et al., 2020). VR technology has been extensively applied in design practices across various industries, including automotive design, interior design, and product design. During co-design activities, it is crucial to incorporate external design activi- ties to facilitate the presentation of participants’ ideas and achieve cognitive alignment among collaborators. Designers use external activities to materia- lize their internal representations (Eastman and Computing, 2001), thereby making it easier for collaborators to understand and align their cognitive perspectives. The integration of VR and simulation technologies offers a straightforward way for executing external design activities and facilitating communication among participants.
In this paper, we explore co-design for service robot applications in virtual reality environments. We propose a VR co-design framework based on the Bodystorming, an embodied design method, and describe the development of a corresponding prototype system. Additionally, the use cases of this system are presented and discussed.
VR FRAMEWORK FOR COLLABORATIVE SERVICE ROBOT APPLICATION DESIGN Our method is founded on the theoretical principles of bodystorming, which is a type of brainstorming where participants use their bodies to gain physical experiences, gain contextual understanding, and ideate new products or servi- ces (Oulasvirta et al., 2003). This immersive method for co-design is based on embodied cognition theory and leverages techniques such as role-playing and physical interaction, utilizing props, prototypes, and physical spaces to simulate the application scenarios of emerging technologies (Boletsis et al., 2017; Oulasvirta et al., 2003; Schleicher et al., 2010). Our proposed framework leverages the features of multi-user immer- sive virtual reality to facilitate the execution of co-design for service robot
Co-Design of Service Robot Applications Using Virtual Reality 67
applications by means of bodystorming in a virtual environment. Virtual bodystorming allows for co-design to be carried out in an affordable, highly engaging manner (Boletsis et al., 2017). Our VR system can provide an immersive setting by simulating Service robot application environments and related objects, allowing for multi-user collaborative bodystorming in a vir- tual environment. Moreover, users have the ability to manipulate the VR environment and simulate different modes of human-robot interaction. There are two primary benefits of co-designing in a VR environment: Firstly, it eliminates environmental constraints, facilitating convenient simulation of application environments and remote multi-user collaboration. Secondly, it allows for rapid prototyping of robots’ appearance design and human-robot interaction.
Our framework comprises two types of design activities, as shown in Figure 1. These two activity types are closely interlinked and continuously cycle between “ideation-validation” until insights are obtained from the co- design process. The first type of design activity involves collaborative ideation and prototyping of service scenarios and robots in the application context. The second type of design activity involves participants engaging in role- playing as robots and users to create a performative prototype, validating the design concepts generated in the first type of design activity. To support these two design activities, necessary VR tools are required:
1. Scene editing tool: In order to simulate the target application scenarios and allow participants to physically experience the limiting factors of the scene and gain design inspiration, the system needs to provide tools for selecting environments and editing objects in the environment, to sup- port participants in jointly simulating the application scenarios of service robots. 2. Robot prototyping tool: To facilitate the rapid prototyping of service robots in a virtual environment and aid participants in evaluating their
Figure 1: VR framework for co-design of service robot applications.
appearance design, interaction characteristics, and hardware limitations, the system must provide a tool that enables the creation and modification of robot appearance, while also supporting the simulation of human- robot interfaces and interactions. 3. Avatar tool: In collaborative virtual reality, users’ avatars can foster social presence and improve cognitive synchronization among participants. Additionally, the presentation of a service robot application scenario pro- totype requires the selection of avatars based on user type, who will then perform their respective behaviors. As a result, the avatar tool must be capable of replacing different types of avatars and synchronizing users’ postures through motion tracking. 4. Robot Wizard of Oz tool: In order to simulate the behavior of a service robot for role-playing purposes, the Wizard of Oz tool should support intuitive interaction in a VR environment to control the robot’s position, movements of its components, screen displays, and other aspects.
Design Activity 1: Ideating and Making In this design activity, Participants select the appropriate environmental model based on the target application scenario and immerse themselves in the environment to gain an intuitive understanding of the target service robot application scenario. The shared environment allows for synchronizing infor- mation among collaborators in an intuitive manner, while also providing the flexibility to explore the environment and generate new ideas. With the modular robot prototype tool, participants can directly prototype the service robot in the application scenario. From a user-centric perspective, this activity can support critical aspects during the design exploration stage, such as une- arthing key environmental factors and limitations, exploring robot forms and interaction modes that meet application requirements, and exploring feasible interaction modes and interfaces. By engaging in ideation and collaborative creation within a shared VR environment, participants can discuss design points, generate potential robot forms, and prototype in a collaborative and efficient manner.
Design Activity 2: Role-Playing Role-playing is an essential component of bodystorming, especially in service design, where the entire service process can be presented in the form of a per- formance (Boletsis et al., 2017). In the context of service robot applications, the robot’s interaction process can also be presented through role-playing. In this activity, participants will play the roles of both users and service robots. By treating the service robot as a humanoid character, we can fully consi- der the user experience of service robot applications. Through role-playing, the key experiences and design points in the service process can be fully explored, and specific system development requirements can be identified. Additionally, role-playing enables continually iterate on the proposed solu- tions during the ideation process. The iterated solutions can be recorded as performance-based prototypes and presented to more stakeholders to obtain feedback.
Co-Design of Service Robot Applications Using Virtual Reality 69
SYSTEM As described in the previous section, our proposed approach is a multi-user collaborative VR system that requires the development of tool components to support co-design of service robot applications in VR. We selected the Oculus Quest Headset as our VR device, as it can run VR applications independen- tly, thereby reducing the cost of user participation. For system development, we chose the Unity Engine and programmed in C#. The scalability of Unity and its support for VR allowed us to conveniently implement the necessary features. The overall system framework is depicted in Figure 2.
User Interaction: Our Avatar Tool is built on the Meta Avatar SDK provi- ded by Oculus and offers users 16 avatar options to choose from. The Oculus Quest’s 6-DoF tracking system, along with the Oculus Touch controllers, allows for real-time tracking of user avatars’ head and hand movements. Through the use of inverse kinematics algorithms, the system can perform real-time calculations of users’ skeletal data, enabling the real-time rendering of users’ virtual movements. Participants can interact with 2D interfaces and the 3D environment, utilizing the VR tools provided to explore the virtual space, build robot prototypes, and engage in role-playing activities.
Immersive Environment: Unity engine is one of the most commonly used 3D game development engines. Its editor facilitates the effortless production and modification of 3D environments, and the engine’s rendering pipeline
Figure 2: VR system for co-design of service robot applications.
supports high-precision rendering. Moreover, Unity is a favored choice for constructing immersive virtual reality environments, as evidenced by the numerous VR applications developed using this engine. We utilize Unity to simulate service robot application environments.
Networking: Our data synchronization among multiple users is facilitated by Photon Unity Networking (PUN), which enables remote network synch- ronization or local network server deployment. Our current system employs a local network server, where a PC operates the project program as the server. The Oculus Quest on the user side connects to the server through the local network, enabling enabling real-time data synchronization.
Assets Management: Our asset management system utilizes the Unity Addressable Asset System, which facilitates managing assets in the Unity editor and allows for loading assets at runtime. Our system provides envi- ronment assets, item assets, and robot module assets for co-design purposes, which have different asset requirements based on the specific usage scenario. Therefore, we implement a flexible asset management approach and integrate hot loading on the user application side.
Creating the environment for the target service robot application scena- rio is a crucial part of the ideation process. In our system, participants can switch between environment models and add item props through a 2D UI (see Figure 3). Co-design organizers can load custom scene and object models in the Unity editor, according to the requirements of the service robot application design scenario.
The robot prototyping tool allows for the addition and modification of robot modules, including the head, body, chassis, and arms, as well as inte- ractive modules such as display screens and indicator lights (see Figure 4). Users can edit the size and position of each module. Additionally, users can custom their own modules via the asset management system. Throughout the co-design process, participants can use the Wizard of Oz tool to control the robot and engage in role-playing as needed.
Figure 3: UI for avatar selection (left), environment (middle), and props (right).Figure 4: Robot prototype tool (left and middle); role-playing activity observed via the unity editor (right).
Co-Design of Service Robot Applications Using Virtual Reality 71
CASE We invited three groups of university students, including majors in interaction design, service design, and industrial design, computer science, and electronic engineering, to experience our prototype system. Before each group began co-designing, we worked with the participants to establish an initial design direction and provide training on how to operate the system.
The study designed three different service robot applications (see Figure 5). The first is a social robot functioning as a learning companion. It utilizes a camera to detect the user’s learning status and provides feedback through voice and screen. The second application is a mall guide robot that com- municates with users through voice and interface. The robot’s expression is displayed on the top screen, while the mall map information is on the bottom screen. The size and position of the screens were adjusted and con- firmed repeatedly by the participants using robot prototyping tools. The third application involves a supermarket shopping robot that was intended to recommend products, navigate, and transport goods. However, during the ideation activity in the virtual supermarket environment, they found that there were many limitations in the scene, such as limited space, dynamic chan- ges in the environment, and unclear user goals. This made the application design face many problems in both functionality and interaction that needed to be solved.
In these cases, the participants used our system to create a prototype of a physical form for a robot. Through role-playing, they demonstrated a perfor- mative prototype to discuss the design concepts for service robot applications, as well as to understand the constraints and opportunities for such applica- tions. Through our method, participants were able to explore the contextual environment, consider the requirements and design factors for service robot applications in real-life scenarios, and generate design ideas through physi- cal cognition. However, this method still has many drawbacks. For example, the functions and forms of service robots can vary greatly across different application scenarios, which requires robot prototyping tools to have diverse robot component modules and simulations for functional implementation to better identify user experience issues. Our current system can only simu- late basic functions such as position movement, screen display, and indicator light changes, but cannot simulate functions such as object recognition and manipulation. These functions are closely related to the feasibility and user experience of service robot applications. In addition, we currently only use static 3D models to simulate application environments, but real spaces are
Figure 5: Service robot application prototype.
dynamic and there are different types of people engaged in various activi- ties within the space. Our current method is limited to static environments, which results in significant differences between the simulated and actual envi- ronments. These differences may lead users to overlook many considerations for scene factors when using our system for ideation.
CONCLUSION
We propose a VR-based co-design method for service robot applications. This method supports co-design participants to physically experience the applica- tion context through virtual reality and simulation, and generate design ideas through robot prototyping and role-playing to explore the constraints and opportunities of service robot application design. We develop a prototype system and present use cases of this system. Our research can serve as an example for researching how to design service robots using VR technology. Our goal is to propose a co-design method that combines VR technology and embodied design methods in an affordable and highly engaging way. Our next step is to further study how to fully utilize human embodied cognition to optimize our method. We will also evaluate our method through quantitative research.
REFERENCES
Babel F, Vogt A, Hock P, et al. (2022) Step Aside! VR-Based Evaluation of Adaptive Robot Conflict Resolution Strategies for Domestic Service Robots. International Journal of Social Robotics 14(5): 1239–1260. Bazzano F, Gentilini F, Lamberti F, et al. (2016) Immersive Virtual Reality-Based
Simulation to Support the Design of Natural Human-Robot Interfaces for Service Robotic Applications. In: Augmented Reality, Virtual Reality, and Computer Gra- phics (eds LT De Paolis and A Mongelli), Cham, 2016, pp. 33–51. Lecture Notes in Computer Science. Springer International Publishing. Belanche D, Casaló LV, Flavián C, et al. (2020) Service robot implementation: a theoretical framework and research agenda. The Service Industries Journal 40(3–4): 203–225. Boletsis C, Karahasanovic A and Fjuk A (2017) Virtual Bodystorming: Utilizing Vir-
tual Reality for Prototyping in Service Design. In: De Paolis LT, Bourdot P, and Mongelli A (eds) Augmented Reality, Virtual Reality, and Computer Graphics. Lecture Notes in Computer Science. Cham: Springer International Publishing, pp. 279–288. Dow S, Ju W and Mackay W (2013) Projection, Place, and Point-of-View in Research Through Design. In: The SAGE Handbook of Digital Technology Research. 1 Oli- ver’s Yard, 55 City Road, London EC1Y 1SP United Kingdom: SAGE Publications Ltd, pp. 266–285. Eastman C and Computing D (2001) New Directions in Design Cognition. In: Design
Knowing and Learning: Cognition in Design Education. Elsevier, pp. 147–198. Franco dos Reis Alves S, Uribe Quevedo A, Chen D, et al. (2021) Leveraging Simulation and Virtual Reality for a Long Term Care Facility Service Robot During COVID-19. In: Symposium on Virtual and Augmented Reality, Virtual Event Brazil, 18 October 2021, pp. 187–191. ACM.
Co-Design of Service Robot Applications Using Virtual Reality 73
Herzog O, Forchhammer N, Kong P, et al. (2022) The Influence of Robot Designs on Human Compliance and Emotion: A Virtual Reality Study in the Context of Future Public Transport. ACM Transactions on Human-Robot Interaction 11(2): 21:1-21:17. Oulasvirta A, Kurvinen E and Kankainen T (2003) Understanding contexts by being
there: case studies in bodystorming. Personal and Ubiquitous Computing 7(2): 125–134. Schleicher D, Jones P and Kachur O (2010) Bodystorming as embodied designing.
Interactions 17(6): 47–51. Vaziri D, Golchinfar D, Stevens G, et al. (2020) Exploring Future Work - Co- Designing a Human-robot Collaboration Environment for Service Domains. In: Proceedings of the 2020 ACM Designing Interactive Systems Conference, Eindhoven Netherlands, 3 July 2020, pp. 153–164. Wijnen L, Bremner P, Lemaignan S, et al. (2020) Performing Human-Robot Intera-
ction User Studies in Virtual Reality *. In: 2020 29th IEEE International Confere- nce on Robot and Human Interactive Communication (RO-MAN), Naples, Italy, August 2020, pp. 794–794. IEEE.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Virtual Environments and Game Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-72-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional