Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Environment Providing Necessary Information to Users Using Multiple IoT Avatars

Graduate School of System Design and Management, Keio University, Yokohama, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1007328

Abstract

Recently, smartphones have enabled people to access necessary information anytime, anywhere. However, it is difficult to obtain information in situations where smartphones or GPS cannot be used. We have developed IoT avatar technology as a method to obtain information from the surrounding environment where users exist. IoT avatar is a technology that provides information to users through the conversations with objects to which IoT devices are attached. Users in the field can communicate with remote users or generative AI through the objects. In particular, in this study, a multiple IoT avatar environment was constructed in which several IoT devices work together to provide continuous information to users such as walking guidance even when users move in the field. In the experiment, subjects were guided to the destination room by switching connections to multiple IoT avatar devices. The experimental results showed that multiple IoT avatars could provide continuous guidance to subjects in the field. We also confirmed that though subjects' consciousness of communicating with each object was decreased, they perceived the same personalities of the IoT avatars and were able to obtain continuous information.

INTRODUCTION

Due to the spread of smartphones, we can access necessary information anytime, anywhere. For example, when traveling to an unfamiliar city, we can use digital maps like Google Maps to easily find the route to the destination and search for information about that location. In addition, it is now possible to navigate users along a route using wearable devices such as AR glasses (Ahmed, 2017) (Gu, 2021).

However, for users who cannot recognize visual information or cannot follow normal routes, such as the visually impaired and wheelchair users, it is difficult to obtain information using usual smartphone applications (Ding, 2007). In shopping malls or underground streets where digital maps linked to GPS are not prepared, route guidance using a smartphone cannot be used. Also, in places where web information is not available or conditions change frequently, it is difficult to obtain necessary information from the web site using a smartphone.

24 Ogi and Kida Therefore, it is desired to develop an information provision method that provides necessary information to users who come to the places without making any special preparations. In particular, for users who cannot use smartphones effectively, such as visually impaired people or wheelchair users, a method that provides necessary information in the places in the same way as if an assistant were accompanying them would be effective.

In this research, we have constructed a multiple IoT avatar environment in which objects in the environment are personified and provide necessary information to the user by using IoT avatar technology. In this paper, the concept of the multiple IoT avatar, the system configuration, and the experiment on route guidance using the multiple IoT avatar environment are described.

CONCEPT OF MULTIPLE IoT AVATAR IoT avatar is a technology developed by the authors that personifies objects and have a conversation with users using IoT devices (Kida, 2024). There is research about personifying objects using metaverse spaces and mixed reality technology, but this research personifies real-world objects through natural conversation (Shao, 2019). Figure 1 shows the configuration of the IoT avatar device, and Figure 2 shows the usage of the IoT avatar device attached to a houseplant.

The IoT avatar device is constructed using a Raspberry Pi 4 and a 360-degree camera of Entaniya VR220, an USB microphone, and a Bluetooth speaker are installed in it. The device is compact, measuring 92mm x 63mm, and is attached to various objects exist in the real world.

Figure 1: IoT avatar device developed in this study.
Original PDF Figure 1:
Figure 2: Example of usage of IoT avatar attached to a houseplant.
Original PDF Figure 2:

Image captured by the 360-degree camera is sent through the internet to a remote user, who can then freely look around the transmitted video space using an HMD. This allows the remote user to feel as if his or her viewpoint is located at the camera position of the IoT device. Another unique feature is that the remote user can see the object itself as his or her own body in the 360-degree camera image, that generates the remote user’s feeling as if they are part of the object.

In addition, the IoT avatar device is equipped with a camera and microphone, that allows a natural conversation between the remote user and the local user in front of the IoT device. In this case, the remote user feels as if they are the object, and the local user feels as if they are having a conversation with the object in front of them.

IoT avatar devices can be connected not only to remote users via the internet, but also to generative AI to be used as AIoT avatar. The generatve AI for the IoT avatar is implemented as a web application that responds to user’s voice input in real time using the OpenAI Realtime Console and Realtime API.

When the generative AI is used for the IoT avatar, description like “I am a houseplant.” was given as a prompt so that the AI ​will converse as if it were an object to which the IoT device is attached and will have a conversation as a personified houseplant.

In this study, we aim to use IoT avatars to provide users with necessary information at tourist spots and other locations. Previous research has used avatar robots to provide information to users through conversation at locations such as airports and shopping malls (Fukabori, 2018) (Li, 2024) (Ota, 2025). However, in order to provide services over a wide area, many avatar robots are required, and it is a problem in terms of cost and installation space. On the other hand, IoT avatars are created by attaching IoT devices to objects in the real space, which means that the IoT avatars themselves cannot move. Therefore, we developed a method in which multiple IoT avatar devices are installed in the real environment, allowing the IoT avatars to work together and provide necessary information to users at the location.

Figure 3 shows the concept of a multiple IoT avatar environment. In this example, multiple IoT avatar environment provides wheelchair users with guidance information of route. When a user passes in front of a houseplant in a building, the houseplant provides information about the route they can take using wheelchairs, and as the user follow the guidance, the next pillar cooperates to provide information. This method allows wheelchair users to act on the location using information obtained from surrounding objects, without having to investigate the route they can take in wheelchairs in advance.

Figure 3: Concept of multiple IoT avatar environment.
Original PDF Figure 3:

DEVELOPMENT OF MULTIPLE IoT AVATAR ENVIRONMENT In this study, we developed a multiple IoT avatar system to realize the above concept. Figure 4 shows the configuration of the multiple IoT avatar communication system. For the communication between IoT avatar devices and remote users, SkyWay of NTT Docomo Business which uses WebRTC is used. The IoT avatar device and the remote user’s PC send and receive video and voice data in real time using web browsers via a P2P room in SkyWay. The 360-degree video sent from the IoT device to the remote user is texture mapped onto a virtual sphere implemented using A-Frame on the remote user’s PC, and the remote user can look around it using a HMD of Meta Quest 3.

In the 360-degree video of the IoT avatar seen by the remote user, the position of the IoT avatar next to it was represented as a spherical icon. When the remote user selects an icon in the field of view with the controller and presses a button, the connection to the SkyWay room is switched and then the connection destination of the IoT avatar is changed. This method allows the remote user to continue a conversation with the local user while switching the connection destination according to the movement of the local user. At this time, the IoT avatar device that the remote user has disconnected can be connected to the generative AI and be able to have a conversation with the next user.

360-degree video voice 360-degree video voice SkyWay Web app room1 voice voice IoT avatar1

HMD

local user

SkyWay

Web app

Web app

room2 IoT avatar2

PC remote user

SkyWay

Web app

room3 IoT avatar3 multiple IoT avatar voice

Generative AI voice

Figure 4: Configuration of the multiple IoT avatar communication system.

Figure available in the original PDF paper.

EXPERIMENT ON ROUTE GUIDANCE In this research, we conducted an experiment on providing walking guidance to users using a multiple IoT avatar environment. In the experiment, three IoT avatars were used being switched to provide walking guidance from the elevator hall to the destination room in the university building. The IoT avatar devices were attached to a disinfectant bottle placed in front of the elevator, a digital signage placed in the corridor, and the door of the destination room as shown in Figure 5.

In the experiment, the experimenter wore an HMD as a remote user, connected with each IoT avatar device, and provided directions while conversing as if he was the objects where the IoT avatar devices were attached. The number of experimental participants were 10, and they walked as local users from the elevator hall to the destination room being guided by IoT avatars through conversation. Figure 6 shows the remote user in the experiment and Figure 7 shows examples of the viewing image displayed on the remote user’s HMD.

After the experiment, participants were asked to answer the questions shown in Table 1 using a 7-point Likert scale about the feelings they felt while conversing with the IoT avatars.

Figure 5: IoT avatar devices attached to disinfectant bottle digital signage and door.
Original PDF Figure 5:
Figure 6: Remote user providing guidance using IoT avatars.
Original PDF Figure 6:

next IoT avatar device next IoT avatar device

Figure 7: Examples of image displayed on the remote user’s HMD.
Original PDF Figure 7:

EXPERIMENTAL RESULTS AND DISCUSSIONS

Figure 8 shows the results of the questionnaire. The graph shows the average and standard deviation of the participants’ responses to each question. From these results, we can see that the responses to questions “Q5. Were you able to continue the conversation?” and “Q6. Were you guided without getting lost?” were high, with scores of 6 or above. This means that the purpose of providing guidance using multiple IoT avatars was achieved smoothly. Additionally, the responses to questions “Q3. Did you feel you conversed with objects with the same personality?” and “Q4. Did you feel you conversed with remote users with the same personality?” were high, indicating that the conversation partners were perceived with the same personality and that a continuous conversation was performed.

On the other hand, the responses to “Q1. Did you feel you had a conversation with the object?” was significantly lower than those to “Q2. Did you feel you had a conversation with the remote user?”. In the previous experiment on the conversation with a single IoT avatar showed that participants felt significantly more they had a conversation with the object

Environment Providing Necessary Information to Users Using Multiple IoT Avatars 29 than with the remote user (Kida, 2025), but this experiment showed the opposite results. This is thought to be because the conversation time with each IoT avatar was short due to the task of providing walking guidance, and because the subjects’ perception of conversing with different objects was reduced due to the continuous conversation with multiple IoT avatars. In order to increase the consciousness of the conversation with each object while using multiple IoT avatars, it is necessary to devise methods such as changing the tone of voice according to the objects to create awareness that the objects are different, even if the personalities are the same. This issue must be addressed in the future.

Table 1: Questions asked to the subjects.
Original PDF Table 1:

Q1. Did you feel you had a conversation with the object?

Q2. Did you feel you had a conversation with the remote user?

Q3. Did you feel you conversed with objects with the same personality?

Q4. Did you feel you conversed with remote users with the same personality?

Q5. Were you able to continue the conversation?

Q6. Were you guided without getting lost?

7

6

answers

5

4

3

2

1

0

Q1 Q2 Q3 Q4 Q5 Q6 questions

Figure 8: Results of answers for questionnaire.

Figure available in the original PDF paper.

CONCLUSION

In this research, we developed a multiple IoT avatar environment as a method of providing information to users in the field and conducted an experiment on providing walking guidance to local users. As a result, we were able to achieve walking guidance through providing continuous information to users in the field, but we found a problem that the consciousness of having a conversation with each object was reduced due to the continuous conversation. In the future, methods of personifying objects with individuality of each object while continuing the conversation must be considered. In addition, since the purpose of the experiment was to achieve continuous guidance, the experiment only used the remote user based IoT avatars, but

30 Ogi and Kida future challenges include establishing a method of continuous guidance by switching conversation with generative AI based IoT avatars and remote user based IoT avatars. As a concrete future application, we are planning to apply this technology to barrier free tourist guides to provide information to disabled people.

ACKNOWLEDGMENT

This research was partially supported by JKA promotion funds from AUTORACE (2024M-549) and “Barrier-Free Tourism Support Program Utilizing Advanced Technology” in Tokyo Prefecture.

REFERENCES

Ahmed, D.B., Blanco, L.E.D., Diaz, E.M. (2017) “Performance comparison of wearable-based pedestrian navigation systems in large areas,” 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–7. Ding, D., Parmanto, B., Karimi, H.A., Roongpiboonsopit, D., Pramana, G., Conahan,

T. (2007) “Design Considerations for a Personalized Wheelchair Navigation System”, 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Fukabori, A., Kajitani, K. (2018) “ANA AVATAR”, Journal of the Robotics Society of Japan, Vol.36, No.10, pp. 663–667 (in Japanese). Gu, Y., Chidsin, W., Goncharenko, I. (2021) “AR-based Navigation Using Hybrid

Map”, 2021 IEEE 3rd Global Conference on Life Sciences and Technologies (LifeTech). Kida, Y., Chiba, T., Kinoshita, T., Mikami, A., Yakoh, T., Ogi, T. (2024) “IoT Avatar: Turning Various Objects into Avatars”, The 27th International Conference on Network-Based Information Systems (NBiS 2024/INVITE 2024), pp. 398–405. Kida, Y., Ogi, T. (2025) “IoT Avatar: Various Objects in Real Space are

Anthropomorphized as Avatars”, International Journal of Web and Grid Services, Vol. 21, No. 1, pp. 42–57, Inderscience. Li, Z., Yan, L., Yu, Z., Li, B. Li, J., He, Y. (2024) “Overall Design and Physical Validation of Voice Interaction Based on the ChatGPT Humanoid Robot Brain”, 2024 Asian Conference on Communication and Networks (ASIANComNet). Matsuoka, K., Kida, Y., Kato, T., Ogi, T. (2024) “Multilingual Communication

Robot Using Language Detection System”, The 20th International Conference on Machine Design and Production (UMTIK 2024), pp. 91–100. Ogi, T., Kida, Y. (2024) “Human Relationship Felt in Communication Using Tele- Immersion Avatar Robot”, The 20th International Conference on Machine Design and Production (UMTIK 2024), pp. 101–110. Ota, T., Nishimura, T., Takeuchi, K., Yoshifuji, O., Hatada, Y.; Tanaka, Y. (2025)

“Training of GUI-Based Avatar Robot Operation Through Sharing Operation with Expert”, 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). Shao, Y., Lessio, N., Morris, A. (2019) “IoT Avatars: Mixed Reality Hybrid Objects for CoRe Ambient Intelligent Environments”, Procedia Computer Science, Vol. 155, pp. 433–440, Elsevier.

Cover of the proceedings volume Affective and Pleasurable Design

Explore the complete digital volume

Affective and Pleasurable Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-80-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →