Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Effects of Robot Non-Verbal Behaviors on Human Emotion Recognition in Human–Robot Communication

Faculty of Intelligence and Informatics, Konan University, Kobe, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1007680

Abstract

Social robots are increasingly used in daily environments, where effective emotional communication is essential for smooth human–robot interaction. This study investigates whether robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken narratives.Narrative-based emotional messages representing basic emotions derived from Plutchik’s emotion wheel were constructed and validated in a preliminary experiment with 132 participants. A main experiment was then conducted with twelve adult participants. Emotional narratives were presented under two conditions: with and without robot non-verbal behaviors expressing corresponding emotions. The robot executed predefined gestures, body movements, and gaze behaviors synchronized with key emotional sentences. After each presentation, participants rated perceived emotional content using a five-point emotion recognition scale. Results showed significantly higher emotion recognition scores when robot non-verbal behaviors were present in both joy and sadness conditions. These findings indicate that appropriate robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken messages. From a human factors perspective, this study provides design implications for developing social robots that support intuitive emotional communication.

INTRODUCTION

Social robots are increasingly integrated into daily environments such as public guidance, education, and customer service. In these contexts, smooth emotional communication between humans and robots is essential for building trust, acceptance, and engagement. While verbal expressions and facial displays in human–robot interaction have been widely studied, the contribution of robot non-verbal behaviors, such as gestures, body movements, and gaze, to human emotion recognition has not been sufficiently clarified (Breazeal, 2003; Mutlu et al., 2009).

In human–human communication, non-verbal cues play a crucial role in conveying emotional information and supporting accurate interpretation of spoken messages. Similar mechanisms are expected in human–robot communication; however, empirical evidence quantifying how robot non- verbal behaviors influence human emotion recognition remains limited.

78 Yamanaka From a human factors perspective, understanding these effects is important for designing communicative robot interfaces that support intuitive and low- load interaction.

The objective of this study is to investigate whether robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken narratives. Specifically, we examine the effects of synchronized robot gestures, body movements, and gaze behaviors on perceived emotion recognition. By experimentally comparing narrative presentation with and without robot non-verbal behaviors, this study aims to clarify the role of expressive robot motions in emotional information transmission.

The contributions of this study are threefold: (1) construction and validation of narrative-based emotional stimuli for human–robot communication experiments; (2) experimental evidence demonstrating that robot non-verbal behaviors significantly enhance human emotion recognition; and (3) design implications for expressive behavior generation in social robots.

RELATED WORK

Emotional communication has been a central topic in human–robot interaction research (Breazeal, 2003; Tian et al., 2020). Previous studies have explored how robots express emotions through facial displays, speech prosody, and body motions to improve social presence and user engagement (Saerbeck et al., 2010). Breazeal (2003) demonstrated that sociable robots equipped with expressive behaviors can foster natural interaction with humans. Castellano et al. (2010) further showed that body movements play a significant role in recognizing human emotions, suggesting that non-verbal cues are essential channels for emotional information transmission.

In the context of human–robot communication, several studies have examined robot gestures and gaze behaviors as mechanisms for guiding user attention and enhancing interaction quality. Robot gaze has been reported to influence human perception of attentiveness and communicative intent, while gesture synchronization with speech can support message understanding (Huang and Mutlu, 2013; Mutlu et al., 2009). However, quantitative evaluation of how robot non-verbal behaviors specifically influence human emotion recognition of spoken content remains limited.

To construct emotional stimuli, theoretical emotion models such as Plutchik’s emotion wheel have been widely adopted to represent basic emotions in affective computing and human–robot interaction research (Plutchik, 1980; Russell, 1980). Plutchik’s model provides a structured framework for designing and classifying emotional expressions and has been applied in studies on emotional dialogue systems and expressive robot behaviors.

Building on these findings, the present study experimentally investigates whether robot non-verbal behaviors enhance human recognition of emotional narratives. By combining narrative-based emotional stimuli with synchronized robot gestures, body movements, and gaze behaviors, this work extends previous research by providing empirical evidence on the effectiveness of non-verbal behaviors for emotional information transmission in human–robot communication.

METHOD

Two experiments were conducted. First, emotional narrative stimuli were constructed and validated. Second, the effects of robot non-verbal behaviors on human emotion recognition were examined through a comparative experiment.

  • Construction and validation of emotional narratives Eight basic emotions based on Plutchik’s emotion wheel were selected as target emotions. Narrative-based emotional messages were generated to convey each target emotion through spoken text. To avoid potential gender bias, separate versions were created using male and female university student perspectives. A preliminary validation experiment was conducted with 132 participants, who listened to each narrative and rated perceived emotional content using a five-point scale. The results confirmed that narratives for joy and sadness reliably elicited the intended emotional impressions. Based on these results, joy and sadness narratives were selected for the main experiment.
  • Design of robot non-verbal behaviors A tabletop social robot was used to present non-verbal behaviors synchronized with emotional narratives. For each selected emotion, predefined gestures, body movements, and gaze behaviors were designed to express corresponding emotional states. The robot behaviors were temporally aligned with key emotional sentences in the narratives to reinforce emotional content. In the control condition, narratives were presented without any robot movements.
Figure 1: Experimental setup for presenting emotional narratives with robot non- verbal behaviors.
Original PDF Figure 1:
  • Experimental procedure Twelve adult participants (six male and six female) took part in the main experiment. Emotional narratives were presented under two conditions:

80 Yamanaka (1) narrative with robot non-verbal behaviors, and (2) narrative without robot behaviors. The order of conditions was counterbalanced across participants. After each presentation, participants rated perceived emotional content using a five-point emotion recognition scale. The experiment consisted of practice trials followed by experimental trials, and the entire session lasted approximately 20 minutes per participant.

Figure 2: Procedure of the main experiment for emotion recognition with and without robot non-verbal behaviors.
Original PDF Figure 2:

RESULTS

The effects of robot non-verbal behaviors on human emotion recognition were analyzed for joy and sadness narratives. Figures 3 and 4 show the comparison of emotion recognition scores between the robot condition and the no-robot condition.

Figure 3: Comparison of emotion recognition scores for joy narratives with and without robot non-verbal behaviors.
Original PDF Figure 3:
Figure 4: Comparison of emotion recognition scores for sadness narratives with and without robot non-verbal behaviors.
Original PDF Figure 4:

For the joy narratives, participants reported higher emotion recognition scores when robot non-verbal behaviors were present compared to the no-robot condition. Statistical analysis using paired t-tests indicated a marginally significant difference between the two conditions (p < 0.10). This result indicates that synchronized robot gestures, body movements, and gaze effectively enhanced recognition of joyful emotional content.

Similarly, for the sadness narratives, emotion recognition scores were significantly higher in the robot condition than in the no-robot condition (p < 0.05). Participants reported that robot behaviors helped them better interpret the emotional tone of the spoken messages.

In addition to overall score differences, partial confusion between neighboring emotions on Plutchik’s emotion wheel was observed in both conditions. However, the robot condition consistently showed reduced misinterpretation compared to the no-robot condition.

These results demonstrate that robot non-verbal behaviors significantly enhance human recognition of emotional content conveyed through spoken narratives.

DISCUSSION

The experimental results demonstrate that robot non-verbal behaviors significantly enhance human recognition of emotional content conveyed through spoken narratives. This finding supports the hypothesis that expressive robot gestures, body movements, and gaze function as complementary communication channels, reinforcing emotional information in a manner analogous to non-verbal cues in human–human interaction (Castellano et al., 2010; Huang and Mutlu, 2013).

From a human factors perspective, the presence of robot non-verbal behaviors appears to reduce ambiguity in interpreting emotional messages. Participants reported higher confidence in recognizing emotional content when robot behaviors were synchronized with key emotional sentences. This suggests that appropriately designed robot motions can support intuitive emotional communication and potentially lower cognitive effort required for emotional interpretation.

82 Yamanaka Partial confusion between neighboring emotions on Plutchik’s emotion wheel was observed, indicating that emotional categories with similar expressive features may be difficult to distinguish solely through non-verbal cues. This highlights the importance of carefully designing robot expressive behaviors to emphasize distinctive motion characteristics for each target emotion and avoid unintended misinterpretation.

These findings provide practical design implications for social robots intended for communication support, education, and customer service. Incorporating synchronized non-verbal behaviors can improve emotional information transmission, user engagement, and perceived communicative effectiveness. However, the present study focused on only two emotions and subjective evaluation measures. Future work will extend the range of emotions examined and integrate objective behavioral indicators such as gaze patterns and response time to further clarify human–robot emotional communication mechanisms.

CONCLUSION

This study investigated the effects of robot non-verbal behaviors on human emotion recognition in human–robot communication. Narrative-based emotional messages were presented with and without synchronized robot gestures, body movements, and gaze behaviors, and participants evaluated perceived emotional content.

Experimental results showed that emotion recognition scores were significantly higher when robot non-verbal behaviors were present for both joy and sadness narratives. These findings indicate that appropriate robot non-verbal behaviors enhance human recognition of emotional content conveyed through spoken messages and function as effective complementary channels for emotional information transmission.

From a human factors perspective, the results suggest that integrating expressive non-verbal behaviors into robot design can support intuitive emotional communication, improve user engagement, and increase perceived communicative effectiveness. The present study provides empirical evidence and design implications for developing social robots that facilitate emotional understanding in human–robot interaction.

Future work will expand the range of emotions examined and incorporate objective behavioral measures such as gaze analysis and response time to further clarify the mechanisms of emotional communication between humans and robots.

ACKNOWLEDGMENT

This work was partially supported by the Konan Digital Twin Research Institute. The author would like to express sincere gratitude for the research support and facilities provided.

REFERENCES

Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1–2), 119–155. Castellano, G., Villalba, S. D., & Camurri, A. (2010). Recognizing human emotions

from body movements and gestures. International Journal of Human-Computer Studies, 59(1–2), 1–13. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. Huang, C.-M., & Mutlu, B. (2013). Modeling and evaluating narrative gestures for

humanlike robots. Proceedings of Robotics: Science and Systems. Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., & Hagita, N. (2009). Footing in human– robot conversations: How robots might shape participant roles using gaze cues. Proceedings of CHI, 61–68. Plutchik, R. (1980). Emotion: A Psychoevolutionary Synthesis. Harper & Row, New

York. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. Saerbeck, M., Schut, T., Bartneck, C., & Janse, M. D. (2010). Expressive robots in

education: Varying the degree of social supportive behavior of a robotic tutor. Proceedings of CHI, 1613–1622. Salam, H., Chetouani, M., Oviatt, S., Mahdhaoui, A., & Dutoit, T. (2016). Nonverbal behavior detection and analysis in social interaction: A review. IEEE Transactions on Affective Computing, 7(2), 117–134. Tian, L., Oviatt, S., & Chen, F. (2020). Multimodal human–robot interaction: A

survey of machine perception and affective computing. ACM Computing Surveys, 53(3), 1–36.

Cover of the proceedings volume Human Factors in Robots, Drones and Unmanned Systems

Explore the complete digital volume

Human Factors in Robots, Drones and Unmanned Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-94-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →