Tianyu Hou, Nicoletta Adamo, Nicholas Villani (2022). Micro-expressions in Animated Agents. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe1001081
The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman & Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., & Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., & Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., & Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/
The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness.
Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman, 2009). Prior studies show that micro-expressions may have important effects on viewer’s responses. For instance, a study by Stewart et al. (2009) “illustrates that the presence of facial micro-expressions in political speeches affects emotional response to the speech” (p. 130). Research by Russell et al. (2006) suggests that patients with schizophrenia improved their emotion recognition with “micro-expressions training tool (METT)”. Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness, intensity and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).
PRIOR WORK ON MICRO-EXPRESSIONS IN HUMANS AND ANIMATED CHARACTERS Micro expressions are rapid involuntary facial expressions that conceal suppressed (de- liberate) or repressed (non-deliberate) emotions (Ekman, 2009). Micro expressions reveal one of the six universal emotions commonly referred to as disgust, anger, fear, sadness,
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 happiness, or surprise (Shen, Wu, & Fu, 2012). These expressions range in the amount of time they are present, between 1/3 to 1/25 of a second (Ekman, 2009). Shen, Wu, & Fu (2012) concluded in a study on the duration of micro expressions that the upper limit of duration is around 1/5 of a second. Although originally described as not visible in real time (Haggard & Isaacs, 1966), Ekman (2009) found it was possible to observe micro facial expressions in real time without the use of a camera, despite their fleeting duration.
Micro expressions are considered to reflect the true intent of a person, sometimes of hostile nature (Ekman et al., 1992). Micro expressions are often elicited when someone is being deceptive, therefore it has been proposed that micro expressions may provide an auspicious method for detecting deception (Ekman, 2009). Further, micro expressions appear when someone is trying to conceal genuine emotion(s) (Yan et al., 2013). The detection of micro expressions may reveal accurate suppressed emotion, helping to better understand how one is truly feeling.
Few prior studies have focused on representation and recognition of micro-expressions in animated characters (Zielke & Hardee, 2011; Queiroz et al., 2014). The most relevant prior study is the one by Queiroz et al. (2014) which examined human perception of micro- expressions in animated virtual faces. Findings of the experiment showed that most of the subjects recognized the animated faces’ (macro) emotion and most of the time they perceived the micro-expression, but did not identify it correctly in some samples. Although the authors did not consider factors such as character visual style and degree of exaggeration of the micro-expressions, their work has inspired our research. Findings from their study suggest that conveying micro-expressions in animated agents may visually enhance the agents’ emotional depth and therefore increase the perceived social complexity and believability.
METHODS
The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic).
The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, and the extent to which exaggerating the micro expressions, e.g., increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional intensity. Further, the study examined whether there are differences in perception based on the agent’s design characteristics. The same group of subjects participated in both studies.
The studies used a within subject design and a quantitative research approach. Data was collected in the form of answers to rating questions, which asked participants to rate the naturalness and intensity of the emotions showed by the animated characters on a 1-5 Likert scale (1=low and 5=high). The studies also collected data in the form of correct/incorrect
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 answers to questions which asked participants to select the emotion that they thought the character was truly feeling. The independent variables were the animated character visual style (realistic versus stylized) and the degree of exaggeration of micro-expression (normal/exaggerated), the dependent variables were true emotion recognition, rating of naturalness, and rating of intensity.
SUBJECTS The sample included 275 college students recruited via email; 207 were male, 64 female, 4 non-binary/unwilling to disclose; the age range was 18-25 years old. All subjects partici- pated in both studies.
STIMULI, INSTRUMENTS AND PROCEDURE The researcher used Maya software and two characters downloaded from internet to develop the 3D animations. One character is stylized and the other one is realistic (figure 1). The two characters were rigged with identical facial skeletal deformation systems and the animations were created based on the Facial Action Coding System (FACS). Table 1 shows the Action Units (AU) and head movements used for each emotion.
The stimuli included three sets of eight animations, four per character in each set (24 clips in total). The first set of animations featured the agents performing micro-expressions only and was used in study 1 to examine whether the subjects could recognize the micro- expressions in isolation. Each animation in this group demonstrated one of four of Ekman’s emotions (happy, sad, surprise, and fear) (figure 1). The researchers used the METT (Ekman, 2002) e.g., the micro-expression was presented between two neutral expressions and lasted 208ms at a frame rate of 24fps. The structure of the animations was: neutral face – micro- expression – neutral face.
Figure 1. Stylized character(top); realistic character(bottom) showing the four emotions (from left): fear, happiness, sadness, surprise
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Table 1: Facial AUs and head AUs for each emotion
Table available in the original PDF paper.
The second set of animations (used in study 2) contained macro-expressions (emotion1) + standard (e.g., non-exaggerated) micro-expressions (emotion2). The third set of animations (used in study 2 as well) contained macro-expressions (emotion1) + exaggerated micro- expressions (emotion2). The exaggeration of the micro-expressions was achieved by increasing the magnitude of the AU facial displacements by 30%. In study 2, the animations were 3.29s long with a frame rate of 24fps. The structure of the animations was: neutral face – macro-expression (emotion1) – normal/exaggerated micro-expression (emotion2) – neutral face. All the animations were rendered with a resolution of 1920x1080 pixels and exported as QuickTime movies. The animations did not include any sound and were framed with same camera angle and lighting scheme.
The evaluation instrument for study 1 was an online Qualtrics survey which contained the first set of eight animation clips presented in random order. Subjects were sent an email link to the survey and asked whether they wanted to participate in the study. If they agreed to be in the experiment, they were asked to provide some demographic in-formation, watch the animation clips and for each video identify the agent’s emotion conveyed by the micro expression. Subjects were asked to select the emotion from a pull-down menu which listed Ekman’s six basic emotions + neutral + unable to recognize.
The evaluation instrument for study 2 was another Qualtrics survey which included the second and third sets of animations (16 animations in total) presented in random order. Subjects were asked to watch the animations and for each clip identify the true emotion of the character (conveyed by the micro-expression) and rate the intensity and naturalness of the character emotion on a 5-point Likert scale (1=low; 5 =high). The procedure was the same as in study 1.
FINDINGS Study 1 (recognition of micro-expressions in isolation). Limbrecht et al. (2013) reported that emotion recognition rates above 55% are acceptable and can be chosen for further analysis. The baseline acceptable recognition rate was set to 60% in this study.
For the stylized character, the mean recognition rate across the 4 micro-expressions was 79.45% (happy = 84.73%; sad = 88.73%; fear = 60.73%; surprise = 83.64%). Hence, the overall mean recognition rate was > the baseline acceptable rate, and the mean recognition rate of each individual emotion was also > 60%.
For the realistic character, the mean recognition rate across the 4 micro-expressions was 79.27% (happy = 87.37%; sad = 82.94%; fear = 69.62%; surprise = 77.13%). The overall
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 mean recognition rate was > the baseline acceptable rate, and mean recognition rate of each individual emotion was also > 60%.
A paired sample t-test was used to analyze the effect of visual style on the recognition of micro-expression (1 represents “recognized”, 2 represents “not recognized”). Based on the data analysis, the effect of visual style was not significant (t(2198) = 0.000, p = 1). The stylized character (M = 1.21, SD = 0.404) had the exact same recognition rate as the realistic character (M = 1.21, SD = 0.404).
Study 2 (recognition of true emotion and ratings of naturalness and intensity). Results of pair sample t-tests showed that the effect of degree of exaggeration of micro-expressions was significant on naturalness rating (t(4398) = 2.233, p = 0.026), intensity rating (t(4398) = -3.65, p < 0.001), and true emotion recognition (t(4398) = 10.756, p < 0.001). For the variable naturalness rating, the standard (non-exaggerated) micro-expression group (M = 3.12, SD = 1.07) was rated significantly higher than the exaggerated micro-expressions group (M = 3.05, SD = 1.09). For the variable intensity rating, the standard micro-expression group (M = 3.17, SD = 1.09) was rated significantly lower than the exaggerated micro-expression group (M = 3.28, SD = 1.05). For the variable true emotion recognition, the standard micro-expression group (M = 1.79 (21%), SD = 0.41) had significant lower recognition rate than exaggerated micro-expression group (M = 1.65 (35.4%), S = 0.48). Both groups had low recognition rates, well below the acceptable baseline recognition rate of 60%.
Pair sample t-tests showed that character visual style did not have a significant effect on naturalness rating (t(4398) = -1.144, p = 0.253) and true emotion recognition (t(4398) = - 0.502, p = 0.615). However, it did have a significant effect on intensity rating (t(4398) = - 7.347, p < 0.001); the stylized character’s animations (M = 3.11, SD = 1.06) were rated significantly lower than the realistic character’s clips (M = 3.34, SD = 1.08).
DISCUSSION AND CONCLUSION The main findings from the study show that subjects can accurately recognize micro expressions in animated agents when the micro expressions are presented in isolation, and recognition is not affected significantly by the character’s visual style. Results also show that recognition of micro-expressions (which represent the character’s true emotions) decreases significantly when the animated character is conveying both macro and micro-expressions. Exaggerating the micro-expressions improves recognition rate and perceived emotion intensity but decreases the perceived naturalness of the agents’ emotions. Further, realistic characters may be perceived as having significantly more intense emotions than the stylized characters.
The findings have important implications for research and practice. First, they suggest that it is possible to create affective animated agents that convey recognizable micro- expressions. Second, they suggest that exaggeration, one of the 12 principles of animation, can be used to make the agents’ true emotions clearer and more intense. However, attention
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 must be paid so that the naturalness of the animation is not compromised by the level of exaggeration. Third, when selecting an agent visual style, designers should consider that realistic characters may convey micro-expressions that are perceived as more intense, possibly because realistic characters have a higher level of facial detail than stylized characters.
The work reported in the paper had two main limitations that could be addressed in future research. The first limitation is that the characters did not speak and the animations were not embedded in story scenarios which could have helped the viewers better under- stand the characters’ situations. The lack of a story context made it more difficult for the participants to recognize the true characters’ emotions (conveyed by the micro-expressions) when the characters were expressing both macro- and micro-expressions. Our recommendation is that future studies should use animations that are situated within clear story contexts and possibly with talking characters.
The second limitation is that the study included only two characters with drastically different visual styles. Some of the results (e.g., higher intensity rating for the realistic character) may be due to the intrinsic design characteristics of the characters. Future studies should consider a larger number of agents with a variety of visual styles.
ACKNOWLEDGMENTS
This work is supported in part by NSF-IIS award # 1821894 - Collaborative Research: Multimodal Affective Pedagogical Agents for Different Types of Learners.
REFERENCES
Adamo N., Benes, B, Mayer, R., Lei, X., Meyer, Z., Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration. IHSI 2021. Advances in Intelligent Systems and Computing, vol 1322. Springer, Cham. Ekman, P. (1992). Facial expressions of emotion: an old controversy and new findings.
Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 335(1273), 63-69. Ekman, P. (2002). Micro-expression training tool (mett). San Francisco: University of California. Ekman, P. (2009). Lie catching and micro-expressions. In Clancy Martin (ed.), The
Philosophy of deception. Oxford University Press, 118-133. Haggard, E. A., Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of research in psychotherapy, 154-165, Springer.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Limbrecht, K., Rukavina, S., Scheck, A., Walter, S., Hoffmann, H., Traue, H. C. (2012).
The in-fluence of naturalness, attractiveness and intensity on facial emotion recognition. Psychology Research, 2(3), 166-176. Queiroz, R. B., Musse, S. R., Badler, N. I. (2014). Investigating Macro-expressions and Micro-expressions in Computer Graphics Animated Faces. Presence: Teleoperators and Virtual Environments, 23(2), 191–208. Russell, J. A., Chu, E., Phillips, M. L. (2006). A pilot study to investigate the effectiveness
of emotion recognition remediation in schizophrenia using the micro-expression training tool. British Journal of Clinical Psychology, 45(4), 579–583. Shen, X. B., Wu, Q., & Fu, X. L. (2012). Effects of the duration of expressions on the recognition of micro-expressions. Journal of Zhejiang University Science B, 13(3), 221-230. Stewart, P. A., Waller, B. M., Schubert, J. N. (2009). Presidential speechmaking style:
Emotional response to micro-expressions of facial affect. Motivation and Emotion, 33(2), 125–135. Yan, W. J., Wu, Q., Liu, Y. J., Wang, S. J., & Fu, X. (2013). CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces.10th IEEE inter- national conference and workshops on automatic face and gesture recognition (FG), 1- 7. Zielke, M. A., Dufour, F., Hardee, G. (2011). Creating micro-expressions and nuanced
nonverbal communication in synthetic cultural characters and environments. The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC).
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional