Yuuna Ishikami, Hisaya Tanaka (2023). Evaluation and Validation of Emotional Expression Mimicry Tasks for Highly Sensitive Person Assessment. In: Emerging Technologies in Healthcare and Medicine. AHFE Open Access, vol. 116, pp. 334–343. https://doi.org/10.54941/ahfe1004387
In recent years, Highly Sensitive Persons (HSP) have gained increasing attention. HSP refers to individuals with heightened sensory sensitivity, making them more sensitive to stimuli and frequently more empathetic. In this study, we focused on HSPs’ high empathy and their ability to detect subtle cues from facial expressions. We hypothesized that individuals with HSP tendencies are more likely to perceive and express minor changes in facial expressions. To test this hypothesis, we created deliberate facial expressions representing nine emotional states, i.e., happiness (four levels), neutrality (one level), and sadness (four levels). We measured mouth corner movements using the MediaPipe system, which is a webcam-based motion capture system. The subjects imitated 10 facial expressions, ranging from neutral to happy and sad, each with five levels of intensity. We then examined the correlations between the subjects’ facial expressions and psychological measures, including the Highly Sensitive Person Scale (HSPS) and the Japanese version of the Interpersonal Reactivity Index. The result exhibited correlations in specific intervals. First, there was a strong correlation in the five-level range from neutral to happy (r = 0.67). Second, there was a correlation in the interval from minor expression change from neutral to the second level of happiness (r = 0.50). Third, a correlation was observed in the interval from the second to the fourth level of happiness (r = 0.61). These results suggest that individuals with higher HSPS scores (indicating HSP tendencies) exhibit greater changes in facial expressions when experiencing happiness, which suggests that those with HSP tendencies are more receptive to subtle changes in intentional facial expression mimicking stimuli, particularly happiness.
The Highly Sensitive Person (HSP) concept has gained attention since approx- imately 2020, primarily through social network systems and public dis- closures by celebrities. HSPs are individual with a genetically - based trait characterized by heightened sensitivity to various stimuli, including a lower sensory threshold and stronger emotional reactions, in addition to heightened aesthetic sensitivity (Aron et al., 1997). However, HSPs are not designated in the Diagnostic and Statistical Manual of Mental Disorders, unlike various
Evaluation and Validation of Emotional Expression Mimicry Tasks 335
other conditions, e.g., depression or Attention Deficit/Hyperactivity Disorder (ADHD). Instead, HSP is considered an inborn personality trait.
Subjects with HSP exhibit several specific traits, e.g., deep thinking and processing, heightened sensitivity to excessive stimuli, overall strong emo- tional reactions (especially high empathy), and the ability to perceive subtle cues (Yano et al., 2018). Due to these characteristics, it is believed that indi- viduals with HSP tendencies are more prone to experiencing depression. The growing awareness of HSP has led to an increase in clinics where individuals can seek assessment regarding their HSP tendencies. Typically, these clinics utilize psychological questionnaires for evaluation; however, such question- naires can require the subject to have a certain level of language proficiency and may be susceptible to intentional bias or self-deception (Psycho Psycho., 2022). To the best of our knowledge, no previous study has attempted to eval- uate HSP using physiological method; however, there have been studies on emotional contagion, defined by Hatfield et al. as the phenomenon where emotions arise through the automatic imitation or synchronization of oth- ers’ facial expressions, speech, posture, and actions (Hatfield et al., 1992, 1994). In addition, there have been studies on mirror system activity, which involves the activation of motor-related brain regions when observing the same actions in others or during one’s own action execution. These studies could provide clues related to HSP. Notably, previous research has suggested that subjects with HSP, known for their high empathy, are more suscep- tible to emotional contagion (Kushizaki, M., 2020). Additionally, studies examining the relationship between emotional contagion and mirror system activity have demonstrated that variations in µ-waves (in the α-wave fre- quency range) recorded in specific brain regions, including the left central, central, and right central regions, reflect mirror system activity. These stud- ies have indicated that subjects that are more prone to emotional contagion tend to exhibit higher mirror system activity (Ikeda et al., 2016).
In addition, previous research conducted by our team has demonstrated that measuring emotional contagion and mirror system activity can help iden- tify subjects with a propensity for HSP (Figures 1 and 2) (Ishikami et al., 2022). Based on these findings, we hypothesized a correlation between Event- Related Desynchronization (ERD), α-wave power values, and HSP. Further, by considering HSP’s traits, e.g., sensitivity to subtle cues and strong over- all emotional reactions, we referred to the action units from Ekman’s Facial Action Coding System (FACS), specifically “AU12: Raising Both Lip Cor- ners” and “AU15: Lowering Both Lip Corners.” We theorized that subjects with HSP tendencies would exhibit an upward movement of the corners of the lips when mimicking expressions of happiness and a downward movement when mimicking expressions of sadness.
Thus, in the current study, we attempted to elucidate the characteristics of subjects with HSP by having subjects mimic expressions of happiness, neu- trality, and sadness from nine levels of facial expressions. We measured these changes using MediaPipe, which is a tool that enables facial motion capture using a webcam. We categorized the acquired data into five levels for tran- sitioning from neutrality to happiness and five levels for transitioning from
336 Ishikami and Tanaka
neutrality to sadness. In addition, we included two intermediate levels for transitions from happiness and sadness, respectively, to further evaluate min- imal changes. We compared the results of these expression changes to the scores from the Highly Sensitive Person Scale (HSPS) and the Japanese ver- sion of the Interpersonal Reactivity Index (IRI-J) to gain insights into HSP traits.
Figure 1: HSPS cutoff value of 100 and ERD change rate (Ishikami et al., 2022).Figure 2: HSPS cutoff value of 100 and α-wave band power values for three emotions (Ishikami et al., 2022).
METHOD
Highly Sensitive Person Scale Aron et al. conducted a series of investigations to validate the concept of Sensory Processing Sensitivity (SPS) as distinct from personality traits, e.g., introversion and neuroticism. They conducted qualitative face-to-face inter- views (n = 39), random telephone surveys (n = 299), and five quantitative questionnaire surveys targeting university students from various regions in the United States. The scale for measuring sensory processing sensitivity developed as a result of these studies is known as the Highly Sensitive Person Scale (HSPS). The HSPS comprises three factors, i.e., ease of excitation, low sensory threshold, and aesthetic sensitivity, totaling 27 items. In this study, the subjects were asked to respond to each item using a seven-point Likert scale.
Interpersonal Reactivity Index The IRI scale was designed to measure the trait of empathy comprehen- sively (Himichi et al., 2017). It comprises 28 items and assesses empathy
Evaluation and Validation of Emotional Expression Mimicry Tasks 337
through four dimensions, i.e., empathic concern (EC), Perspective Taking (PT), Personal Distress (PD), and ‘Fantasy Scale (FS). Here, EC measures the propensity to be emotionally moved by others’ emotional experiences, PT gauges the degree to which an individual consider others’ feelings from their perspective, PD evaluates the extent to which an individual becomes anxious or fearful due to observing others’ suffering, and FS examines an individual’s inclination to imagine being in the place of fictional characters in narratives.
EC and PD are considered to reflect the emotional aspects, and PT and FS represent the cognitive aspects of empathy. Among these dimensions, EC and PT are considered central concepts within their respective domains. In this study, the subjects were asked to respond to the IRI-J questionnaire using a five-point Likert scale.
Measuring Mouth Corner Expressions Using MediaPipe In this study, MediaPipe’s Iris Tracking was employed to measure facial changes, specifically capturing changes when intentionally mimicking expres- sions. MediaPipe is a cross-platform and customizable machine learning tool provided by Google that is designed for live and streaming media applica- tions (MediaPipe, 2023). MediaPipe offers various functionalities, including Face Mesh, Iris Tracking, Pose Tracking, and Holistic.
Within MediaPipe’s Iris Tracking, facial coordinates are tracked relative to 478 points. From these, the angles between the upper lip and the corners of the mouth (corresponding to points 164, 78, and 308) were calculated using Equation (1) (Figure 3).
2 + mouthleft 2 −mouthcenter 2
Equation reproduced from the original PDF.
2 × mouthright × mouthleft (1)
Figure 3: Diagram to calculate the angle between the nose and both corners of the mouth.
EXPERIMENT Experimental Content
In this experiment, the subjects were tasked with imitating the expressions of happiness, neutrality, and sadness from images with varying intensities. This allowed us to investigate whether individuals with higher HSPS scores are
338 Ishikami and Tanaka
more capable of detecting subtle changes in stimulus images and recognizing the emotions conveyed by those images. As a result, they could raise or lower the corners of their mouths, potentially experiencing emotional contagion. In addition, to assess whether individuals exhibited HSP tendencies, we also used the IRI-J as a measure to explore the subjects’ level of empathy, which is acharacteristic associated with HSP.
Environment and Subjects The subjects were seated in a chair, and a camera (iPhone 13 Pro) was posi- tioned at a sufficient distance to clearly capture the subjects’ faces. To ensure that there were no disruptions during the MediaPipe measurements, care was taken to avoid any unwanted background interference behind the subjects. Here the study included a total of seven subjects (six males and one female, all in their twenties).
Experimental Apparatus and Procedure We ensured that nothing was visible behind the subjects, and we captured their faces at a distance that clearly displayed their facial expressions. In this experiment, the subjects were shown prerecorded facial photographs representing four levels of happiness, neutrality, and four levels of sadness. The subjects were asked to mimic these expressions, and we captured their mimicked expressions.
We measured the facial coordinates using the MediaPipe software and calculated the changes in facial expressions by determining the angles of both corners of the mouth relative to the nose base (MediaPipe Landmark: 78th, 164th, and 308th points), which corresponds to “AU12: Raising both lip corners” and “AU15: Lowering both lip corners” as defined in Ekman’s Facial Action Coding System (FACS) (Ekman et al., 2002). We compared the changes in mouth corner angles at each level with the results of the previously administered HSPS and the IRI-J.
Note that this experiment was conducted according to the guidelines out- lined in “Psychophysiological Measurement for New Interface Development 2021-A-29”.
RESULTS AND DISCUSSION
Facial Expression Mimicry When the subjects were asked to mimic the expressions of happiness, we observed that the angle formed between the upper lip and both corners of the mouth increased significantly for all subjects as the intensity of hap- piness increased (Figure 4). In contrast, when the subjects were asked to mimic the expressions of sadness, there was variation among the subjects. For example, some subjects showed an increase in the angle as the intensity of sadness increased, others exhibited a reduction in the angle, and some subjects exhibited little change (Figure 5).
In addition, as shown in Figures 4 and 5, it was apparent that, for expres- sions of happiness, subjects with higher scores on psychological indicators
Evaluation and Validation of Emotional Expression Mimicry Tasks 339
Figure 4: Angle changes based on intensity when mimicking happy expression.Figure 5: Angle changes based on intensity when mimicking sad expression.
followed a logarithmic curve y = a log x + b, and those with lower scores followed an exponential curve y = eax. In contrast, for the expressions of sadness, no clear trend was observed between the scores on the psychologi- cal indicators and the angle changes. This may be attributed to the inherent difficulty in expressing sadness compared to other emotions (Kagamihara, 2017).
340 Ishikami and Tanaka
Relationship Between Facial Expression Mimicry and Psychological Indicators We compared the changes in facial expression with the HSPS and IRI-J scores. Here, we calculated the coefficient a (where a > 0) for the approximation curves, which was then used to represent the change in mouth corner angle (hereinafter referred to as ‘Mouth Changes’). We analyzed the changes in mouth corner angles for five levels of happiness intensity beginning from neutrality, five levels of sadness intensity beginning from neutrality, and more specific intervals, including from neutrality to the second level of happiness, neutrality to the second level of sadness, from the second level of happiness to the fourth level of happiness, and from the second level of sadness to the fourth level of sadness. We then compared these with the HSPS scores.
First, we compared the mouth changes for the five levels of happiness inten- sity and five levels of sadness intensity with the HSPS and IRI-J scores. The results demonstrated that, for the mouth changes, individuals with higher HSPS scores exhibited larger changes when imitating the expressions of hap- piness (Figure 7) (r = 0.67). However, no significant difference was observed when imitating the expressions of sadness. In addition, when we exam- ined specific intervals, e.g., neutrality to the second level of happiness and from the second level of happiness to the fourth level of happiness, indi- viduals with higher HSPS scores exhibited greater changes in the mouth corner angle (Figure 8 and Figure 9) (r = 0.50 (Neutral – Happy2), r = 0.61 (Happy2 – Happy4)).
We also compared the mouth changes with the IRI-J scores; however, no significant differences were observed for the expressions of both happiness and sadness.
From these findings, it was evident that individuals with higher HSPS scores were better at mimicking subtle changes in facial expression. The lack of significant differences in the mouth changes when mimicking the expres- sions of sad may be attributed to the inherent difficulty in expressing sadness compared to other emotions.
Figure 6: Correlation between HSPS scores and mouth changes (Neutral - Happy).
Evaluation and Validation of Emotional Expression Mimicry Tasks 341
Figure 7: Correlation between HSPS scores and mouth changes (Neutral - Happy2).Figure 8: Correlation between HSPS scores and mouth changes (Happy2 - Happy4).
CONCLUSION
In this study, we focused on HSP characteristics, particularly their heightened empathy and ability to detect subtle cues. We hypothesized that individu- als with HSP tendencies would be more receptive to subtle changes in facial expressions when intentionally mimicking those expressions, which could be expressed as their own facial expressions. To evaluate HSP objectively, we compared the changes in facial expressions when subjects intentionally mim- icked the expressions of happiness (four levels), neutrality, and sadness (four levels) with their HSPS and IRI-J scores.
An experiment was conducted, where the subjects were tasked with mim- icking the expressions of happiness, neutrality, and sadness. Then, we categorized the changes in expressions into five levels of intensity for both happiness and sadness beginning from neutrality. We compared these results with the subjects’ HSPS and IRI-J scores. The findings demonstrated that individuals with higher HSPS scores exhibited larger changes in facial expres- sion intensity when mimicking the expressions of happiness; however, no significant difference was observed for the expressions of sadness.
342 Ishikami and Tanaka
In addition, we evaluated the changes in facial expressions when the sub- jects mimicked subtle facial expressions by narrowing down the intervals of the expression intensity. Here, we found that individuals with higher HSPS scores exhibited larger changes in intensity in the intervals from neutrality to the second level of happiness and from the second level of happiness to the fourth level of happiness. However, for the expressions of sadness, no significant differences were observed in the mouth changes concerning the psychological indicators.
From these findings, we infer that individuals with higher HSPS and IRI-J scores, thereby indicating HSP tendencies, are more capable of perceiving and expressing subtle changes in facial expressions when intentionally mimicking such expressions.
In future research, we plan to revisit the calculation method used to mea- sure changes in facial expressions, especially for the challenging area of sadness expressions, and investigate their correlation with the HSPS results. In addition, by increasing the number of subjects, we plan to conduct a more detailed exploration of the relationship between HSP and the changes in facial expressions during intentional mimicry to further assess HSP through intentional expression mimicry.
REFERENCES
Aron, E. N., & Aron, A. (1997). Sensory-processing sensitivity and its relation to introversion and emotionality. Journal of Personality and Social Psychology, 73, 345–368. Ekman, P., Friesen, W. V., & Hager, J. V. (2002). Facial action coding system (2nd ed.).
Salt Lake City, UT: Research Nexus eBook. Ergotron. (2021). Ergonomic Equation Based on Ergonomics. https://www.ergotron .com/ja-jp (Accessed: August 10, 2021) Fujimura, T., & Umemura, H. (2018). Development and validation of a facial
expression database based on the dimensional and categorical model of emotions. Cognition and Emotion, 32, 1663–1670. Hatfield, E., Cacioppo, J., & Rapson, R. (1992). Primitive emotional contagion. In M. S. Clark (Ed.), Review of Personality and Social Psychology. Newbury Park, CA: SAGE. Hatfield, E., Cacioppo, J., & Rapson, R. (1994). Emotional Contagion. New York:
Cambridge University Press. Himichi, T., Oyamauchi, H., Goto, T., Fujita, Y., Kawamura, Y., Davis, M. H., & Nomura, R. (2017). Development of the Japanese version of the interpersonal reactivity index. Psychological Research, 88, 61–71. Ikeda, Y., Nishimura, Y., Kim, Y. K., & Higuchi, S. (2016). Relationship between
individual characteristics of emotional contagion and characteristics of brain mirror system activity. Journal of the Anthropological Society of Japan, 21(2), 69–74. Ishikami, Y., & Tanaka, H. (2022). Emotional Contagion and Mirror System Activity in the Highly Sensitive Person. 8th International Symposium on Affective Science and Engineering. Kagamihara, H. (2017). Understanding and expression of facial expressions in
infancy - Appropriateness of facial expression materials that are easy to under- stand and intentions of infants as seen by adults. Studies in Child Care, 55(2).
Evaluation and Validation of Emotional Expression Mimicry Tasks 343
Kushizaki, M. (2020). The Chemistry of a Sensitive Mind - An Introduction to HSP. MediaPipe. (2023). https://google.github.io/mediapipe/ (Accessed: February 24, 2023). Psycho Psycho. (2022). What is Psychological Testing? Retrieved from: https://psyc
ho-psycho.com/psychological-test/. Yano, K., & Oishi, K. (2018). The relationships among daily exercise, sensoryprocessing sensitivity, and depressive tendency in Japanese university students. Personality and Individual Differences, 127, 49–53.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Emerging Technologies in Healthcare and Medicine
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-92-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional