Sida Song (2026). EmoMasque: A Virtual Jewelry Emotional Regulation System Based on Facial Emotion Recognition and Multisensory Feedback. In: Affective and Pleasurable Design. AHFE Open Access, vol. 204, pp. 255–265. https://doi.org/10.54941/ahfe1007351
In the current era of human-centered design, emotional design has become a crucial paradigm for enhancing user experience. A common issue in emotional design today is the singularity of emotional regulation methods, as users are unable to directly and accurately perceive their emotions or actively participate in the process of transforming their emotional states. This paper proposes EmoMasque, an interactive virtual jewelry system designed to address this gap. The system transforms abstract emotions into perceivable and adjustable visual forms, guiding users in improving their emotions autonomously. EmoMasque integrates three core modules: a real-time facial emotion recognition module based on Plutchik’s Wheel of Emotions, which captures eight basic emotions and maps them to a dynamically color-changing virtual mask; a multisensory interaction module, allowing users to select specific facial areas such as the forehead, eyes, nose, mouth, and ears to generate corresponding visual stimuli for targeted emotional regulation; and an AI-driven dialogue agent that offers personalized healing suggestions based on sensory therapy theories. the system visualizes emotional data, assisting users in actively participating in emotional health management. The system also incorporates a long-term user follow-up mechanism and an AI iterative optimization mechanism to enhance the accuracy and adaptability of the suggestions. Preliminary user tests have shown that this system significantly improves users' ability to perceive emotions and their positive emotional experiences. This research provides a practical and evidence-based framework for the intersection of emotional design and pleasurable design.
In today’s fast-paced society, emotional distress has become a common mental health issue. According to the Report on the Development of Chinese National Mental Health (2023–2024), young white-collar workers experience different levels of anxiety, depression, or emotional exhaustion, and this trend is growing. The public still lacks sufficient knowledge about mental health, especially in areas such as parenting, emotion regulation, and the identification of mental disorders, where improvement is urgently needed.
256 Song Although professional psychological counselling and drug interventions are very important for patients with severe conditions, access to professional help remains limited for people with mild to moderate emotional distress due to factors such as cost, time, and social reasons (Kazdin and Blase, 2011). In this context, digital emotion regulation tools (e.g., meditation apps, mood tracking applications) have emerged and become common methods for the public to manage their own emotions.
However, existing digital emotion regulation tools share a common problem: emotions are internal mental activities that are “invisible” and “non- quantifiable” (Meng, 2005). Most users can only vaguely feel “happiness”, “irritation”, or “lowness”, but they find it hard to clearly define the intensity, level, or mixed nature of their emotions. Studies show that understanding emotions is the basis for improving emotion regulation, and rumination or simply expressing emotions cannot effectively regulate them (Pennebaker and Chung, 2011). Current tools often use passive methods such as verbal guidance, text recording, and abstract charts. As a result, users lack active participation and sensory immersion, which prevents them from exploring their own emotions. This leads to limited effects on emotion regulation and low user retention (Lyubomirsky and Layouts, 2013).
To address the above problems, this study proposes the following core research question: how can we use interactive visual design to turn abstract emotions into a tangible, acceptable, and actively adjustable embodied experience, so as to effectively improve users’ emotional awareness and emotion regulation ability?
The innovative contributions of this paper are mainly reflected in three aspects: Mindfulness intervention mechanism: Based on facial emotion recognition and AR virtual mask visualisation technology, this study combines emotion detection with the psychological concept of mindfulness. This helps users move from vague emotion perception to clear self-awareness, filling the gap in existing tools regarding the acceptance of emotions.
Fun interaction and autonomy design: Unlike traditional apps that rely on passive report reception, this study designs a virtual jewellery interaction system based on five facial areas. Users can freely choose the forehead/cheeks, eyes, nose, mouth, or ears to experience changes in jewellery elements that match their current emotions. By using autonomy and playfulness, this design reduces users’ psychological resistance to emotion regulation interventions and naturally connects with sensory therapy suggestions.
AI personalised guidance: The AI dialogue agent is based on psychologically recognised theories of sensory emotion regulation and pre- trained questionnaire data (including users’ emotional response tendencies to various sensory stimuli). It provides targeted sensory therapy suggestions according to the facial area selected by the user. At the same time, the system includes a long-term user follow-up and feedback mechanism, allowing the AI to continuously improve its response strategies and achieve dynamic adaptation of personalised emotion regulation.
THEORETICAL BASIS AND RELATED WORK Emotional Design and Pleasurable Design
Donald Norman, in his book Emotional Design, proposed three levels of emotional processing: visceral, behavioural, and reflective (Norman, 2004).
EmoMasque: A Virtual Jewelry Emotional Regulation System Based on Facial Emotion 257 The visceral level focuses on the immediate emotional responses caused by sensory stimuli. The behavioural level concerns the sense of fluency and achievement during use. The reflective level involves users’ self-identity, meaning-making, and cultural resonance. In this study, the AR emotion visualisation mask based on facial recognition works at the visceral level – real-time colour and shape changes arouse users’ curiosity and attention toward their own emotions. The interaction based on five facial areas works at the behavioural level – free choice and instant feedback enhance users’ sense of participation and agency. The reflective dialogue guided by AI works at the reflective level – helping users understand the sources of their emotions and build self-regulation strategies.
Patrick W. Jordan further proposed four dimensions of pleasurable design: physio-pleasure, socio-pleasure, psycho-pleasure, and ideo-pleasure (Jordan, 2000). Among these, physio-pleasure (e.g., colour) and psycho-pleasure (e.g., control, sense of achievement) are the main dimensions that this system focuses on. By actively selecting facial areas and receiving visual jewellery feedback, users experience the cognitive pleasure of being able to understand their own emotions.
Facial Emotion Recognition Facial expressions, as external signals of emotions, have been shown by a large number of studies to be consistent across cultures. Ekman identified six basic emotions: happiness, sadness, anger, fear, surprise, and disgust (Ekman and Friesen, 1971). Plutchik’s wheel of emotions model further organises emotions into a three-dimensional structure – intensity, similarity, and polarity – providing a strong theoretical framework for emotion classification and visualisation (Plutchik, 1980). This study adopts the eight core emotions from Plutchik’s wheel (joy, trust, fear, surprise, sadness, disgust, anger, anticipation) and their corresponding colours as the basis for AR mask colour mapping.
At the technical level, facial landmark detection algorithms such as CLMtrackr (Constrained Local Model Tracker) have been able to track facial key points in real time and recognise basic emotions (Baltrušaitis et al., 2018). This system uses Google MediaPipe technology and builds a working prototype through manually defined rules.
The Five Senses and Emotion Regulation A large amount of psychological research has confirmed that sensory stimuli play a significant role in regulating emotions. In terms of vision, colour psychology suggests that warm colours (red, orange, yellow) are often associated with energy and excitement, while cool colours (blue, green, purple) are linked to calmness and relaxation (Valdez and Mehrabian, 1994). In terms of hearing, natural sounds (flowing water, bird songs) and soothing music can lower cortisol levels and relieve anxiety (Thoma et al., 2013). In terms of smell, scents such as lavender, citrus, and mint produce calming or refreshing effects by directly acting on the limbic system (Herz, 2009). In terms of touch, soft textures, warm temperatures, and moderate pressure can promote the release of oxytocin, creating a sense of safety (Morrison et
258 Song al., 2010). In terms of taste, sweet and umami flavours activate the brain’s reward pathways, generating feelings of pleasure (Kringelbach et al., 2012).
However, existing emotion regulation tools rarely match these sensory stimuli dynamically with the user’s current emotional state. This study uses an AI dialogue agent to generate personalised sensory therapy suggestions based on the user’s detected emotion and self-selected facial area, achieving a three-part connection among emotional state – sensory preference – therapy plan.
Emotional Competence Emotional awareness refers to an individual’s ability to recognise, understand, and accept their own emotions. It is an important prerequisite for emotion regulation. Research shows that people with low levels of emotional awareness are more likely to experience emotional distress and find it harder to recover from negative emotions (Lane and Schwartz, 1987). One of the key innovations of this study is the use of AR mask visualisation to help users face their own emotions. Users can not only see the colour and shape of their emotions but also receive an interesting report under the system’s guidance, accepting that this is my current emotional state, thereby reducing resistance to or denial of negative emotions.
Self-efficacy refers to an individual’s belief in their ability to successfully complete a certain goal. In emotion regulation, an increase in self-efficacy is positively related to positive emotional experiences (Bandura, 1997). This system enhances users’ sense of control over the emotion regulation process by giving them the freedom to choose facial areas and decide on sensory intervention methods, thus improving their self-efficacy in emotion regulation.
Limitations of Related Systems Some existing emotion detection apps, such as MoodyWatch and AI Emotion Mirror (Ryan and Deci, 2000), mainly focus on emotion recognition and feedback but lack intervention features. Commercial meditation apps (e.g., Headspace, Calm) provide relaxing audio and guidance, but users are in a passive receiving state, and there is little personalised adaptation. This system fills this gap by guiding users to discover their emotions and providing personalised suggestions to improve their emotional state.
SYSTEM DESIGN Design Concept and Interaction Loop
The core design concept of this system is that emotions need to be seen, accepted, and talked about. Based on this, we built a six-step interaction loop of detection – mindfulness – interaction – guidance – healing – feedback (as shown in Figure 1): Detection: The user enters the website and starts the camera for active emotion detection.
Mindfulness: The system generates an AR virtual mask, visualising the current dominant emotion with colours and patterns, helping the user face their own emotional state.
EmoMasque: A Virtual Jewelry Emotional Regulation System Based on Facial Emotion 259 Active interaction: The user can freely choose five facial areas (forehead, eyes, nose, mouth, ears). Clicking on a area changes the virtual jewellery elements at that location, providing fun feedback.
Guidance: Based on the user’s selected facial area, the system starts an AI dialogue agent and informs the user of the sensory channel corresponding to that area (e.g., forehead corresponds to vision, ears correspond to hearing, etc.).
Healing: The AI provides sensory healing suggestions based on psychological evidence (e.g., “smell some citrus essential oil”, “listen to the sound of flowing water”, “eat a small piece of dark chocolate”, etc.).
Feedback: The user can rate the effectiveness of the suggestions. The data is stored in the background for AI iteration. A long-term follow-up mechanism will collect the user’s continued experience to optimise personalised recommendations.
Figure 1: System interaction loop architecture.
AR Mask Based on Facial Emotion Recognition Technical implementation: The system uses a web-based MediaPipe framework for real-time facial landmark tracking (468 3D key points). It extracts key AU intensity values related to emotional expression (e.g., AU4, AU5, AU6, AU9, AU10, AU12, AU15, AU20, AU24, AU26). Based on the theoretical framework of Plutchik’s wheel of emotions, the system establishes mapping rules from AU combinations to the eight core emotions (e.g., joy corresponds to AU6+AU12, anger corresponds to AU4+AU5+AU24, etc.). The system calculates the above AU intensities frame by frame, outputs the probability distribution of the eight emotions through rule matching, and takes the emotion with the highest probability as the current dominant emotion.
Table 1: Mapping rules of Plutchik’s wheel of emotions.
Subtle tension in the face, e.g., lips slightly parted.
AU05 + AU26 + AU12 + AU15 + AU01 + AU02 Neutral – All AU intensities near 0 Face relaxed.
Visualisation mapping: According to Plutchik’s wheel of emotions, each emotion corresponds to a specific colour: anger (#E34B4C), anticipation (#FCA349), joy (#FEF56C), trust (#5CDE53), fear (#52B49B), surprise (#4A52E0), sadness (#B14AEE), disgust (#EE68E4). The background colour of the mask changes in real time based on the dominant emotion. In addition, the saturation and brightness of the mask also change according to the intensity of the detected emotion. The overall shape of the mask is a combination of abstract patterns representing the eight emotions, giving it a modern look and aesthetic appeal.
Figure 2: Eight emotion colour cards and visual graphics.Figure 3: Mask design process and rendering of effects.
EmoMasque: A Virtual Jewelry Emotional Regulation System Based on Facial Emotion 261 Mindfulness psychological mechanism: After the user completes the detection, the interface will display a visual report of their emotions. Through this non-judgemental visual guidance, users can better accept their current emotions and reduce resistance or feelings of shame.
Fun Visual Interaction Correspondence between facial areas and sensory channels: Based on human anatomy and sensory physiological mechanisms, this study establishes a functional correspondence between five facial anatomical areas and the five sensory channels. The forehead and cheek area corresponds to touch, as the skin in this area is rich in mechanoreceptors and is sensitive to temperature, pressure, and texture changes. The eye area corresponds to vision, serving as the core organ of visual perception, responsible for receiving light and colour information. The nose area corresponds to smell, sensing odour molecules through the olfactory epithelium. The mouth area corresponds to taste, relying on taste buds on the tongue to identify basic tastes such as sweet, sour, salty, bitter, and umami. The ear area corresponds to hearing, collecting sound waves through the outer ear and transmitting them to the inner ear for sound signal processing. This mapping provides an anatomical and physiological basis for subsequent sensory healing interventions based on the user’s self-selected area.
Virtual jewellery interaction: Each facial area has pre-designed virtual jewellery elements. When the user clicks on a area or button, the corresponding jewellery element appears with animation effects (rotation, scaling, glowing) and is accompanied by soft sound effects, providing immediate fun feedback. This design introduces gamification elements, which reduces users’ psychological resistance to emotion regulation interventions and increases their motivation for self-exploration. Through this interaction, users can experience a sense of autonomy. When individuals feel that their behaviour is chosen voluntarily rather than forced, they have stronger intrinsic motivation and more lasting behavioural change (Watson et al., 1988). This study gives users control over the intervention path by allowing them to freely choose facial areas, thereby increasing their willingness to participate and their acceptance of subsequent suggestions.
Figure 3: Schematic diagram of mask changes corresponding to facial sensory areas.
The AI training data consists of two main parts: (1) a knowledge base from psychological literature on the effects of sensory stimuli on emotions (e.g., colour psychology, music therapy, aromatherapy, taste-based emotion regulation, etc.); and (2) a self-designed Questionnaire on the Effects of the Five Senses on Emotions (see appendix), which collects the actual influence of different sensory stimuli (e.g., soft textures, floral scents, sweet tastes, natural sounds, etc.) on users’ emotional changes. The questionnaire uses a 5-point scale and covers both positive and negative emotional changes.
When a user selects a facial area, the AI agent first identifies the currently detected emotion (e.g., “sadness”). Then, based on the sensory channel corresponding to that area, it searches the knowledge base for types of sensory stimuli that have been shown to regulate that emotion. It also refers to the questionnaire data on the user’s (or similar users’) preference for such stimuli, and generates a personalised suggestion text. For example: Sadness + mouth (taste):
Sweet tastes activate the brain’s reward pathway and relieve sadness. Would you like a small piece of dark chocolate or a warm cup of honey water? Let’s taste it mindfully together.
Anxiety + ear (hearing): Soothing natural sounds can lower cortisol levels when you feel anxious. Here is a recording of a flowing stream. Close your eyes and listen for one minute, then tell me how you feel.
The AI uses a first-person, gentle, and non-judgemental tone. After giving a suggestion, it actively asks the user for feedback on their experience (e.g., “How do you feel?”). The user’s feedback is recorded for future iterations.
USER EVALUATION Evaluation Method
We conducted a preliminary user study and recruited 30 young users aged 18-35 (15 males and 15 females). All of them were healthy volunteers with no experience of professional psychological treatment. The study used quantitative questionnaires and qualitative interviews. Each participant completed a system experience session of about 20 minutes, which included: System operation: Users freely explored emotion detection, mask visualisation, five-sense interaction, and AI dialogue.
Post-experience measurement: They completed a self-designed emotional awareness questionnaire (including items such as “I can clearly see my own emotional state”, “I can accept my current emotions”, “I am willing to try AI suggestions”, etc., rated on a 5-point scale).
Semi-structured interviews: Each participant took part in a semi-structured interview lasting about 5-10 minutes. The interview guide focused on the following core questions: How was your experience with the AR mask for emotion visualisation?
EmoMasque: A Virtual Jewelry Emotional Regulation System Based on Facial Emotion 263 Which interaction parts did you find interesting or useful? Did the AI’s suggestions meet your actual needs? Do you have any suggestions for improvement?
The interviews were audio-recorded and later transcribed into text. Thematic analysis was used to organise and summarise the data.
Results
The average score on the emotional awareness questionnaire was 4.2 out of 5. Among the items, “I can clearly see my own emotional state” scored 4.5, and “I can accept my current emotions” scored 4.0. In the interviews, several users said: “When I saw the mask turn blue, I suddenly realised that I was actually feeling a bit low. Before that, I only had a vague feeling of discomfort.” “When the mask added changes to the ear area, I became interested in the AI’s suggestions about hearing.” The average usefulness rating of the AI suggestions by users was 4.1 out of 5. One user commented: “The AI did not preach. It naturally suggested that I listen to the sound of water. I followed the advice, and I actually felt calmer.”
DISCUSSION
The preliminary evaluation results confirmed the effectiveness of the system design. Through AR mask visualisation of emotions, users’ emotional awareness was significantly improved. Through the fun five-sense interaction, users showed a high willingness to participate. Through personalised AI suggestions, users experienced noticeable emotional improvement. It is particularly worth noting that users gave positive feedback on the step of facing emotions. This suggests that in emotion regulation interventions, helping users see and accept their emotions may be a more critical starting point than directly providing solutions.
Limitations and Future Work This study has the following limitations:
Sample limitation: All participants were young and healthy individuals. The system’s effectiveness for people with severe emotional disorders has not been tested.
Short-term evaluation: Only the effects of a single use were measured. Longitudinal studies are needed to examine long-term emotional improvement and behavioural change.
Technical limitations: Facial emotion recognition becomes less accurate under poor lighting or when the face is turned at an angle. The emotion recognition only used simple rules and did not employ or train a more complete model or dataset. 3D rendering may cause performance issues on low-end devices.
Future work: Conduct longitudinal studies with larger samples and longer periods to evaluate the long-term impact of the system on self-efficacy in emotion regulation.
264 Song Integrate wearable devices (e.g., heart rate variability data from smartwatches) as objective emotion indicators to improve detection accuracy.
Expand to multi-user social scenarios, allowing users to anonymously share their emotional mask art and form supportive communities.
Further improve the naturalness and empathy of AI dialogue by introducing emotion perception and response models based on affective computing.
Future work could also extend the current emotion recognition from basic emotions (e.g., joy, sadness, anger) to more complex emotional states (e.g., guilt, pride, jealousy). As noted by Hassannataj Joloudari et al. (2026), complex emotion recognition remains significantly underdeveloped in affective computing, and emerging meta-learning approaches—such as few-shot learning and continual learning—offer promising pathways to overcome data scarcity and improve system adaptability across diverse users and contexts.
FURTHER CONCEPTS FOR THE AI FOLLOW-UP MECHANISM In the future, the system will implement more intelligent user follow-up. The AI can actively ask during follow-up, for example: Did you try the sweet food suggested last time? How did it work for you? Based on the user’s answer, it will adjust subsequent suggestions. At the same time, the system can build a user emotion profile, analyse emotion patterns across different seasons, times, and situations, and provide predictive and preventive suggestions. The ultimate goal is to build a personalised AI assistant that can regulate emotions.
CONCLUSION
This paper has proposed and implemented an AR virtual jewellery system based on facial emotion recognition and five-sense interaction. It aims to solve the problems of lacking self-awareness and insufficiently personalised regulation suggestions in existing digital emotion regulation tools. Through a closed-loop design of “detection – mindfulness – interaction – guidance – healing – feedback”, the system helps users visualise, accept, and talk about their abstract emotions. Preliminary user testing has verified the system’s effectiveness in improving emotional awareness, positive emotions, and user engagement. In the future, as AI technology continues to evolve and the follow-up mechanism improves, this system has the potential to become a truly personalised companion for emotional health.
ACKNOWLEDGMENT
I would like to thank every teacher who has guided me, every user who participated in the experiments, and my family for their support. The successful completion of this study would not have been possible without your help.
REFERENCES
Baltrušaitis, T., Zadeh, A., Lim, Y.C. and Morency, L.P. (2018). OpenFace 2.0: Facial behavior analysis toolkit. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi’an, China, 15–19 May 2018, pp. 59–66. Bandura, A. (1997). Self-efficacy: The Exercise of Control. New York: W.H. Freeman. Chinese Academy of Social Sciences (2021). Report on the Development of Chinese
National Mental Health (2019–2020). Beijing: Social Sciences Academic Press. (In Chinese) Ekman, P. and Friesen, W.V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), pp. 124–129. Hassannataj Joloudari J, Maftoun M, Nakisa B, Alizadehsani R, Yadollahzadeh-
Tabari M and Gaftandzhieva S (2026) Complex emotion recognition system using basic emotions via facial expression, electroencephalogram, and electrocardiogram signals: A review. Front. Psychol. 17:1682883. doi: 10.3389/ fpsyg.2026.1682883 Herz, R.S. (2009). Aromatherapy facts and fictions: A scientific analysis of olfactory effects on mood, physiology and behavior. International Journal of Neuroscience, 119(2), pp. 263–290. Jordan, P.W. (2000). Designing Pleasurable Products: An Introduction to the New
Human Factors. London: Taylor & Francis. Kazdin, A.E. and Blase, S.L. (2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science, 6(1), pp. 21–37. Kringelbach, M.L., Stein, A. and van Hartevelt, T.J. (2012). The functional human
neuroanatomy of food pleasure cycles. Physiology & Behavior, 106(3), pp. 307– 316. Lane, R.D. and Schwartz, G.E. (1987). Levels of emotional awareness: A cognitivedevelopmental theory and its application to psychopathology. American Journal of Psychiatry, 144(2), pp. 133–143. Lyubomirsky, S. and Layouts, K. (2013). How do simple positive activities increase
well-being? Current Directions in Psychological Science, 22(1), pp. 57–62. Meng, Z.L. (2005). Emotion Psychology . Beijing: Peking University Press. (In Chinese) Morrison, I., Löken, L.S. and Olausson, H. (2010). The skin as a social organ.
Experimental Brain Research, 204(3), pp. 305–314. Norman, D.A. (2004). Emotional Design: Why We Love (or Hate) Everyday Things.
New York: Basic Books. Pennebaker, J.W. and Chung, C.K. (2011). Expressive writing: Connections to physical and mental health. In Friedman, H.S. (ed.) The Oxford Handbook of Health Psychology. New York: Oxford University Press, pp. 417–437. Plutchik, R. (1980). A general psychoevolutionary theory of emotion. In Plutchik,
R. and Kellerman, H. (eds.) Theories of Emotion. New York: Academic Press, pp. 3–33. Ryan, R.M. and Deci, E.L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), pp. 68–78. Thoma, M.V., La Marca, R., Brönnimann, R. et al. (2013). The effect of music on the
human stress response. PLoS ONE, 8(8), e70156. Valdez, P. and Mehrabian, A. (1994). Effects of color on emotions. Journal of Experimental Psychology: General, 123(4), pp. 394–409. Watson, D., Clark, L.A. and Tellegen, A. (1988). Development and validation of brief
measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), pp. 1063–1070.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Affective and Pleasurable Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-80-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional