Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Text vs. Images: Understanding emotional expressions on social media during COVID-19 pandemic

Department of Art and Art History, Santa Clara University, 500 El Camino Real, Santa Clara, CA 95053, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1002031

Abstract

Due to the global spread of COVID-19, people all around the world have been forced to change the way they communicate and interact with others. Keeping social distance and wearing masks helps prevent the spread of coronavirus, and also makes online social platforms increase in demand in an unprecedented way (Flynn, 2008). Prolonged social isolation during COVID-19 is likely to have negative effects on mental health and communication on an individual. Researchers have found evidence for caused and elevated anxiety disorders such as somatization, post-traumatic stress disorder, panic disorders and depression amongst individuals during the COVID-19 pandemic (Meikle, 2016). Numerous studies have found that people only show their “good side” and positive emotions on social media. How does social media reveal our anxiety disorders during Covid? Do emotions expressed in pictures match with its text content on social media? In this research, 500 most recent selfies from individual accounts between December 1st and 10th in 2021 from age ranges 13 to 55 years old were downloaded for the study. The study used IBM Watson tone analyzer and Sky-Biometry as tools for linguistic analysis and emotion detection. In addition, the research compared imagery and text content in social media as a function of emotional expression and methods.Keywords: Emotional Expressions, Communication, Social Media, COVID-19, Photography Posts, Text, Instagram, Social Network, Attention Theory, Mental Health

INTRODUCTION

Human beings are social creatures that have the need to connect with others (Perry, 2021). Numerous social scientists also have found that our emotion, experience, and sense of self all depend on our interaction and relationships with other humans (Flynn, 2008). Face-to-face communication throughout human history has been treated as the most effective and efficient way to meet people’s need for social connectedness (Grieve, 2013). However, technology and the internet’s growth have revolutionized communication in everyday life (Dentzel, 2013). Online communication brings us together in topic-based communities that have no geographic restriction (Dentzel, Z., 2013). Large number of studies have found that social media is playing an important role in people’s social and emotional development, especially young people (O’Keefe, 2011). According to Kepios, a marketing consultancy, there are

© 2022. Published by AHFE Open Access. All rights reserved. 18

Text vs. Images: Understanding Emotional Expressions 19

approximately 4.55 billion social media users worldwide in October 2021, equating to 57.6 percent of the total global population (DataReportal, 2022). The study also showed that the daily time spent on social media by users worl- dwide increased from 90 minutes in 2012 to 145 minutes per day in 2020 (Statista, 2021).

The COVID-19 pandemic has transformed our lives, the way we commu- nicate and relationships over the past two years- which has created a “New Normal” that remains a part of our daily lives (Flynn, 2008). Learning and working from home has become a lifestyle. Just because we’re physically iso- lating, doesn’t mean we have to be socially isolated. Social media plays a vital role for individuals and communities to stay connected in a global health crisis (Flynn, 2008). A growing percentage of people, including health care workers in the U.S., live with mental-health problems since the start of the pandemic (Czeisler, 2020). Do people express authentic emotions on social media during Covid? Do emotions expressed in pictures match with its text content on social media? The present study aims to improve our understan- ding of emotional expression and methods during COVID-19 pandemic by providing a content analysis on Instagram.

Self-Presentation Theory Sociologist Erving Goffman formulated the theory in his book The Presen- tation of Self in Everyday Life in 1956 (Goffman, 1978). Goffman believed an individual in typical social interactions can be understood as resembling performers in action on a stage. He thinks the “self” that people present in social interaction with other people is a modified and performed image that individuals attempt to present in front of others. Many scholars support the self-presentation theory in the online world- including professor of com- munication, Graham Meikle at the University of Westminster. In his book, Graham indicates that sharing online can be the performance of the virtual presentations of self, which can be distinct from his or her “real world” self. Strategies on how to present and brand yourself on social media are rising in popularity. As “impression management” presented in Goffman’s theory, personal branding is intangible: it’s not what you say it is, rather, it is what other people say it is (Neumeier, 2005). In Donna Freitas’s book, The Hap- piness Effect, he describes personal branding as “a response to the image or name of a particular company, product, or person” (Freitas, 2017).

Dynamic of COVID-19 in the United States COVID-19 is an unprecedented tragedy and has been recognized as one of the most devastating challenges of the century after the Second World War (Gautam, 2020). As of January 13, 2022, there are more than 5.5 million peo- ple who have died globally of COVID-19 since the beginning of the pandemic (WHO, 2022). With a global total of more than 312 million cases, the United States is now the country with the largest number of reported confirmed cases (WHO, 2022). According to the Centers for Disease Control and Prevention (CDC), the highly contagious delta variant led to rising numbers of cases in the U.S. and rose again by June 2021- especially in states like Arkansas,

20 Li

Figure 1: Daily trends in number of COVID-19 Cases in the United States reported to CDC.
Original PDF Figure 1:
Figure 2: Social media Users, 14-to 24-years-old, reported impact on well-being, 2017. (Source: Royal society for public health).
Original PDF Figure 2:

Nevada, Missouri, and Wyoming (Soucheray, 2021). Accumulating evide- nce suggests that COVID vaccines do help reduce virus transmissions (Juno, 2021). However, with the current Omicron variant, COVID-19 in the United States is still complex and difficult. Over 62 million cases have been reported across the United States, with 1.39 million cases in the US in a single day reported on January 13, 2022 (WHO, 2022).

Social Media and Mental Health A 2017 survey found that social media platforms have detrimental effects on users’ mental health. The research suggested that social media can have positive effects on wellbeing, such as providing users extra scope for self- expression, self-identity, and community-building. But it has also increased anxiety, depression, sleep deprivation, body image issues and the feeling of FOMO (fear of missing out). Professor Jeremy Tyler at the Center for the Tre- atment and Study of Anxiety at the Perelman School of Medicine agreed with the negative impact of social media and how it can exacerbate social anxiety, comparison, and perfectionism. He also indicates that social media can be an important crutch for those looking for connection during the pandemic (Fullerton, 2020).

Expressions of Emotions in Text and Image The seven universal emotions are anger, fear, disgust, happiness, sadness, sur- prise, and contempt. These were first described back in the late 1800’s by

Text vs. Images: Understanding Emotional Expressions 21

Figure 3: The study of happiness and sadness from the basic facial action coding system.
Original PDF Figure 3:
Figure 4: Maslow’s hierarchy of needs vs John Antonios’ social media hierarchy of needs.
Original PDF Figure 4:

Charles Darwin. Both positive and negative emotions are necessary for opti- mal brain functioning of human health (Baumgardner, 2009). According to the Oxford Handbook of Positive Psychology, positive emotions are defined as “pleasant or desirable situational responses… distinct from pleasurable sensation and undifferentiated positive affect”. Negative Affect refers to emo- tions such as anger, fear, disgust, and sadness (Cohn, 2009). As “a picture is worth a thousand words” (Advice, 1911), marketing industry influencer Krista Neher found that our brain can process images up to 60,000 times faster than words (Cohn, 2009). Emotions can be shown in an image and evoke feelings. Research conducted by Darwin discovered that facial muscle movements and consequent furrows produced on the skin worked together to show emotions. Facial Action Coding System (FACS) is a system of facial muscle movements that correspond to a displayed emotion (Darwin, 1998). Besides image, emotions can also be identified via text. As a subfield of machine learning, deep learning is used to detect and recognize types of fee- lings through the expression of texts- such as anger, disgust, fear, happiness, sadness, and surprise (Yean, 2015).

Virtual Happiness Social media is a shared industry and sharing is social (Meikle, 2016). People share things online to express their opinions, feelings, and experience. Similar to Maslow’s hierarchy of needs, social media’s hierarchy of needs according to John Antonios dictates a person’s behavior in a virtual world. Those needs are existence (presence), structure, community building, personal branding, and optimization & monetization. Social media produces a world in which the problems of real life are hidden in order to create a facade of happiness

22 Li

Figure 5: Percentage of posts showing a specific type of happiness. N = 200.
Original PDF Figure 5:

and perfection (Freitas, 2017). My previous research focused on understan- ding people’s emotional expression through social media and in particular, Instagram. The research found that physical appearance was by far the most important factor for young adults’ happiness than relationships, achievement, or travel/experience (Li, 2020).

METHOD

Procedure The objective for this study was to examine how does social media reveal our anxiety disorders during the pandemic. This study conducted a content analysis of Instagram posts tagged with hashtags #covidlife, #covidusa, and #quarantine. 500 most recent selfies from individual accounts between December 1st and 10th in 2021 from age ranges 13 to 55 years old were downloaded for the study. To compare imagery and text content in social media as a function of emotional expression and methods, the study used open-source two recognition software. IBM Watson tone analyzer uses lin- guistic analysis to detect emotional tones in written text, and SkyBiometry as emotion detection found in selfie picture. All emotions were counted and categorized into three groups: positive, negative, and pre-affective. In addi- tion, the study examined whether the emotions explicitly detected was clearly inconsistent when they were visual or textual for the same post. All variables were coded dichotomously (0 = absent, 1 = present) by one coder. For the reliability analysis, we randomly selected 10% of the material. The analy- sis generally indicated reliable measurement. Krippendorff’s α values were acceptable all (all values, α > .76).

Variables Emotions Detection. The following emotions were coded when they were detected on the selfie picture: neutral, angry, disgusted, scared, happy, sad and surprised. If more than one emotion is present, the stronger one is shown. All emotions were counted and categorized into three groups: positive, nega- tive, and pre-affective (can be positive and/or negative, depending on the goal conduciveness of the event).

Text vs. Images: Understanding Emotional Expressions 23

Linguistic Tone Analysis. The following tones were coded when they were found in the text post: joy, fear, sadness, anger, analytical, confident and tentative. If more than one tone is present, the stronger one is shown. All tones were counted and categorized into three groups: positive, negative, and neutral.

Inconsistency. The research examined whether the emotions explicitly dete- cted was clearly inconsistent when they were visual or textual for the same post. Inconsistency was coded as present (0 = absent, 1 = present). For insta- nce, when the post presented a positive emotion found in selfie picture with a contradictory (negative or neutral tone) text post.

Number of Likes. The research explored the relationships between response behavior and emotions detected from the post. The study collected and analyzed how many likes each post received.

Demographics. A total of 500 most recent selfies from individual accounts between December 1st and 10th in 2021 were downloaded from both sex groups, 250 males and 250 females. The age of the individual was coded, whether the post included young individuals, older adults, or held no age- related information.

RESULTS

The study found that positive emotions are still more prevalent than nega- tive and pre-affective emotions on Instagram, for both selfie pictures and text posts during the COVID-19 pandemic. In Figure 6, positive emotions (44%) were detected on the selfie pictures as 10% more than negative emotions and 22% more than pre-affective emotions. Regardless of biological sex, positive emotions (54%) were detected on the text posts as 46% more than nega- tive emotions and 16% more than pre-affective emotions. Compared with words, the results showed that people are more likely to reveal their negative emotions (e.g. sadness, angry) on image than text.

The research also examined gender-based differences in emotion expres- sion on both pictures and words. Among posts, the highest percentage was positive emotions from females for both visual (28%) and textual (31%) contents. Within the negative and pre-affective emotions, there were same

Figure 6: Percentage of posts showing emotion categories. N = 500.
Original PDF Figure 6:

24 Li

Figure 7: This pie chart illustrates the percentages of each emotion categories detected on the selfie pictures between males and females.
Original PDF Figure 7:
Figure 8: This pie chart illustrates the percentages of each emotion categories detected on the text posts between males and females.
Original PDF Figure 8:
Table 1. Instagram post of the total number of likes & inconsistency between picture and text.

Table available in the original PDF paper.

Biological Sex Likes Inconsistency Posts Inconsistency Rate Male 23325 135 54% Female 65515 130 52% Total 88840 265 53%

percentages for males and females found in the text posts. But in selfie pictu- res, the percentages of negative and pre-affective emotions from males were higher in comparison with females.

In terms of the relative numbers of likes received, females were received 2.8 times more likes than males. However, it is important to consider that there were variety numbers of likes for each post based on the popularity of an individual. Moreover, the study found that both male and female posted a significantly larger number of posts (53%) which visually and textual are inconsistent in emotion expression.

DISCUSSION

Communication in virtual world nowadays tends to be less honest online. Among all posts, more than a half percent of them contained inconsistencies in emotion expression between visual and text contents. Expressing a full range of emotions, whether positive or negative, is essential and necessary to

Text vs. Images: Understanding Emotional Expressions 25

our well-being. People should never feel guilty or ashamed about what they perceive to be negativity (Rodriguez, 2013). However, the research found that social media creates space with a facade of happiness and perfection for a various reason. Individuals still tend to be post more positive emotional expressions than negative and pre-affective emotions on Instagram during the COVID-19 pandemic.

Numerous studies show that females are much more likely than male to care about what others think of them (Harrison, 2013). Today, the results showed women post more positive emotional contents in both visual and textual. Compared with men, posts from women also received more liking of numbers. In other words, women have become more care self- presentation of themselves on social media, which is heavily influenced by social media’s hierarchy of needs.

LIMITATIONS This study has several limitations and must be considered when interpre- ting these findings: (1) The search strategy was limited to Instagram posts that are in English only, which may have resulted in decreasing of cross- cultural generalizability. (2) The sample of posts was small compared with the number of posts that are available on those hashtags within a few days. (3) The researcher only investigated static image posts, and did not collect video posts that are available on the same hashtags because of the difficulty in evaluating automatic emotion detection in video. However, my previous research studied the importance role of moving images to recognize one’s emotions comparison to static images in video chatting and how the sound changes whole emotional status instantly regardless of given visual contexts (Li, 2018). (4) Finally, the average overall accuracy rates provided by IBM Watson tone analyzer and SkyBiometry need to be mentioned. IBM Wat- son tone analyzer was tested against standard emotion sets like SEMEVAL and ISEAR, and was found to be around 41–68% accurate, beating out the best reported accuracy of the models which came in around 37–63% (IBM Watson). Although SkyBiometry can read facial emotion with mask on, its accuracy rates decrease with face half-coved. The accuracy for SkyBiometry is not available, but it’s been reported as one of the top 15 best face reco- gnition API in 2021 (Company’s News, 2021). Further studies are needed to continually evaluate on visual and textual post contents once the latest versions with higher recognition accuracy rate.

REFERENCES

Advice, S. G. S. (1911). Syracuse Post Standard. March, 28, 18. Baumgardner, S.R., & Crothers, M.K. (2009). Positive Psychology. Pearson Education, Inc. Czeisler, M. É., Lane, R. I., Petrosky, E., Wiley, J. F., Christensen, A., Njai, R., ...

& Rajaratnam, S. M. (2020). Mental health, substance use, and suicidal ideation during the COVID-19 pandemic—United States, June 24–30, 2020. Morbidity and Mortality Weekly Report, 69(32), 1049. Company’s News (2021). 15 Best Face Recognition APIs in 2021: RecFaces’ Overview. Retrieved from https://recfaces.com/articles/face-recognition-apis

26 Li

Cohn, M. A., & Fredrickson, B. L. (2009). Positive emotions. Oxford handbook of positive psychology, 2, 13–24. 9. https://datareportal.com/social-media-users Daily Time Spent on Social Networking by Internet Users Worldwide from 2012 to

2020 (in minutes). Retrieved from https://www.statista.com/statistics/433871/da ily-social-media-usage-worldwide/ Dentzel, Z. (2013). How the internet has changed everyday life. Ch@ nge, 19.

O’Keefe, G., Clarke-Pearson, K. (2011). Clinical report – The impact of social media on children, adolescents, and families. Pediatrics, 127, 800–804. Darwin, C., & Prodger, P. (1998). The expression of the emotions in man and animals. Oxford University Press, USA. Flynn, C. P. (2008). Social creatures: a human and animal studies reader. Lantern

Books. Freitas, D. (2017). The happiness effect: How social media is driving a generation to appear perfect at any cost. Oxford University Press. Fullerton, N. (2020). Instagram vs. Reality: The Pandemic’s Impact on Social Media

and Mental Health. Grieve, R., Indian, M., Witteveen, K., Tolan, G. A., & Marrington, J. (2013). Faceto-face or Facebook: Can social connectedness be derived online?. Computers in human behavior, 29(3), 604–609. Global Social Media Stats. Retrieved from https://datareportal.com/social-media-

users Goffman, E. (1978). The presentation of self in everyday life (Vol. 21). London: Harmondsworth. Gautam, S. (2020). The influence of COVID-19 on air quality in India: A boon or inutile. Bulletin of Environmental Contamination and Toxicology. https://doi.org/ 10.1007/s00128-020-02877-y. Harrison, Jamie (2013). Battle of the Sexes: Are Women More Self-conscious in the

Office than Men? Retrieved from https://www.blackenterprise.com/study-workpl ace-perception-office-women-care-more/ IBM Watson Tone Analyzer Review. Retrieved from https://www.slant.co/options /26844/~ibm-watson-tone-analyzer-review Juno, Jennifer., Immunity., Wheatley, Adam (2021). Mounting evidence sugge-

sts COVID vaccines do reduce transmission. How does this work? Retrie- ved from https://www.gavi.org/vaccineswork/mounting-evidence-suggests-covid -vaccines-do-reduce-transmission-how-does-work Li, Q., & Kim, Y. A. (2020, July). Happiness Participatory Media: Cultural Differences in Happiness on Instagram. In International Conference on Applied Human Factors and Ergonomics (pp. 394–402). Springer, Cham. Li, Q., & Kim, Y. A. (2018, July). Analysis of Facial Emotion Recognition Techno-

logy and Its Effectiveness in Human Interaction. In International Conference on Applied Human Factors and Ergonomics (pp. 217-226). Springer, Cham. Meikle, G. (2016). Social media: Communication, sharing and visibility. Routledge. Neumeier, M. (2005). The Brand Gap: Revised Edition. Peachpit Press. Perry, Rose., Braren, Stephen Braren. (July 2021). Social Connectedness 101: Defining Social Connectedness. Retrieved from https://www.thesocialcreatures .org/thecreaturetimes/social-connectedness-101-defining-social-connectedness#: ~{}:text=Humans%20are%20social%20creatures%20who,even%20how%20l ong%20we%20live. Rodriguez, Tori (2013). Negative Emotions Are Key to Well-Being. Retrieved from

https://www.scientificamerican.com/article/negative-emotions-key-well-being/

Text vs. Images: Understanding Emotional Expressions 27

Social Video: The Marketing Trend that’s Worth Investing. Retrieved from in https: //www.bluecorona.com/blog/social-video-marketing-trend/ Soucheray S (June 29, 2021). US COVID-19 cases rise, likely due to Delta variant.

CIDRAP. Retrieved July 11, 2021. WHO Coronavirus (COVID-19) Dashboard. Retrieved from https://covid19. who.int/ Yean, Chew (2015). Emotion Detection and Recognition from Text Using Deep Learning. Retrieved from https://devblogs.microsoft.com/cse/2015/11/29/emotion-de tection-and-recognition-from-text-using-deep-learning/

Cover of the proceedings volume Human Factors in Communication of Design

Explore the complete digital volume

Human Factors in Communication of Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-25-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →