Matthew John Sino Cruz, Marlene De Leon (2022). Analysis of citizen's sentiment towards Philippine administration's intervention against COVID-19. In: Artificial Intelligence and Social Computing. AHFE Open Access, vol. 28, pp. 50–58. https://doi.org/10.54941/ahfe1001446
The COVID-19 pandemic affected the world. The World Health Organization or WHO issued guidelines the public must follow to prevent the spread of the disease. This includes social distancing, the wearing of facemasks, and regular washing of hands. These guidelines served as the basis for formulating policies by countries affected by the pandemic. In the Philippines, the government implemented different initiatives, following the guidelines of WHO, that aimed to mitigate the effect of the pandemic in the country. Some of the initiatives formulated by the administration include international and domestic travel restrictions, community quarantine, suspension of face-to-face classes and work arrangements, and phased reopening of the Philippine economy to name a few. The initiatives implemented by the government during the surge of COVID-19 disease have resulted in varying reactions from the citizens. The citizens expressed their reactions to these initiatives using different social media platforms such as Twitter and Facebook. The reactions expressed using these social media platforms were used to analyze the sentiment of the citizens towards the initiatives implemented by the government during the pandemic. In this study, a Bidirectional Recurrent Neural Network-Long Short-term memory - Support Vector Machine (BRNN-LSTM-SVM) hybrid sentiment classifier model was used to determine the sentiments of the Philippine public toward the initiatives of the Philippine government to mitigate the effects of the COVID-19 pandemic. The dataset used was collected and extracted from Facebook and Twitter using API and www.exportcomments.com from March 2020 to August 2020. 25% of the dataset was manually annotated by two human annotators. The manually annotated dataset was used to build the COVID-19 context-based sentiment lexicon, which was later used to determine the polarity of each document. Since the dataset contained unstructured and noisy data, preprocessing activities such as conversion to lowercase characters, removal of stopwords, removal of usernames and pure digit texts, and translation to the English language were performed. The preprocessed dataset was vectorized using Glove word embedding and was used to train and test the performance of the proposed model. The performance of the Hybrid BRNN-LSTM-SVM model was compared to BRNN-LSTM and SVM by performing experiments using the preprocessed dataset. The results show that the Hybrid BRNN-LSTM-SVM model, which gained 95% accuracy for the Facebook dataset and 93% accuracy for the Twitter dataset, outperformed the Support Vector Machine (SVM) sentiment model whose accuracy only ranges from 89% to 91% for both datasets. The results indicate that the citizens harbor negative sentiments towards the initiatives of the government in mitigating the effect of the COVID-19 pandemic. The results of the study may be used in reviewing the initiatives imposed during the pandemic to determine the issues which concern the citizens. This may help policymakers formulate guidelines that may address the problems encountered during a pandemic. Further studies may be conducted to analyze the sentiment of the public regarding the implementation of limited face-to-face classes for tertiary education, implementing lesser restrictions, vaccination programs in the country, and other related initiatives that the government continues to implement during the COVID-19 pandemic.
SARS-CoV-2 virus, popularly known as COVID-19, is a communicable dis- ease that affected 114 countries since it started in the year 2019. On March 11, 2020, the World Health Organization officially categorized the disease as a pandemic due to the increasing number of cases around the world. To date, there have been 252,902,685 cumulative cases from all over the world since the pandemic started (WHO 2021). To prevent the rapid spread of the disease, WHO issued guidelines that the public should follow. Follow- ing the guidelines set by WHO, the Philippine government imposed several restrictions and initiatives to mitigate the effect of the pandemic. Some of the initiatives implemented by the government include travel restrictions, community quarantine, and suspension of face-to-face classes and work arrangement to name a few (Inter-agency Task Force 2020). These initia- tives have triggered untimely closure of some business establishments and unemployment due to the economic downturn during the pandemic. The ini- tiatives imposed by the government, the situation of the country during the
Analysis of Citizen’s Sentiment Towards Philippine Administration’s Intervention 51
pandemic, and how the situation is being handled have resulted in different reactions from the citizens. These reactions are mostly expressed using several social media platforms.
During the pandemic, social media platforms served as a medium for the government and WHO to provide relevant information to the citizens about the COVID-19 situation in the country. Among these social media platforms are Facebook and Twitter. As of March 2021, Facebook reported that there are about 1.88 billion daily active users (Facebook 2021) and 199 million Twitter users (Twitter 2021). Due to a large number of users on these plat- forms, they were used as a means to report daily updates about COVID-19 situation in the country. These platforms allow the citizens to express their sentiments about how the government handles the pandemic (Imran et al. 2013). These sentiments can provide information about the thoughts of the citizens towards these initiatives.
The nature of posts and tweets can be qualified as unstructured and noisy textual data. This is because social media users tend to use shortened and misspelled words when writing posts and tweets due to the limited characters allowed (Bhargava et al. 2019, Virmani et al. 2017, Habib and van Keulen 2014). They also don’t use explicit sentiment words to express their opinion which makes it hard to classify whether the text expresses positive or nega- tive sentiment (Son et al. 2019, Wei et al. 2019). Another challenge that needs to be addressed in the field of sentiment analysis is the use of colloquial and curse words which are not recognized in the Standard English dictionary. In addition, Filipino social media users tend to use mixed language or dialect in writing comments, posts, and tweets. This poses a challenge in the field of sentiment analysis since most of the sentiment classifier models are desi- gned to classify text data written in one language (Bhargava et al. 2019). In addition, although there are sentiment lexicons available, very few are desi- gned for text data written in the Filipino language that includes colloquial and curse words (Samuel et al. 2020).
In this study, a hybrid Bidirectional Recurrent Neural Network - Long Short-Term Memory - Support Vector Machine (BRNN-LSTM+SVM) senti- ment classifier model was developed to determine the sentiment of the citizens towards the initiatives of the Philippine government in mitigating the effe- cts of COVID-19 pandemic in the country. The performance of the model was measured and compared to the performances of the native Bidirectional Recurrent Neural Network and Support Vector Machine in classifying posts and tweets about the pandemic.
The main contribution of the paper is the development of a hybrid senti- ment classifier model to determine the sentiment of the public towards the different initiatives imposed by the government during the pandemic. In addi- tion, a method in generating a corpora-based COVID-19 sentiment lexicon was designed and implemented. Lastly, a COVID-19 sentiment dataset was created from both Twitter and Facebook datasets.
The remainder of this paper is outlined as follows, Section 2 discusses the methodology employed in the study, Section 3 provides the discussion of the results after performing a series of experiments, and Section 4 provides the conclusion and future direction of the study.
52 Sino Cruz and De Leon
METHODOLOGY
Data Collection and Annotation The dataset collected came from Facebook and Twitter from March 2020 to August 2020. The dataset collected was based on the keywords related to the initiatives of the Philippine government. The initiatives used as a basis for keywords searched, include travel restrictions, community quarantine, social amelioration program, ‘Balik-probinsya’ program, suspension of face to face classes and work arrangement, among others. The tweets and comments were extracted using Facebook and Twitter API and www.exportcomments.com. After performing the extraction process, there were 16,000 tweets and com- ments collected. 25% of the collected tweets and comments were manually annotated. The manually annotated dataset was used to establish the ground truth for the study. The human annotators also corrected words that are mis- spelled and inappropriately abbreviated which helps the automatic translator in translating the dataset. To determine the raters’ consistency test, the result of the labeling process was tested using Cohen Kappa’s inter-rater reliability (McHugh 2012). The computed kappa value was 0.78 which is interpre- ted as a substantial agreement which means that the agreement between the annotators is considered acceptable.
Preprocessing Stage Preprocessing was performed to clean the Twitter and Facebook datasets. It includes the conversion of characters to lowercase, removal of URL, removal of punctuation marks, removal of texts containing pure numbers, removal of irrelevant comments such as those that provide financial advising and information regarding flight cancellations, and removal of stopwords, and stemming. The task also includes the translation of the dataset to the English language. A python program, which implements Google Translate API, was written to perform the translation. The translated dataset was used to build a word list which was tagged with their appropriate part of speech labels using NLTK POS tagger.
Generation of Corpora-based Sentiment Lexicon Sentiment lexicon was built using the words included in the word list and was tagged as a verb, adjective, and adverb (Zhang et al. 2018, Deng et al. 2019). It also includes those words tagged as curse words (Samuel et al. 2020) and hashtags since they convey sentiments relevant to the study. Each word from the word list is read and checked in the manually labeled dataset to determine which sentence does the word mostly appears. These words are labeled as positive, negative, or neutral. Words that are labeled as neutral are excluded from the sentiment lexicon.
Sentiment Polarity Identification The sentence sentiment polarity was identified by determining the senti- ment orientation of each word in a sentence based on a sentiment lexicon. The sentiment score is computed based on the difference of the number of
Analysis of Citizen’s Sentiment Towards Philippine Administration’s Intervention 53
word sentiment polarity. These sentences are labeled as positive, negative, or neutral.
Building the Model For this study, the hybrid approach of BRNN-LSTM-SVM was built by inte- grating the SVM as one of the layers of the BRNN-LSTM model. SVM is the last layer of the BRNN-LSTM model. The preprocessed dataset was divided into training, validation, and testing set. 80% of the dataset was ran- domly selected for training and the remaining 20% was left for validation and testing. Global Vector (GloVe) was used to generate word embeddings for the dataset. The generated word embeddings were used as input to the input layer of the model. To avoid overfitting, a 10-fold cross-validation was implemented.
Performance Evaluation In this study, the performance of the proposed hybrid BRNN-LSTM + SVM model is compared to the performance of the traditional machine learning models, BRNN-LSTM and SVM. The performance of these models was com- pared in terms of the sentiment lexicon used and in classifying Facebook and Twitter datasets. The metrics used to measure the performance of the propo- sed hybrid sentiment classifier model includes precision, recall, f1-score, and accuracy.
RESULTS AND DISCUSSION
Sentiment Polarity of Facebook and Twitter COVID-19 Datasets The COVID-19 Corpora-based sentiment lexicon and NRC sentiment lexi- con were used as the basis for identifying the sentiment polarity of each sentence in the unlabeled Twitter and Facebook datasets. Table 1 shows the comparison of the sentiment polarity of the datasets using the two lexicons.
Using Corpora-based sentiment lexicon, 53% of the sentences from both Twitter and Facebook datasets has negative sentiment. This surpassed those sentences/documents with Neutral and Positive sentiments (see Table 1). On the other hand, the NRC sentiment lexicon has resulted to a different sen- timent orientation for both datasets. As observed in table 1, 48.36% of the Facebook dataset was tagged with Positive sentiment which covers the majo- rity of the dataset. For the Twitter dataset, Tweets tagged with Positive and Negative sentiments are almost the same in number which covers around 35% to 37% of the entire dataset. Both datasets labeled using these sentiment lexicons were used to train, validate, and test the machine learning models for performing sentiment classification tasks.
Sentiment Classifier Model Performance Based on Sentiment Lexicons The performance of the sentiment classifier models were measured by using the datasets that are automatically labeled using the corpora-based sentiment
54 Sino Cruz and De Leon
Table 1. Comparison of sentiment polarity using the two lexicons.
lexicon and NRC sentiment lexicon (Mohammad and Kiritchenko 2020). The automatically labeled datasets were used to train, validate, and test the performance of sentiment classifiers. Tables 2 and 3 show the performance of the models based on the automatically labeled datasets using both sentiment lexicons.
Analysis of Citizen’s Sentiment Towards Philippine Administration’s Intervention 55
Table 4. Comparison of sentiment classifier models using two different datasets.
It can be observed that even though the same datasets were used as inputs into the sentiment classifier models, there is still a difference in terms of their performance which can be attributed to the sentiment lexicons used to identify the dataset’s polarity. It can be gleaned that for both Facebook (see Table 2) and Twitter (see Table 3) datasets, the sentiment classifier models produced higher accuracy when corpora-based sentiment lexicon was used to identify the polarity of both datasets. Since corpora-based sentiment lexi- con was generated based on the corpus used in the study, the sentiment of each word contained in the lexicon was based on the context in which these words were used.
The use of generated Corpora-based sentiment lexicon has proven to be effective in improving the performance of machine learning algorithms such as SVM in performing several text classification tasks including sentiment analysis and topic modeling (Lin et al. 2014, Deng et al. 2019).
Sentiment Classifier Model Performance Based on Two Datasets The performance of sentiment classifier models was also measured and compared based on the datasets used (see Table 4).
It can be observed that SVM performs better with a Facebook dataset with 91% accuracy compared to its performance using the Twitter dataset with 89% accuracy. Likewise, BRNN, with 96% accuracy, and BRNN-LSTM + SVM, with 95% accuracy, perform better when the Facebook dataset was used compared to their performance using the Twitter dataset.
In addition, it is important to note that even though the accuracy of the models are high, there are drawbacks concerning its ability to classify datasets with positive sentiment. As shown in Table 4, each model (for both Twitter and Facebook datasets) posed high precision, recall, and F-score for iden- tifying textual data with negative sentiment. However, the same metrics are relatively low when dealing with textual data having positive sentiment. This means that even though the sentiment classifier models achieved relatively high accuracy, it would be easier for the model to detect textual data with the negative sentiment but may suffer detecting textual data with positive sentiment. To address this concern, stemming was applied for both datasets.
56 Sino Cruz and De Leon
Table 5. Comparison of sentiment classifier models using two different datasets after Stemming.
The performance of the models after applying the stemming process is shown in Table 5.
It can be observed that the accuracy achieved by the SVM, BRNN, and hybrid BRNN-LSTM + SVM sentiment classifiers declined when stemming was applied to the Facebook dataset. On the other hand, a slight increase in accuracy was observed on the stemmed Twitter dataset for SVM. In addi- tion, the accuracy achieved for BRNN and hybrid BRNN-LSTM + SVM after applying stemming to the Twitter dataset was maintained even before the dataset underwent the stemming process. However, it is important to note that even though the accuracy of the models declined after applying stem- ming, the precision, recall, and F-score for determining textual data with positive sentiments increased. This means that the stemming process helped the models for detecting both positive and negative sentiments among tex- tual data by overcoming lexical sparsity and issues on ambiguity (Elfaik and Nfaoui 2021). Hence, it indicates that the sentiment classifier model, when implemented with stemming, has a small error rate and will be more accurate in predicting classes of objects (Rianto et al. 2021).
CONCLUSION AND FUTURE WORK The study contributes to the body of knowledge in the field of natural langu- age processing by proposing a hybrid model for classifying non-English text data for their sentiments. By conducting series of experiments, it was found out that the use of corpora-based sentiment lexicon for determining sente- nce sentiment polarity helps improve the performance of the hybrid model in classifying textual data. This is because the sentiment lexicon was able to capture the context of each word used in the datasets. However, it is impor- tant to note that gaining higher classification accuracy may pertain to its ability to classify one sentiment class but may suffer identifying the other class. Hence, the stemming process was proven to be effective in improving the performance of the sentiment classifier models in classifying the text data for both sentiment classes.
As shown in the study, the results indicate that the citizens harbor negative sentiments towards the initiatives of the government in mitigating the effects of the COVID-19 pandemic. The government can use the result of the study
Analysis of Citizen’s Sentiment Towards Philippine Administration’s Intervention 57
as a basis in evaluating the initiatives implemented during the pandemic and to further refine the existing plans related to current crisis.
For future work, other existing deep learning models may be combined with the proposed model to improve its performance. Other features such as n-grams may also be considered to improve the performance of the pro- posed model. Furthermore, the impact of increasing and decreasing hidden layers in a deep learning model in performing sentiment analysis may also be considered for future work.
REFERENCES
Bhargava, R., Arora, S. and Sharma, Y. (2019). Neural Network-Based Architecture for Sentiment Analysis in Indian Languages. Journal of Intelligent Systems [online], 28(3), pp. 361–375. Available from: https://www.degruyter.com/docume nt/doi/10.1515/jisys-2017-0398 [Accessed 30 August 2021] Deng, D., Jing, L., Yu, J., Sun, S. and Ng, M. (2019). Sentiment Lexicon Construction
With Hierarchical Supervision Topic Model. IEEE/ACM Transactions on Audio, Speech, and Language Processing [online], 27(4), pp. 704–718. Available from: https://ieeexplore.ieee.org/document/8607058 [Accessed 30 August 2021]. Elfaik, H. and Nfaoui, E. (2020). Deep Bidirectional LSTM Network Learning-Based Sentiment Analysis for Arabic Text. Journal of Intelligent Systems [online], 30(1), pp. 395–412. Available from: https://www.degruyter.com/document/doi/10.1515/ jisys-2020-0021 [Accessed 20 July 2021]. Facebook, 2021. Facebook Reports First Quarter 2021 Results. [online]. Availa-
ble from: https://s21.q4cdn.com/399680738/files/doc_news/Facebook-Reports- First-Quarter-2021-Results-2021.pdf [Accessed 21 July 2021]. Habib, M. and van Keulen, M., 2014. Proceedings of the Third Workshop on Semantic Web and Information Extraction. In: Third Workshop on Semantic Web and Information Extraction [online], Dublin, Ireland: Association for Com- putational Linguistics and Dublin City University, pp. 9–16. Available from: https://aclanthology.org/W14-6202.pdf [Accessed 19 November 2021]. Imran, M., Elbassuoni, S., Castillo, C., Diaz, F. and Meier, P., 2013. Extracting
Information Nuggets from Disaster-Related Messages in Social Media. In: 10th International ISCRAM Conference [online], Available from: https://mimran.me/ papers/imran_shady_carlos_fernando_patrick_iscram2013.pdf [Accessed 17 July 2021]. Inter-agency Task Force (IATF) (2020). Omnibus guidelines on the implementation of community quarantine in the Philippines [online]. Available from: https: //www.officialgazette.gov.ph/downloads/2020/05may/20200429-Omnibus-Guid elines-on-the-Implementation-of-Community-Quarantine-in-the-Philippines.pdf [Accessed 30 August 2021] Lin, L., Li, J., Zhang, R., Yu, W. and Sun, C. (2014). Opinion mining and
sentiment analysis in social networks: A retweeting structure-aware appro- ach. In: 2014 IEEE/ACM 7th International Conference on Utility and Cloud computing [online], pp. 890–895. Available from: IEEE Xplore [Accessed 21 July 2021]. McHugh, M., 2012. Interrater reliability: the kappa statistic [online]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/ [Accessed 20 July 2021].
58 Sino Cruz and De Leon
Mohammad, S. and Kiritchenko, S., (2020). NRC Emoticon Lexicon. In: Sentiment140 Lexicon [online], Available at: https://saifmohammad.com/WebPages/ AccessResource.htm [Accessed 20 December 2020]. Rianto, Mutiara, A., Wibowo, E. and Santosa, P. (2021). Improving the accuracy
of text classification using stemming method, a case of non-formal Indonesian conversation. Journal of Big Data [online], 8(1). Available from: https://jour nalofbigdata.springeropen.com/articles/10.1186/s40537-021-00413-1 [Accessed 17 November 2021]. Samuel, J., Rahman, M., Ali, G., Samuel, Y., Pelaez, A., Chong, P. and Yakubov, M. (2020). Feeling Positive About Reopening? New Normal Scenarios From COVID-19 US Reopen Sentiment Analytics. IEEE Access [online], 8, pp. 142173– 142190. Available from: https://ieeexplore.ieee.org/document/9154672 [Accessed 21 November 2021]. Son, L., Kumar, A., Sangwan, S., Arora, A., Nayyar, A. and Abdel-Basset, M. (2019).
Sarcasm Detection Using Soft Attention-Based Bidirectional Long Short-Term Memory Model With Convolution Network. IEEE Access [online], 7, pp. 23319– 23328. Available at: https://ieeexplore.ieee.org/document/8641269 [Accessed 21 November 2021]. Twitter, 2021. Q1 2021 Letter to Shareholders. [online]. Available from: https://s22.q4cdn.com/826641620/files/doc_financials/2021/q1/Q1%2721- Shareholder-Letter.pdf [Accessed 21 July 2021]. Virmani, C., Pillai, A. and Juneja, D. (2017). Extracting information from social network using NLP. International Journal of Computational Intelligence Research [online], 13(4), pp. 621–630. Available from: https://www.ripublication.com/ijci r17/ijcirv13n4_15.pdf [Accessed 21 November 2021]. Wei J., Liao J., Yang Z., Wang S., and Zhao Q. (2020). BILSTM with multi-polarity
orthogonal attention for implicit sentiment analysis. Neurocomputing [online], 383, pp.165–173. Available from: https://ieeexplore.ieee.org/document/9151169 [Accessed November 21, 2021]. WHO Coronavirus (COVID-19) Dashboard. (2021). WHO Coronavirus (COVID-
19) Dashboard. [online] Available at: https://covid19.who.int/ [Accessed 21 November 2021]. Zhang, L., Wang, S. and Liu, B. (2018). Deep learning for sentiment analysis: A survey. WIREs Data Mining and Knowledge Discovery [online], 8(4). Available at: https://arxiv.org/ftp/arxiv/papers/1801/1801.07883.pdf [Accessed 21 October 2021].
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Artificial Intelligence and Social Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-04-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional