Kohei Otake, Ryo Morooka (2024). Examination of Evaluation Indices for Micro-Influencers Considering Community Structure and Post Contents. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 159, pp. 182–192. https://doi.org/10.54941/ahfe1005581
Social networking services (SNS) have become indispensable communication tools. Consequently, influencer marketing, which leverages users with a significant influence on SNS, has garnered significant attention. Among these influencers, micro-influencers, who have substantial influence within specific domains, are particularly interesting to both academia and industry. This study proposes evaluation indices that can effectively select micro-influencers for product promotion using follower data and past posts from SNS accounts. Specifically, we propose four evaluation indices for micro-influencers: Virality, Commonality, Expertise and Credibility (VC-EC indices). VC indices are based on network features, whereas EC indices are based on language features. In this study, we present the concepts and specific calculation methods for the proposed indices. In addition, we demonstrate how to discover micro-influencers using the proposed methods with practical examples from accounts operated by actual stores.
In recent years, social networking services (SNS) have become widely popular globally and have established themselves as communication tools. SNS refers to services that have functions to promote and support connections between people, with the value of the service stemming from communication among users. According to a survey by ICT Research Institute, the number of SNS users has reached 82.7 million, with over 80% of domestic Internet users reported to be using SNS (ICT Research & Consulting Inc., 2022). Under these circumstances, the use of SNS in the field of marketing is becoming essential. Such marketing initiatives that utilize SNS are called social media marketing (SMM). From an academic perspective, research on SMM is actively conducted in the fields of marketing and consumer behavior. Through a systematic review focused on the marketing field, Mari has classified the research areas related to SMM into four categories:
Examination of Evaluation Indices for Micro-Influencers Considering 183
eWOM/UGC, social listening, community, and communication channel (Mori. H, 2023).
Focusing on specific research topics, influencer marketing, which is a general term for marketing utilizing influencers on SNS, is attracting significant attention from both industry and academia. The term “influencer” originates from the word “influence,” meaning impact, inspiration, or effect, and refers to people or things that have a substantial impact on society. Specifically, they include celebrities, fashion models, athletes, experts, and individual bloggers with a strong influence on the Internet. On the other hand, users who do not have influence in the real world but have approximately 10,000 to 100,000 followers are called micro-influencers. Many micro-influencers gain influence by disseminating specialized information in specific domains on SNS; in their areas of expertise, they can even surpass the influence of real-world influencers. Micro-influencers are gaining attention in business because they are more cost-effective than influencers and are expected to have strong promotional effects in specific domains.
In a study focusing on the role and impact of influencers on consumer behavior, Chen et al. proposed and tested an integrated model to explain the role of advertising value and source credibility (Chen. L and Shupei. Y, 2019). The results indicated that the informational value, trustworthiness, attractiveness, and similarity positively influenced “trust in branded posts” and “trust in branded posts” influenced brand awareness and purchase intentions. Jiang evaluated the role of live commerce streamers as influencers (Jiang. Y, 2022). The results showed that influencers act as intermediaries between brands and customers, positively influencing customer engagement.
Some studies have attempted to identify influencers using social media data. Kitajima et al. targeted multiple cosmetic brands, obtained data on consumer connections (follow-follower relationships), and clarified the network structure formed among users (Kitajima. Y, et al., 2022). They identified micro-influencers for each brand, as well as mega-influencers in the cosmetics industry. Zhang et al. measured the similarity between brands and influencers using hashtags on social media and demonstrated the effectiveness of their proposed method (Zhang. Y, et al., 2023). Yamamoto et al. proposed a method for predicting the rise and fall in Bitcoin prices based on influencers’ tweets, suggesting that influencers’ tweets could affect cryptocurrency prices (Yamamoto. H, et al., 2019).
As exemplified by these studies, research on influencers includes efforts to clarify hypotheses about the roles and impact of influencers based on qualitative data, such as surveys using structural equation modelling, as well as studies aimed at identifying and utilizing influencers using actual data on connections and texts related to influencers. However, in practice, when engaging in influencer marketing, it is necessary to find the right person among many candidates to promote a company’s products or services. In other words, the candidates must be evaluated and selected based on certain criteria. Currently, many companies that conduct influencer marketing focus on the number of followers, which represents the number of subscribers a user has (degree in the network structure). Number of subscribers is an important
184 Otake and Morooka
indicator of information dissemination. However, as mentioned previously, micro-influencers with tens of thousands of subscribers, who can be expected to have a significant impact in specific domains, may be overlooked if only the number of subscribers is considered. Discovering micro-influencers requires a more precise evaluation from multiple perspectives, such as the attributes of candidates, network attributes, and past posts. This study proposes quantitative evaluation metrics for micro-influencers that focus on the connections between followers and posted texts. Specifically, we propose evaluation indices based mainly on connection and posting information obtained from X using social network analysis and natural language processing.
UNDERSTANDING NETWORK STRUCTURES AND EXAMINING FEATURES FOR EVALUATION INDICES First, we present an overview of our research on store account characteristics using a social network analysis (Morooka. R, et al., 2023). Social network analysis, based on graph theory, is an analytical method that explores the relationship structures among components in various subjects, such as human relationships, distribution networks, and webpage links.
We obtained store follower data using XAPI and conducted social network analysis. The target stores are Stores A and B, located in Tokyo and operated by a company running electronics retail stores. Store A primarily sells home appliances and PC-related products, whereas Store B primarily sells anime- related goods. For the analysis, we used data on user IDs, account names, and other information of the followers of Stores A and B (referred to as primary followers in this study), as well as data on users following these primary followers (referred to as secondary followers in this study). Store A had 1,445 primary followers and 1,554,035 secondary followers, whereas Store B had 3,702 primary followers and 2,955,182 secondary followers. However, we excluded accounts that we determined were operated by companies from our analysis. Using these data, we constructed social networks for Stores A and B. In these networks, nodes represent primary followers and edges represent the presence of common secondary followers between primary followers, with the number of common secondary followers used as the edge weight. We also performed community detection on the constructed social networks. Based on the edge list and weights that indicate the node connections in each store, community detection was performed using the Modularity Q (Newman. M. E. J and Girvan. M, 2004) (Equation (1)).
Equations and mathematical formulas are available in the original paper PDF.
Aij −kikj
ci, cj
2m ij
where, m represents the sum of the weights of all edges in the network, Aij represents the weights of the edges of node i and node j, ki represents the sum of the weights of the edges bound to node i and kj represents the sum of the weights of the edges bound to node j. δ is the Kronecker delta, ci represents the community to which node i belongs and cj represents the community to
Examination of Evaluation Indices for Micro-Influencers Considering 185
which node j belongs. In this study, we performed community detection using the Louvain method (Traag. V. A, et al., 2019) with modularity.
As a result of community detection, five communities were identified in Store A and four communities were identified in Store B. Based on the results of community detection, we attempted to understand the characteristics of each community. Specifically, we reviewed the profiles of the top ten accounts with the highest PageRank in each community. The results for Stores A and B are listed in Tables 1 and 2.
Table 1. Characteristics of accounts with high pagerank in each community (Store A).
Table available in the original PDF paper.
Community Characteristics of Community Number of Users No.
341
0 Giveaway accounts 573 1 Personal accounts related to games (including PC) and anime
79
2 Giveaway accounts 47 3 Store accounts related to electronics retail stores and home appliances
24
4 Store accounts related to electronics retail stores and home appliances
Table 2. Characteristics of accounts with high pagerank in each community (Store B).
Table available in the original PDF paper.
Community Characteristics of Community Number of Users No.
987
0 Accounts that post news about anime and manga, and information related to hobbies
314
1 Personal accounts of anime enthusiasts, artists, and hobbyist gamers
30
2 Accounts of electronics retail stores 47 3 Corporate accounts that sell home appliances and stationery
Table 1 shows that among the followers of Store A, there is a large community (Number 0) consisting of accounts aimed at giveaway campaigns. Another large community (Number 1) comprised of personal accounts related to games, anime, and PC games.
From Table 2, it is evident that in the large communities (Numbers 0 and 1) of Store B, many accounts post news about anime and manga, as well as information related to hobbies. This indicates that these communities were composed of users interested in anime and manga.
Furthermore, in both networks, it was found that small communities (Store A: Number 3 and 4, Stores B: Numbers 2 and 3) included many accounts of nearby electronics retail stores and major electronics retail stores.
The results of these analyses reveal that despite the stores being in the same chain, the interests and concerns of followers differ between stores. It
186 Otake and Morooka
was also found that both Stores A and B have communities formed around the categories of products they handle and that there are differences in network structures. The features obtained from these network structures (network features) can be valuable sources of information for discovering micro-influencers suitable for stores.
On the other hand, when selecting candidates, checking their past posts is crucial not only to ensure compatibility with the products they will be promoting but also from a risk management perspective. However, the aforementioned network analysis alone can only provide a rough idea of the topics at the community level, as inferred from the profile information of users in the community. It cannot capture the specific content of individual user posts. Therefore, in this study, we examine whether posts that users have made in the past, used as features (language features), can be utilized as evaluation indicators.
First, we obtained post data for accounts with more than 1,000 followers among the primary followers of Stores A and B. Due to API constraints, the collected post data were limited to the most recent week based on the collection date and included the account’s own posts and quoted retweets. The reason for setting a lower limit on the number of followers for data collection is that in practical terms, a minimum number of followers is necessary when requesting influencer marketing and to avoid complicating the network structure. Table 3 shows the information used in this study from the collected post data, along with an overview.
Table 3. Information on posts used in this study and their overview.
Table available in the original PDF paper.
Information Overview Time Date of the post Content Post contents Like Number of likes on the post User ID User identifier Profile Profile text
In the next section, we propose an evaluation index for micro-influencers that combines the features extracted from collected post data (language features) with those obtained from the network (network features).
PROPOSAL OF EVALUATION INDICES FOR MICRO-INFLUENCERS Based on the elements constituting the reliability of influencers proposed by Chen et al. (Chen. L and Shupei. Y, 2019), we propose four evaluation indices for micro-influencers: Virality, Commonality, Expertise and Credibility (VC- EC indices). VC indices are based on network features, whereas EC indices are based on language features. An overview of each index is provided below:
Virality: The degree to which information can be spread within the network.
Examination of Evaluation Indices for Micro-Influencers Considering 187
Commonality: The extent to which the followers of the store account and the micro-influencer overlap.
Expertise: The extent to which similar words representing product features are used in posts.
Credibility: Whether the account is operated appropriately by a company requesting influencer marketing.
Each evaluation index is expressed as a value between zero and one and is used in the selection of candidates.
Virality A major reason for employing influencers for promotions is the expectation that the requested information will be widely and quickly disseminated to others. We propose Virality as an evaluation index, which indicates the degree to which a node (candidate) in the network has the power to transmit information to other nodes. Virality is a metric calculated using network features that represent centrality in the context of social network analysis. Various centrality indices can be considered; however, in this study, we use PageRank, which is also known as the metric Google uses to evaluate the importance of web pages (Page. L, et al., 1999). The basic idea of PageRank is that pages with more links from other pages rank higher and links from highly ranked pages are highly valued. In addition, links from pages with fewer links to other pages are more highly valued because they are considered more selective. In this study, we use the normalized PageRank value for each node as an evaluation index for Virality (Equation (2)).
Viralityi = Pageranki −Pagerankmin Pagerankmax −Pagerankmin where, Pagerankmin refers to the minimum value among the evaluated targets and Pagerankmax refers to the maximum value. Virality ranges from 0 to 1, with a value closer to 1 indicating a higher Virality within the network.
Commonality When selecting micro-influencers, it is particularly important to consider which users you want your promotion to reach. For example, the appropriate micro-influencer differs depending on whether the target users are fans of the service or brand, or potential customers. We propose Commonality as an evaluation index, which represents the degree of overlap between the store and each node (candidate). Commonality is calculated using network features by comparing the primary followers (current fans) of a store with the followers of each node to evaluate the extent of common followers. As a specific evaluation method, we used the Dice coefficient, which is a measure of the similarity between two sets. Let A be the set of primary followers of the store, and Bi be the set of followers of node i. The similarity between the two sets, Dice(A, Bi) is represented as follows (Equation (3)).
A ∩Bi
+ |Bi| (3)
188 Otake and Morooka
The calculated Dice(A, Bi) values are normalized and used as the Commonality for each node (Equation (4)).
Commonalityi = Dice(A, Bi) −Dicemin Dicemax −Dicemin where, Dicemin refers to the minimum value among the evaluated targets and Dicemax refers to the maximum value. Commonality ranges from 0 to 1, with a value closer to 1 indicating that the node has followers similar to the store’s account followers. Therefore, if an analyst wants to approach a company’s current customers, they or should select nodes with high Commonality. Conversely, if they want to approach potential customers, they must select nodes with a low Commonality.
Expertise When requesting promotions for actual products or services, the higher the match between the domain in which the influencer excels and the target, the greater the expected effect. We propose Expertise as an evaluation index, which represents the alignment of the influencer with the product using language features. Specifically, we evaluate the influencer’s past posts using words that represent the product’s characteristics (hereafter referred to as feature words) and semantically similar words, expressing this as Expertise.
First, we selected feature words for the product. This varies depending on the product in question, but here we assume a product characterized by the word “ゲーム(means “game” in English).” Influencers who frequently post content including this feature word “ゲーム” are considered to have knowledge about the product and its domain, making them suitable candidates for promotion. However, because not all posts directly include the set feature word, the surrounding words with similar meanings need to be evaluated. Therefore, we used distributed representations of words to obtain similar words with high cosine similarity to the feature words. To obtain the distributed representations, we used chiVe, a pre-trained Word2vec model adapted for Japanese (Manabe. Y, et al., 2019). Table 4 shows an example of the top five words with the highest cosine similarity collected for the feature word “ゲーム.” Hereafter, the English translations are given in parentheses.
Table 4. Top 5 words similar to the feature word “ゲーム(game)” and their cosine similarity.
Table available in the original PDF paper.
Rank Similar Words Cosine Similarity 1 RPG 0.69 2 オンラインゲーム(Online Game) 0.68 3 ミニゲーム(Mini-game) 0.67 4 game 0.66 5 ロールプレイングゲーム(Role-playing game) 0.65
Next, we evaluated the influencers’ posts using feature words and similar words. Let k denote a feature word. The evaluation value s(k)
i for candidate i
Examination of Evaluation Indices for Micro-Influencers Considering 189
is represented as follows (Equation (5)).
n X
s(k)
dk,jNi,k,j (5)
where, dk,j represents the similarity of word j to feature word k, and Ni,k,j represents the number of posts by influencer i that include a similar word j related to feature word k. It is assumed that multiple feature words are set for the product. For example, for a gaming PC, feature words such as “game,” “graphics,” and “memory” can be considered. Therefore, let l be the total number of feature words and let the evaluation value si for each account be the sum of the evaluation values for all feature words (Equation (6)).
l X
s(k)
i (6) k = 1 The normalized value of the calculated evaluation value si was used as the Expertise Index for each node (Equation (7)).
where, smin refers to the minimum value among the evaluated targets and smax refers to the maximum value. Expertise ranges from 0 to 1, with a value closer to 1 indicating that the posts are more representative of a product’s features.
Credibility Owing to the nature of social media, information can spread rapidly, both positively and negatively. Therefore, it is crucial to carefully select influencers for business purposes to avoid spreading inappropriate expressions or misinformation that can lead to “flaming.” We propose Credibility as an evaluation index. Specifically, we used language features to identify inappropriate expressions in past posts and calculated the credibility scores.
We created a “List of Words That Can Potentially Hurt People” by identifying inappropriate posts. This list comprises 109 words commonly perceived as abusive or sexual harassment on the Internet (for example, words like “バカ(idiot),” “あほ(fool),” “デブ(fat),” and “殺す(kill)”). Using the words from this list, we evaluated the credibility of the text data posted in the previous week.
Let Ni be the number of tweets posted by candidate i in the past and Mi be the number of tweets containing words from the “List of Words That Can Potentially Hurt People.” The Credibilityi for each candidate is represented as follows (Equation (8)): (8)
where Credibility ranges from 0 to 1, with a value closer to 1 indicating a lower risk of inappropriate posts. However, as a preprocessing step for
190 Otake and Morooka
evaluating Credibility, it is necessary to exclude accounts that have made very few posts or accounts that explicitly state in their profiles that their purpose is for sweepstakes or free gifts.
Selection Method for Micro-Influencers Using Evaluation Indices Using the four indices described in the previous section, we outlined a concrete method for selecting influencers. First, we set a lower limit for Commonality based on the purpose of the product or service promotion. For the PR aimed at existing customers, we set a high lower limit for Commonality. If the goal is to reach potential (unknown) customers, we set a lower limit for Commonality. We then considered whether to prioritize Expertise or Virality. Although it is ideal to have high values for both, it is necessary to establish a priority here. Next, we sorted the candidates using the selected indices and selected the micro-influencers from the top candidates displayed. Furthermore, Credibility checks must be conducted. Excluding influencers with significantly low Credibility can reduce unnecessary risks.
As a practical example, we targeted the actual followers of Store A and selected the top five micro-influencers, prioritizing Expertise as the most important index. The indices of the top five micro-influencers are listed in Table 5. Through discussions with analysts involved in influencer marketing, we set the following conditions for the accounts to be evaluated.
Must be a primary follower of the store
Must be a general account, not a corporate account
Must have more than 1,000 followers
Must have posted within the last week
Must be used for regular communication purposes A total of 174 accounts met these conditions. Additionally, as feature words representing the product, assuming a gaming PC, we set “メモリ(PC memory),” “グラフィック(graphics),” and “ゲーム(game).” Based on the distributed representations of these words, we collected the top 30 words with the highest cosine similarity and used them to evaluate Expertise.
Table 5. Five micro-influencers selected from Store A’s followers and their four evaluation indices.
Table available in the original PDF paper.
Rank Account ID Virality Commonality Expertise Credibility
In Table 5, the influencers were sorted based on their Expertise values; thus, the influencer with the highest Expertise value was ranked first. From the perspective of Virality, the second-ranked ID 65 had a very high value of 0.75, whereas accounts such as ID 43 and ID 17 had values below 0.1.
Examination of Evaluation Indices for Micro-Influencers Considering 191
Regarding Commonality, ID 65, which had high Virality, also had the highest Commonality value of 0.16, whereas ID 31, which ranked first in Expertise, had a relatively low value of 0.06. Regarding Credibility, all the influencers had values exceeding 0.9, indicating that more than 90% of their posts did not contain inappropriate words.
Considering these results, if the priority is alignment with the product features, ID 71, with high Expertise and moderate Virality, is a strong candidate. On the other hand, while ID 65 had lower Expertise than ID 71, it boasted high values in both Virality and Commonality, making it attractive for reaching both current and potential customers. Thus, analysts can make selections from multiple perspectives.
CONCLUSION
This study proposes evaluation indices that can effectively select micro- influencers for product promotion. First, we compared the trends in follower networks through social network analysis, targeting two actual store accounts of electronics retail stores. We also examined the network features that could be obtained from the created networks. Furthermore, we collected post data related to potential micro-influencer candidates that appeared in the network and investigated the use of these posts as language features through natural language processing analysis. Based on the results of the analysis, we propose Virality, Commonality, Expertise and Credibility (VC-EC indices). These indices are calculated based on the attributes of the followers of micro-influencers’ followers and their past posts, enabling marketers and analysts to evaluate micro-influencers quantitatively when making their selections.
In future studies, it will be necessary to evaluate the effectiveness of the proposed indices; preparations are currently underway. Specifically, we plan to collaborate with companies engaged in influencer marketing to select influencers using the proposed method and to verify its effectiveness through actual initiatives. In addition, we will develop a decision support tool by visualizing the proposed indices.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number 21K13385.
REFERENCES
Chen. L and Shupei. Y (2019), “Influencer Marketing: How Message Value and Credibility Affect Consumer Trust of Branded Content on Social Media,” Journal of Interactive Advertising, Vol. 19, Issue. 1, pp. 58–73. ICT Research & Consulting Inc. (2022), “Overview of 2022 Social Media Usage
Trends Report.” (in Japanese) Jiang. Y (2022), “Influence of Influencer in Live Commerce on Customer Engagement,” Journal of Japan Management Diagnosis Association, Vol. 22, pp. 89–95. (in Japanese)
192 Otake and Morooka
Kitajima. Y, Otake. K and Namatame. T (2022), “Evaluation of Consumer Network Structure for Cosmetic Brands on Twitter,” International Journal of Advanced Computer Science and Applications, Vol. 13, Issue. 2, pp. 46–55. Manabe. Y, Oka. T, Kaikawa Y, Takaoka. K, Uchida. Y and Asahara. M (2019),
“Japanese Word Embedding based on Multi-granular Tokenization Results,” Proceedings of the 25th Annual Meeting of the Association for Natural Language Processing. (in Japanese) Mori. H (2023), “Social Media Marketing: 12 Years of Research,” Quarterly Journal of Marketing, Vol. 43, Issue. 2, pp. 70–77. (in Japanese) Morooka. R, Otake. K and Namatame. T (2023), “Visualization of Community
Structure Using Follower Data on Twitter and Comparison Between Stores,” Lecture Notes in Computer Science, Social Computing and Social Media, HCII 2023, Vol. 14025, pp. 418–428, Springer. Newman. M. E. J and Girvan. M (2004), “Finding and evaluating community structure in networks,” Physical Review E, 69, 026113, pp. 1–15. Page. L, Brin. S, Motwani, R and Winograd, T. (1999), “The PageRank Citation
Ranking: Bringing Order to the Web,” Technical Report, Stanford Digital Library Technologies Project, SIDL-WP-1999-0120. Traag. V. A, Waltman, L and van Eck, N. J. (2019), “From Louvain to Leiden: guaranteeing well-connected communities,” Scientific Reports 9, 523, 25 pages. Yamamoto. H, Sakaji. H, Matsushima. H, Yamashita. Y, Osawa. K, Izumi. K and
Shimada. T (2019), “Prediction of Crypto-Asset Price using Influencer Tweets,” JSAI Technical Report, Type 2 SIG, Vol. 2019, Issue FIN-022, pp. 25–30. Zhang. Y, Murayama. R, Okubo. R and Yamasaki. T (2023), “Recommending Social Media Influencer with High Sales using Followers’ Information,” The 37th Annual Conference of the Japanese Society for Artificial Intelligence, 3Xin4-18.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional