Melissa Hunsicker Walburn, Angela Walters, Stacey Smith (2023). Comparative Evaluation of Social Media Data Analytics Tools in Event Management: Usability and Effectiveness. In: Human-Centered Design and User Experience. AHFE Open Access, vol. 114, pp. 544–552. https://doi.org/10.54941/ahfe1004274
The advent of social media analytics tools has brought a transformative shift to event management, enabling more dynamic, effective, and personalized event experiences. However, navigating through the sea of available options to select the most suitable tool can be challenging. This study aims to demystify this selection process by comparing the usability and effectiveness of four popular social media analytics tools for event management. The evaluation procedure involved two key steps. Initially, Nielsen's ten usability heuristics were applied to assess the usability of each tool. Subsequently, each tool's effectiveness was examined based on seven Measures of Effectiveness (MOEs) for Event Management (EVMAN), identified through an extensive literature review. Evaluation results exhibited strong consistency, which underscores the validity of our methodological approach. Our findings provide valuable insights by highlighting the strengths and weaknesses of each tool, thus aiding event managers in their selection process and offering educators a framework for classroom adoption. Furthermore, our study contributes to the literature by introducing a distinctive evaluation framework that integrates usability heuristics and event management-specific measures of effectiveness. By bridging this gap, our research paves the way for more knowledgeable, effective, and streamlined adoption of social media data analytics tools in event management.
The era of widespread digitization introduced an array of sophisticated technological tools designed to optimize and enhance various sectors, includ- ing the dynamic field of event management. One key area of innovation is the advent of social media data analytics tools, which are revolutionizing the ways in which event management professionals monitor, measure, and amplify their operations. Despite the increasing prevalence of such tools, the question of their usability and effectiveness remains an essential considera- tion. Usability encompasses the ease with which users can interact with a tool or system, while effectiveness refers to the tool’s ability to fulfill its intended function, achieve desired outcomes, and meet user needs.
Comparative Evaluation of Social Media Data Analytics Tools in Event Management 545
Understanding the usability and effectiveness of these tools is crucial because it significantly impacts their adoption, use, and ultimately, their potential benefits. Usability deficits can lead to user frustration, reduced pro- ductivity, and even abandonment of the tool. On the other hand, tools with high usability and effectiveness can enhance workflow, improve decision- making, and boost the overall success of event management processes. While there are well-established heuristics for evaluating usability (Nielsen, 1990), there has been limited research on methods for evaluating the usability and effectiveness of these tools, specifically within the field of event management.
RELATED WORK
During the first decade of the 21st century, multiple social media platforms started to gain significant user bases which led businesses and researchers alike to recognize the wealth of data these platforms could provide and over time, the value of the data compiled. Early in the evolution of social media analytics, outcomes were often dependent on manual processes, with businesses and marketers monitoring posts and hashtags and manually track- ing mentions and sentiments. However, as the volume of social media data increased this became impractical, if not unacceptable, in terms of meeting efficiency expectations in the market.
This precipitated the development and adoption of automated social media data analytics tools during the early 21st century. Drawing from the insights of recent literature reviews conducted by Rahman & Reza (2022) and Zachlod et al. (2022), a broad spectrum of sectors are turning to the exploration of social media data analytics. These diverse fields range from marketing and crisis management to public health and customer relation- ship management, amongst others. Models of evaluation are expanding and evolving with the collection of data from real estate, large, complex organi- zations, and beauty trade fairs (Zhang et al., 2022). Despite this widespread interest and engagement, these reviews uniformly recognize social media data analytics as a nascent field of study.
Measures of Effectiveness in Event Management Research within the field of Event Management (EVMAN) provides key insights into the various ways social media data analytics tools can be effectively utilized by event managers. First, these tools assist in audience seg- mentation, enabling event managers to personalize their marketing and com- munication efforts according to demographic characteristics and user behav- ior patterns (Kaplan & Haenlein, 2010). Moreover, these platforms track engagement metrics such as likes, shares, comments, and mentions, offer- ing real-time insights into audience participation and engagement (Fan & Gordon, 2014). Sentiment analysis, another prominent feature of these tools, interprets the emotional undertone of social media posts, thereby providing a snapshot of audience sentiment towards the event (Stieglitz et al., 2018).
Social media data analytics tools can also aid in identifying key influencers in the target audience, empowering event managers to amplify their promo- tional activities and enhance their reach strategically (Freberg et al., 2011). All in real -time, these tools enable timely insight into emerging trends or
issues during the event (Ghiassi et al., 2013). Post-event analysis conducted with these tools provides comprehensive data about audience engagement and feedback, offering value to continuous improvement processes for future planning (Zeng et al., 2010). Lastly, some advanced social media analytics tools can conduct predictive analysis, using historical data to project future trends for attendance or engagement, thereby aiding more effective event planning strategy (Kietzmann et al., 2011).
Building on these identified applications of social media data analytics tools in event management, we proceed to elucidate the parameters for their effectiveness. The term “measures of effectiveness” (MOEs) is not new and has seen widespread use across a variety of academic disciplines, albeit tai- lored to fit each unique context. The concept is rooted deeply in the discipline of systems engineering, and has been adopted as a quantitative method to assess the degree to which a system or process achieves its intended pur- pose or objectives (International Council on Systems Engineering [INCOSE], 2015). In the traditional systems engineering context, MOEs are often asso- ciated with the key performance parameters (KPPs) of a system, which are those most critical attributes or characteristics that articulate a system’s over- all performance as identified by stakeholders (INCOSE, 2015). Consequently, MOEs offer a methodical way to measure these KPPs, ensuring the system is fulfilling its defined objectives.
This study extrapolates this concept from systems engineering and applies it to the field of event management in the context of social media analytics tools. Here, the “system” under study is the social media analytics tool, and the MOEs are a suite of measures that aid in assessing the effectiveness of these tools in meeting the predefined objectives of event management such as audience engagement, real-time feedback, trend identification, and others. The use of MOEs in this research offers a structured approach to compare and assess the effectiveness of different social media analytics tools, ground- ing the evaluation in a well-established systems engineering principle, and thus enhancing the validity of our methodology.
Upon a comprehensive synthesis of the relevant literature, seven pivotal Measures of Effectiveness (MOEs) have been discerned within the scope of EVMA. Table 1 succinctly encapsulates these MOEs along with their respective definitions.
Table 1. Measures of effectiveness (MOE) for event management (EVMA).
Table available in the original PDF paper.
Measure of Effectiveness Definition MOE1 Audience Engagement Assessment of quantifiable user interactions and participation with an event’s social media content MOE2 Real-Time Feedback
The capture and analysis of instantaneous feedback during a event MOE3 Trend Identification
The identification of popular trends and topics emanating from the target audience’s social media activities during an event MOE4 Attendee Behavior Analysis
Systematic evaluation of the actions and interactions of attendees within the event’s social media sphere MOE5 Event Promotion
The strategic utilization of social media channels to publicize an event and stimulate anticipation MOE6 Participation Interaction
The extent to which attendees actively engage with the event through social media channels
Comparative Evaluation of Social Media Data Analytics Tools in Event Management 547
STUDY The present study utilized a multi-method approach to evaluate the usabil- ity and effectiveness of four distinct social media data analytics tools in the context of event management. Central to the study’s design was the applica- tion of Nielsen’s heuristics, a set of established usability principles, and an innovative tool developed specifically for this research based on an extensive review of the literature in this field.
In the first stage of the study, three expert reviewers conducted a heuristic evaluation of the four selected social media data analytics tools. Following Nielsen’s principles, the reviewers independently assessed each tool, con- sidering aspects such as the system’s visibility, user control and freedom, consistency with standards, and error prevention capabilities, among others. This heuristic evaluation offered insights into the overall usability of each tool, identifying potential areas of strength and weakness.
Parallel to the heuristic evaluations, the study leveraged a newly-developed assessment tool to evaluate the effectiveness of the social media data ana- lytics tools. Derived from prior research, this tool was designed to assess effectiveness along seven dimensions (see Table 1). Each dimension was oper- ationalized based on evidence-based measures identified in the literature. The three expert reviewers utilized this tool to assess each of the four social media data analytics tools’ effectiveness in event management context.
In essence, this research is a synergetic endeavor of heuristic evaluation and specific effectiveness assessment, designed to furnish a holistic evalu- ation of the selected social media data analytics tools. By comparing the standard deviations of the results obtained from the three evaluators, the research aims to explore the consistency of heuristic evaluations across dif- ferent contexts. The findings derived from this study serve not just as an exploration of the current capabilities and limitations of the tools, but also set the stage for future research applications and opportunities to signifi- cantly impact the continually evolving, and competitive, landscape of event management.
RESULTS AND DISCUSSION
Regarding the overall usability of each tool, for Tool #1, the averages ranged from 4.00 (H7) to 6.67 (H1 and H6), indicating that while this tool excelled in terms of visibility of system status (H1) and recognition rather than recall (H6), it fell short on flexibility and efficiency of use (H7). For Tool #2, the averages were relatively more balanced, ranging from 4.00 (H3) to 6.67 (H6). It’s noteworthy that recognition rather than recall (H6) was again a strong point, while match between system and the real world (H3) was the weak- est.Tool #3 saw averages from 4.00 (H3) to 6.33 (H6). Similar to Tool #2, it had its highest score in recognition rather than recall (H6) and the lowest in match between system and the real world (H3). Finally, Tool #4 had aver- ages ranging from 2.33 (H3) to 4.67 (H6), which were lower than the other tools across the board, implying this tool might have more significant usabil- ity challenges. In sum, across all tools, recognition rather than recall (H6) consistently emerged as a strength, while match between system and the real
world (H3) and flexibility and efficiency of use (H7) were common areas of improvement.
When analyzing the results of each tool, we found a remarkable level of consensus among our expert evaluators. This was evidenced by the low stan- dard deviations in the heuristic scores across all tools. For seven out of ten heuristics, the standard deviation was zero, demonstrating a complete agree- ment among the evaluators. For the remaining three heuristics, the standard deviation was 0.57, reflecting a slight variability in scoring. However, this variability is minimal and does not detract from the overall strong consensus achieved in the heuristic evaluations. The high degree of agreement among our evaluators reinforces the robustness of heuristic evaluation methods and the reliability of our findings.
Similarly, when evaluating effectiveness utilizing the seven Measures of Effectiveness (MOEs) predicated on extensive literature review, we observed substantial agreement in the evaluations, which can be seen in Figure 1. Five out of seven measures yielded a standard deviation of zero, indicating complete unanimity among the evaluators. The remaining two measures pre- sented a standard deviation of 0.57 and 1.15 respectively, suggesting some degree of variability. However, considering the overall context, this marginal disparity in evaluations does not detract from the prevalent consensus among the experts.
Figure 1: Comparative evaluations of four social media data analytic tools. Results show low standard deviations and high consensus across both data sets.
Comparative Evaluation of Social Media Data Analytics Tools in Event Management 549
Given the level of agreement between evaluators among both data sets, findings suggest tool #1 would be effective in audience engagement, real-time feedback, and event promotion (Figure 2). However, it may be less capable in analyzing social media activity and engagement following an event. Tool #2 and Tool #3 had strong evaluations across all areas, especially in real-time feedback, attendee behavior analysis, event promotion, and post-event anal- ysis. When considering both usability and effectiveness, Tool #2 has a higher overall score. Tool #4, on the other hand, had lower scores across all measures of effectiveness. With scores of 5 for trend identification, attendee behavior analysis, event promotion, post-event analysis, and participation interaction, it suggests that this tool may have limitations in these areas compared to the other tools.
Figure 2: Comparative evaluations results by tool, indicating correlation between usability and effectiveness.
The results show a correlation between the scores from Nielsen’s usability heuristics and these measures of effectiveness. Tools #2 and #3 that scored high in usability also demonstrated high effectiveness across all areas, while Tool #4 had lower scores in both evaluations.
The synergies observed in the evaluations of both usability, based on Nielsen’s heuristics, and effectiveness, based on the seven MOEs, are quite striking. Despite evaluating distinct aspects of the tools and despite the eval- uators being different individuals, the level of consensus across both usability and effectiveness evaluations is highly significant. This consistency reinforces the validity of our evaluation methodologies and underscores their ability to yield reliable and robust assessments, irrespective of the evaluator or the dimension being evaluated.
These findings have relevant implications for research related to evaluation methodologies employed in studies concerning social media data analytics tools, as well as the potential to influence the competitive landscape of event management. Our findings suggest that these methodologies, whether for usability or effectiveness, can be applied reliably across different evaluators and tools. This consistency, in turn, strengthens the credibility of research outcomes and paves the way for the development of more rigorous, stan- dardized methods for evaluating social media analytics tools. Furthermore, the substantial overlap between usability and effectiveness evaluations pro- vides a new perspective for understanding these two dimensions, suggesting
that they may be interconnected and mutually reinforcing. This opens up new avenues for research exploring the relationship between usability and effectiveness in the context of social media analytics tools.
Despite the significant findings and contributions of this research, there are several limitations that warrant discussion. The primary limitation of this study stems from the inability to evaluate the effectiveness of the tools in providing data for predictive analysis. Predictive analytics is a crucial aspect of social media data analytics, enabling event managers to anticipate future trends, audience behaviors, and engagement patterns. However, our evalua- tion methodologies were not designed to adequately assess this capability of the tools, which could limit the comprehensiveness of our findings.
Additionally, the study only considered four social media analytics tools for event management. There are numerous such tools available in the mar- ket, and our findings might not generalize to other tools not included in the study. This limitation may restrict the broader applicability of our results.
Finally, the rapidly evolving nature of social media platforms and analytics tools introduces a temporal limitation to our research. Given the constant technological advancements and changes in user behavior, the usability and effectiveness of these tools may vary over time. Thus, findings of this study represent a snapshot in time and might not be entirely applicable in the long term.
CONCLUSION AND FUTURE WORK Our comparative evaluation of four social media analytics tools, based on Nielsen’s heuristics and the framework proposed, has produced critical insights into the usability and effectiveness of these tools in the context of event management. While the study demonstrated a strong consensus among expert evaluators, it also highlighted the interconnected nature of usability and effectiveness as an area of study opportunity. However, the results must be interpreted in light of certain limitations, including the inability to evaluate predictive capabilities and potential evaluator bias.
The findings of this study lay the groundwork for a range of future research opportunities. Firstly, future studies could build on this research by includ- ing predictive analytics as an additional measure of effectiveness. This will enable a more comprehensive evaluation of social media analytics tools and contribute to a richer understanding of their capabilities.
Secondly, to address the limitation of potential evaluator bias, future studies could employ field research and usability testing with a larger, more diverse user base. This would provide a more representative view of the usability and effectiveness of these tools and establish a more holistic understanding of the user experience.
Establishing benchmarks for future analysis, based on the results of this study, would provide a standardized framework for evaluating social media analytics tools. This could pave the way for more rigorous and comparative research in this area.
Finally, given the rapid evolution of social media platforms and analyt- ics tools, continuous evaluation and research are imperative. Future research could focus on longitudinal studies to understand how the usability and effec- tiveness of these tools change over time. With the advent of new technologies
Comparative Evaluation of Social Media Data Analytics Tools in Event Management 551
such as artificial intelligence and machine learning in data analytics, their impact on social media analytics tools and their usability and effectiveness would be a promising area for future exploration.
In conclusion, this study marks a significant step in understanding the value of the usability and effectiveness of social media data analytics tools to event management. The study also uncovers new avenues for research that can extend our knowledge in this evolving field. By acknowledging the limi- tations of the present study and leveraging them as opportunities for future research, we can continue to enhance the quality and depth of knowledge in the field of social media analytics for event management.
REFERENCES
Brodie, R. J., Ilic, A., Juric, B., & Hollebeek, L. (2013). Consumer engagement in a virtual brand community: An exploratory analysis. Journal of Business Research, 66(1), 105–114. https://doi.org/10.1016/j.jbusres.2011.07.029 Fan, W., & Gordon, M. D. (2014). The power of social media analytics. Communi-
cations of the ACM, 57(6), 74–81. https://doi.org/10.1145/2602574 Freberg, K., Graham, K., McGaughey, K., & Freberg, L. A. (2011). Who are the social media influencers? A study of public perceptions of personality. Public Relations Review, 37(1), 90–92. https://doi.org/10.1016/j.pubrev.2010.11.001 Ghiassi, M., Skinner, J., & Zimbra, D. (2013). Twitter brand sentiment analysis:
A hybrid system using n-gram analysis and dynamic artificial neural network. Expert Systems with Applications, 40(16), 6266–6282. https://doi.org/10.1016/j. eswa.2013.05.057 International Council on Systems Engineering (INCOSE). (2015). Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities. Wiley. Kaplan, A. M., & Haenlein, M. (2010). Users of the world, unite! The challenges
and opportunities of Social Media. Business Horizons, 53(1), 59–68. https://doi. org/10.1016/j.bushor.2009.09.003 Kietzmann, J. H., Hermkens, K., McCarthy, I. P., & Silvestre, B. S. (2011). Social media? Get serious! Understanding the functional building blocks of social media. Business Horizons, 54(3), 241–251. https://doi.org/10.1016/j.bushor.2011.01. 005 Malthouse, E. C., Calder, B. J., Kim, S. J., & Vandenbosch, M. (2016). Evidence that
user-generated content that produces engagement increases purchase behaviours. Journal of Marketing Management, 32(5–6), 427–444. https://doi.org/10.1080/ 0267257X.2016.1148066 Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 249–256. https://doi.org/10.1145/97243.97281 Rahman, M. S., & Reza, H. (2022). A Systematic Review Towards Big Data Analytics
in Social Media. Big Data Mining and Analytics, 5(3), 228–244. https://doi.org/ 10.26599/BDMA.2022.9020009 Stieglitz, S., Mirbabaie, M., Fromm, J., & Melzer, S. (2018). The Adoption of social media analytics for crisis management – Challenges and Opportunities. Research Papers. https://aisel.aisnet.org/ecis2018_rp/4 Zachlod, C., Samuel, O., Ochsner, A., & Werthmüller, S. (2022). Analytics of social
media data – State of characteristics and application. Journal of Business Research, 144, 1064–1076. https://doi.org/10.1016/j.jbusres.2022.02.016
Zeng, D., Chen, H., Lusch, R., & Li, S.-H. (2010). Social Media Analytics and Intelligence. IEEE Intelligent Systems, 25(6), 13–16. https://doi.org/10.1109/MIS.2010. 151 Zhang, H., Zang, Z., Zhu, H., Uddin, M. I., & Amin, M. A. (2022). Big data-assisted
social media analytics for business model for business decision making system competitive analysis. Information Processing & Management, 59(1), 102762. https://doi.org/10.1016/j.ipm.2021.102762
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human-Centered Design and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-90-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional