Lorelie Grepo, Belinda Esther Tan, Jose Antonio Clemente (2025). Understanding university instructors’ cumulative user experience with a Learning Management System: A semester-long mixed methods study. In: Usability and User Experience. AHFE Open Access, vol. 194, pp. 52–62. https://doi.org/10.54941/ahfe1006667
1 Department of Industrial Engineering and Operations Research, College of Engineering, University of the Philippines Diliman, Quezon City, Philippines ↗
2 Department of Psychology, College of Social Sciences and Philosophy, University of the Philippines Diliman, Quezon City, Philippines ↗
Educational technologies are rapidly advancing, with LMS being used by most academic institutions to facilitate and support learning. The COVID pandemic catalysed use of technologies to support non-traditional learning setup in the Philippines. While LMS use is widespread and its usability is highly explored, research on long-term use remains limited, particularly regarding how user behavior and system interactions evolve over time, which is crucial to understand the context of use and identification of areas for improvement to better support users. In addition, only a select few examine this from teachers’ viewpoints. This study explores the university instructors’ experiences over a semester with a Moodle-based LMS using a mixed methods approach. The results presented here are part of a broader study, covering the “during semester” experiences with the LMS. Ten (10) instructors participated in the study, accomplishing 3 weekly surveys and a diary every month. The weekly surveys measured their usage and ratings on the communication, course management and system usability, as well as their experiences. The monthly diary enabled participants to reflect on the challenges they faced, opportunities for improvement and the tools they used to achieve their goals. Combining these methods helped users track their UX progression over time. This study employed a post-phenomenological approach, which examines the dynamic human-technology-world relationship, where technological mediation co-shapes human subjectivity and the objectivity of their world, viewing technology as inviting and inhibiting human action and practices in any given situation. This semester-long study found that three components, Course Management, Communication and System Usability, have yielded overall positive results, both quantitatively through the weekly ratings and qualitatively, as reported in weekly and monthly diaries. Instructors perceived the LMS to invite actions related to teaching and learning rather than inhibiting them. Results show that while the UX scores for all three components have been positive, the qualitative reports were not always consistent with these positive ratings, specifically for System Usability, highlighting the importance of the mixed approach to understanding user experiences. The top three recommendations for LMS improvement included training and support, communication tools, and the user interface.
Understanding University Instructors’ Cumulative User Experience 53
INTRODUCTION
The COVID-19 pandemic posed significant instructional challenges for Filipino faculty members, most of whom were primarily trained in traditional face-to-face teaching and had minimal experience with online education before the crisis (Arinto, 2016; Moralista & Oducado, 2020). Learning Management System (LMS) platforms generally fall into three categories: Open Source, Commercial, and In-House Developed (Hock et al., 2015). Open Source LMSs are widely preferred in educational institutions due to lower cost, support infrastructure, and potential for integration with other e-learning systems (Lopes, 2014).
This study focuses on a Moodle-based LMS (Modular Object-Oriented Dynamic Learning Environment), an open-source LMS (Poulova et al., 2015, p. 1303) and one of the most widely used e-learning platforms in higher education (Kuran et al., 2017; Teo et al., 2019). An LMS UX study rated usability positively, particularly in student management tasks such as file submission, time management, and progress tracking, but also suggested improving user-friendliness, especially in the informativeness of course content (Maslov et al., 2021). They also highlighted the crucial role of instructors in shaping students’ perceptions and engagement in Moodle, emphasizing the need for both teachers and administrators to actively manage UX of LMS (Maslov et al., 2021).
Building on this, this study examines university instructors’ experiences with a Moodle-based LMS over the course of a semester, employing a mixed-methods approach to gain deeper insights into its practical use and long-term implications. Despite the widespread adoption of LMS, research on its long-term use remains limited, particularly in understanding how user behaviour and system interactions evolve over time. Despite studies exploring LMS usability, few have examined its sustained impact from the perspective of educators. Addressing these gaps is crucial for identifying areas of improvement and enhancing user support.
METHODS
This study employed a mixed methods approach, collecting quantitative and qualitative data through online surveys and diaries. This study also adopted a post-phenomenological approach, which explores the dynamic relationship between humans, technology, and the world (Hauser et al., 2018). It examines how technological mediation co-shapes human subjectivity and perceptions of reality, viewing technology as both “inviting”and “inhibiting”actions and practices in various situations. This paper reports on the UX part of a wider thesis study.
Ten instructors who taught during the First Semester of academic year 2022–2023 participated in the study. They were recruited from four academic clusters—Arts and Letters, Management and Economics, Science and Technology, and Social Sciences and Law—to ensure representation across disciplines.
The participants completed three weekly surveys and a diary entry online every month for the entire semester. The weekly surveys measured usage
(whether they used the LMS that week or not), ratings on communication, course management, system usability, and overall experiences, using a scale from −3 (most negative) to +3 (most positive). The three key constructs— system usability, course management, and communication—were based on the Faculty LMS Functionality Instrument (FLFI; Liu et al., 2019). The FLFI (Liu et al., 2019) integrated insights from Rogers’ (2003) Diffusion of Innovations Theory on LMS adoption stages, Malikowski et al.’s (2007) model on key LMS functions, and their own research, which addressed system usability—an aspect not fully covered by the other frameworks. Communication refers to LMS tools that facilitate interaction between instructors and students, such as announcements, chat, feedback, and forums. Course Management focuses on LMS functions related to content organization, course layout, user enrollment, assignments, and grading. System Usability evaluates LMS attributes, including user-friendliness, customization, browser compatibility, and access to technical support. The weekly surveys also adopted the seven-point scale from Feng & Wei’s (2019) study, which ranged from −3 (very negative) to +3 (very positive).
The monthly diary asked participants to reflect on challenges, opportunities for improvement, and the tools they used to achieve their goals. To address high attrition rate, discontinued or withdrawn participants were asked for consent to retain the data collected from them and be included in data analysis.
The weekly ratings were analyzed using the mode since scores were categorical and the distributions were positively skewed. UX Curves were generated using Google Sheets and compiled in MS PowerPoint. Framework analysis was used to analyze the qualitative data using NVivo 12 and Google Sheets, involving theme identification, indexing, charting, and pattern interpretation (Goldsmith, 2021).
RESULTS AND DISCUSSION
The study included ten participants (n = 10), the majority of whom were male (nine out of ten), with ages ranging from 20 to 39 years. Six participants belonged to the Science and Technology cluster, and most taught undergraduate courses exclusively. Table 1 summarizes their demographic characteristics.
Table 1: Demographic characteristics of instructors (n = 10).
Table available in the original PDF paper.
Category Subcategory Frequency Sex Male 9 Female 1 Age 20-29 7 30-39 3 Academic Cluster Social Sciences and Law 1 Arts and Letters 1 Science and Technology 6 Management and Economics 2
Understanding University Instructors’ Cumulative User Experience 55
There were a total of 14 weekly surveys and five diary entries for the study. P1 and P2 completed all weekly surveys and monthly diaries, while P5 had no submissions. Participation varied across time points due to recruitment challenges, leading to distinct participation patterns. Weeks differed for each instructor, primarily due to variations in use, as illustrated in Figure 1. Notable cases include P3, who submitted multiple ratings for week 3 and completed the semester early. For P3, the graph displays the average of those week 3 ratings. P6 discontinued in the middle of the semester, and P8 stopped towards the end but still participated in the debriefing. P9’s participation was delayed due to a communication issue.
Figure 1: UX Curves for instructors’ LMS use.
Understanding University Instructors’ Cumulative User Experience 57
Communication, Course Management, and System Usability—yielded overall positive results, both quantitatively through weekly ratings and qualitatively in weekly and monthly diaries. Figure 1 shows the participant ratings for the three components over the semester. The differences in their perceived experiences are seen in the curves shown. Average scores were used to represent non-use or missing data. P2 and P3 for example, relatively had the same ratings over time while P1, P4, P7 and P9 showed a fluctuating trend for their ratings.
Quantitative results show that instructors perceived the LMS as inviting actions related to teaching and learning rather than inhibiting them. UX scores for Communication range from slightly negative (–1) to very positive (+3), with lower ratings (0 or –1) appearing at the beginning and middle of the semester, which can be explained by reduced communication needs during reading breaks and holidays. However, the most frequently reported satisfaction level is +2 (occurring 34 times), indicating a predominantly positive perception. Course Management scores varied from very negative (–3) to very positive (+3), with the majority being positive, particularly regarding course management activities, course pages, and content. Similar to Communication, the most frequently reported level of satisfaction is +2 (occurring 34 times). System Usability scores ranged from very negative (–3) to very positive (+3), reflecting varied experiences related to the LMS’s user-friendliness and customization features. Despite this variation, the most frequently reported satisfaction level remains +2 (occurring 33 times), reflecting an overall positive perception.
For qualitative results, a reporting system from Midgley et al. (2015) was adapted to indicate the frequency of experiences in the diaries: Most refers to findings from 32 or more reports, Many from 22 to 31 reports, Some from 12 to 21 reports, and Few from fewer than 12 reports. Most reports highlighted the LMS’s strong support for course management, particularly in consolidating and organizing resources in a unified space. Instructors appreciated how the LMS simplifies access to class materials, reducing the need for multiple emails. Qualitative findings also reinforce a positive perception of the LMS for communication, with many instructors using it for announcements to ensure timely updates. However, while UX scores for System Usability are generally positive, qualitative reports reveal challenges, particularly in assessment and grading, with many instances of inhibited actions due to platform glitches. This inconsistency between UX scores and qualitative findings is particularly evident in System Usability scores, as shown in Table 2.
Table 2: Summary of results.
Table available in the original PDF paper.
Components Quantitative (Mode) Qualitative Results Comparison Positive (+2) (34 times) Most Invite (32 and up Consistent Communication Positive (+2) (34 times) Many Invite (22-31 reports) Consistent Course Management
reports) System Usability Positive (+2) (33 times) Many Inhibit (22-31
Inconsistent
reports)
Key features that invited action included course organization and correspondence, assessment and grading, and access to class resources. The platform provided a centralized space for class administration, individual grade distribution, and communication tools, streamlining course management. Easy access to resources enabled instructors to upload, organize, and distribute materials efficiently. Time-saving features, such as immediate feedback and grading options, simplified assessments, while scheduling flexibility supported asynchronous learning.
However, usability issues within the LMS often inhibited these actions, as instructors found the interface complex and unintuitive, leading to frustration and inefficiency. The abundance of prompts, buttons, and windows increased task complexity, making navigation between students’ work time-consuming. This qualitative assessment contrasts with other studies (Demir et al., 2022; Olugbade et al., 2023), where educators perceived Moodle as user-friendly and easy to use. As seen in this study, such differences may reflect how quantitative tools yield different results from qualitative assessments. These findings align more with students’ perceptions of Moodle, where weak elements such as communication (including group communication), user- friendliness, navigation difficulty, course enrollment, layout/UI, discussion forums, and overall usability issues have been highlighted (Maslov et al., 2021). Additionally, a reported system crash caused inconvenience for both instructors and students. Limited customization options further restricted instructors’ ability to tailor the platform to their needs. For example, face- to-face tests were recorded as assignments rather than exams, potentially leading to underutilization of available features due to the system’s lack of intuitiveness. This finding contrasts with the common perception that open-source LMSs are highly flexible. However, it supports Olugbade et al. (2023), suggesting that universities can identify areas for improvement and customization within Moodle based on their unique institutional context.
The top three areas for LMS improvement recommended by instructors were training and support, communication tools, and the user interface. Instructors highlighted the need for training on advanced features such as grouping students, merging LMS classes, anonymous forums, activity logs, gamification, and advanced scripts. This aligns with recent research showing that Moodle’s user experience (UX) depends on its design and maintenance— when developed by experts and properly managed, it is well-received, but poor design or inadequate hosting can negatively impact usability (Maslov et al., 2021). The institution managing the LMS was also perceived to prioritize acquiring new users over supporting existing ones, highlighting the need for ongoing training not only for beginners but also to help experienced instructors fully utilize the platform’s capabilities For communication tools, instructors suggested features like “Read Receipts” or emoticons to gauge message visibility and better notification management to tailor alerts for both instructors and students. Some also recommended reminders for students to check their email notification settings to ensure they receive important updates. Since many instructors and
Understanding University Instructors’ Cumulative User Experience 59
students currently rely on external communication channels—either through personal communication or emails, as noted by Maslov et al. (2021)— enhancing built-in LMS tools could improve engagement and streamline course interactions.
For the user interface (UI), many instructors recommended enhancing intuitiveness and user-friendliness, particularly for beginners and Gen Z students, by incorporating modern design elements to prevent monotony and improve engagement. Some suggested a clearer help section to address navigation issues, while others expressed concerns about excessive customizations and buttons that could overwhelm users. Additionally, instructors proposed features such as applying changes across all course pages and automating tasks like grouping students based on task completion. However, as mentioned, recent studies (Demir et al., 2022; Olugbade et al., 2023) on faculty use of Moodle LMS do not fully align with these findings, suggesting the need for further mixed methods research on their instructor experience.
Collecting these recommendations is essential because instructors, as primary LMS users, have direct experience with its limitations and can offer insights that may be overlooked by developers or administrators, which is aligned with user-centric approaches (Diefenbach, 2018). Addressing their feedback ensures the LMS remains practical and efficient, ultimately improving adoption rates, increasing engagement, and leading to better learning experiences for teachers and their students alike.
CONCLUSION
This study explored university instructors’ experiences with a Moodle- based LMS over a semester using a mixed-methods approach. The results show that LMS usage varies significantly among individuals, making the study of cumulative UX complex. Due to these differences, instructors’ UX Curves had to be separated to reflect distinct usage patterns and objectives, even among those in the same field of study. By using UX Curves to visually illustrate their usage patterns and track use changes over time, along with writing monthly diaries, the study inadvertently provided instructors with an opportunity to reflect on their teaching practices during the COVID-19 pandemic. This further highlights the value of longitudinal research in capturing evolving user experiences. While participant attrition is a common challenge in longitudinal studies, it is important to emphasize the potential benefits of the study’s outcomes to encourage continued participation.
Similarly, there is no single way to define cumulative UX, as some researchers associate UX over time with concepts such as long-term user experience (Kujala et al., 2011; Luojus, 2012) and temporality (Karapanos et al., 2009). Luojus (2012) describes long-term UX as the accumulation of motivated actions (i.e., temporary UXs) and introduces the expanded UX (eUX) framework. While recent systematic mapping and literature review studies on LMS usability and UX evaluation (Júnior et al., 2022; Talib et al., 2023) have examined assessment methods and key characteristics, they have largely overlooked UX changes over time. This gap highlights the need for
further research to better understand long-term user interactions and system improvements.
Although not discussed in the previous section, post-interviews revealed that some participants encountered new experiences during the study, such as challenges related to academic integrity or temporary disability, leading to new suggestions for improvement that cross-sectional studies might be able to capture. Overall, the qualitative data collected from this study helped identify pain points, motivations, and unmet needs, particularly as instructors reflected on factors influencing their continued LMS use. Thus, revealing that beyond identifying key characteristics, future studies should also prioritize gathering user-driven suggestions to enhance LMS usability and overall experience.
ACKNOWLEDGMENT
We sincerely thank the University of the Philippines Diliman Office of the Vice Chancellor for Academic Affairs - Diliman Interactive Learning Center and the faculty members who participated in the study for their support and valuable contributions. The first author also acknowledges the University of the Philippines Diliman Office of the Chancellor, through the Office of the Vice Chancellor for Research and Development, for the Thesis and Dissertation Grant to fund this study. The first author also acknowledges the Office of the Chancellor of the University of the Philippines Diliman, through the Office of the Vice Chancellor for Research and Development, for funding support through the Thesis and Dissertation grant.
REFERENCES
Arinto, P. (2016). Issues and Challenges in Open and Distance e-Learning : Perspectives from the Philippines This paper reports part of the findings of a collective case study of ODeL course design practice. International Review of Research in Open and Distributed Learning, 17(2), 162–180. https://doi.org/ https://doi.org/10.19173/irrodl.v17i2.1913 Demir, F., Bruce-Kotey, C., & Alenezi, F. (2022). User Experience Matters: Does One
size Fit all? Evaluation of Learning Management Systems. Technology, Knowledge and Learning, 27(1), 49–67. https://doi.org/10.1007/s10758-021-09518-1 Diefenbach, S. (2018). Positive technology-A powerful partnership between positive psychology and interactive technology. A discussion of potential and challenges. Journal of Positive Psychology and Wellbeing, 2(1), 1–22. Feng, L., & Wei, W. (2019). An empirical study on user experience evaluation
and identification of critical UX issues. Sustainability (Switzerland), 11(8), 2432. https://doi.org/10.3390/su11082432 Goldsmith, L. J. (2021). Using framework analysis in applied qualitative research.
Qualitative Report, 26(6), 2061–2076. https://doi.org/10.46743/2160-3715/ 2021.5011 Hauser, S., Oogjes, D., Wakkary, R., & Verbeek, P.-P. (2018). An Annotated Portfolio on Doing Postphenomenology Through Research Products. https:// doi.org/10.1145/3196709.3196745
Understanding University Instructors’ Cumulative User Experience 61
Hock, S. Y., Omar, R., & Mahmud, M. (2015). Comparing the Usability and Users Acceptance of Open Sources Learning Management System (LMS). International Journal of Scientific and Research Publications, 5(4). Júnior, D. G. S., Hernández-Ramírez, R., & Estima, J. (2022). Systematic Mapping
of Methods Used to Evaluate the Usability and UX of Learning Management Systems. In N. Martins & D. Brandão (Eds.), Advances in Design and Digital Communication II (pp. 122–133). Springer International Publishing. https:// doi.org/10.1007/978-3-030-89735-2_11 Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J.-B. (2009). User experience over time. 729. https://doi.org/10.1145/1518701.1518814 Kujala, S., Roto, V., Väänänen-Vainio-Mattila, K., Karapanos, E., & Sinnelä, A.
(2011). UX Curve: A method for evaluating long-term user experience. Interacting with Computers, 23(5), 473–483. https://doi.org/10.1016/j.intcom.2011.06.005 Kuran, M. ¸S., Pedersen, J. M. and Elsner, R. (2017), “LMSs on blended learning courses: an experience-based observation”, International Conference on Image Processing and Communications, Springer, Cham, pp. 141–148. Liu, J. C., Brantmeier, N., Wilcox, D., Griffin, O., Calcagno-Roach, J., &
Brannon, R. (2019). Faculty Perceived Functionality of Learning Management System: Development and Validation of a Scale. In W. W. K. Ma, W. W. L. Chan, & C. M. Cheng (Eds.), Shaping the Future of Education, Communication and Technology (pp. 165–177). Educational Communications and Technology Yearbook. Springer, Singapore. https://doi.org/10.1007/978-981-13-6681-9_13 Lopes, A. P. (2014). Learning Management Systems in Higher Education. Proceedings of EDULEARN14 Conference, 5360–5365. https://doi.org/10.4018/978-1-4666- 0011-9.ch608 Luojus, S. (2012). Integrating momentary and long-term UX: A theoretical approach.
Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI 2012, 353–356. https://doi.org/10.1145/2414536.2414593 Malikowski, S. R., Malikowski, S. R., Thompson, M. E., & Theis, J. G. (2007).
A Model for Research into Course Management Systems: Bridging Technology and... Journal of Educational Computing Research, 36(2), 149–173. Maslov, I., Nikou, S., & Hansen, P. (2021). Exploring user experience of learning management system. The International Journal of Information and Learning Technology, 38(4), 344–363. https://doi.org/10.1108/IJILT-03-2021-0046 Midgley, N., Parkinson, S., Holmes, J., Stapley, E., Eatough, V., & Target, M.
(2015). Beyond a diagnosis: The experience of depression among clinically- referred adolescents. Journal of Adolescence, 44(1), 269–279. https://doi.org/ 10.1016/j.adolescence.2015.08.007 Moralista, R. B., & Oducado, R. M. F. (2020). Faculty perception toward online education in a state college in the Philippines during the coronavirus disease 19 (COVID-19) pandemic. Universal Journal of Educational Research, 8(10), 4736–4742. https://doi.org/10.13189/ujer.2020.081044 Olugbade, D., Ojo, O. A., & Tolorunleke, A. E. (2023). Challenges and Limitations
of Moodle LMS in Handling Large-Scale Projects: West-African Universities Lecturers’ Perspective. Journal of Educational Technology and Instruction, 2(2), Article 2. https://doi.org/10.70290/jeti.v2i2.71 Poulova, P., Simonova, I. and Manenova, M. (2015), “Which one, or another?
Comparative analysis of selected LMS”, Procedia-Social and Behavioural Sciences, Vol. 186, pp. 302–1308. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). The Free Press. Talib, E. A. H., Santosa, P. I., & Wibirama, S. (2023). Evaluation of Learning Management Systems Based on Usability and User Experience:
A Systematic Literature Review. 2023 International Seminar on Intelligent Technology and Its Applications (ISITIA), 691–696. https://doi.org/10.1109/ ISITIA59021.2023.10221015 Teo, T., Zhou, M., Fan, A. C. W. and Huang, F. (2019), “Factors that influence university students’ intention to use Moodle: A study in Macau”, Educational Technology Research and Development, Vol. 67, No. 3, pp. 749–766.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Usability and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-70-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional