Aysha Almazrouei, Afra Almazrouei, Haseena Alkatheeri, Mohammad Amin Kuhail (2026). Design and Evaluation of an AI Academic Advisor: Insights from Student Interactions. In: Human Interaction and Emerging Technologies (IHIET 2026). AHFE Open Access, vol. 236, pp. 11–18. https://doi.org/10.54941/ahfe1008077
Artificial intelligence (AI) is increasingly being explored to improve the efficiency and accessibility of academic advising in higher education. Traditional advising systems often face challenges, including limited advisor availability and increasing administrative workload, which can delay student support. This study presents and evaluates a conversational AI academic advisor built using a Retrieval-Augmented Generation (RAG) architecture that combines a large language model (LLM) with a curated knowledge base. The system allows students to ask advising-related questions and receive responses grounded in official documents. A pilot study with 20 university students was conducted in which participants interacted with the system to explore tasks for advising tasks before completing a usability survey. Results indicate generally positive perceptions of the system’s convenience and accessibility, with some participants reporting that the system was easy to use and helpful for obtaining information, and some indicating that responses were clear and understandable. Students highlighted the speed and immediacy of responses as key advantages. However, some participants reported response delays and limitations in handling complex questions. Overall, the findings suggest that AI advisors can effectively complement traditional advising services for routine inquiries.
Academic advising plays an essential role in supporting student success in higher education. Advisors guide students through course selection, degree requirements, institutional policies, and long-term academic planning. Effective advising has been related to improved academic performance, higher retention rates, and significant student engagement (Dawood, 2024). However, universities increasingly face challenges in ensuring timely advising support due to rising student enrolment and limited availability of advisors.
Artificial intelligence (AI) has emerged as a potential solution to these challenges. Conversational agents, commonly referred to as chatbots or virtual assistants, can provide automated responses to student inquiries and deliver Received May 11, 2026; Revised June 29, 2026; Accepted July 19, 2026; Available online August 25, 2026
academic guidance through natural-language interactions. Developments in natural language processing (NLP) and machine learning have allowed AI systems to understand student questions and generate relevant responses related to courses, program requirements, and university policies (Abdelhamid et al., 2025).
Recent research highlights the growing use of AI-powered academic advising systems in higher education institutions. These systems can provide immediate access to information and reduce faculty administrative workload. Moreover, they can improve the accessibility and efficiency of advising services, allowing students to receive answers at any time without waiting for appointments (Assayed et al., 2024).
Early implementations of conversational advising systems relied primarily on rule-based architectures and predefined response structures. While these systems demonstrated the feasibility of automated advising support, their ability to handle complex or open-ended student questions was limited (Kuhail et al., 2023).
More recently, the emergence of large language models (LLMs) has significantly expanded the capabilities of conversational AI. LLM-based systems can process complex natural language queries and generate more flexible, context-aware responses than earlier chatbot technologies, increasing interest in their use as virtual tutors, teaching assistants, and academic advisors. However, their use in educational contexts raises concerns about reliability and transparency, as LLMs may generate plausible but inaccurate responses due to their probabilistic language generation processes (Urbančič, 2024), potentially undermining trust in academic decision-making contexts. To address these limitations, recent research has explored retrieval- augmented generation (RAG) architectures that combine LLMs with document retrieval mechanisms (Varma et al., 2025). In RAG systems, the AI model retrieves relevant information from curated knowledge bases, such as scheduling information, faculty bios, or policy manuals, and uses these sources as grounding context for generating responses. This approach can improve the factual accuracy and domain alignment of AI-generated answers by ensuring that responses are informed by authoritative sources rather than relying solely on the model’s pre-trained knowledge.
While RAG-based approaches offer promising solutions to the reliability challenges associated with generative AI, the broader adoption of AI-based advising technologies remains challenging. Concerns related to trust, privacy, and student acceptance must be addressed before these systems can be widely implemented. Research indicates that students’ perceptions of ease of use and social influence play an important role in the acceptance of advising chatbots in educational institutions (Bilquise et al., 2023). In addition, while AI systems can automate routine advising tasks, complex academic decisions often require human judgment and contextual understanding (Luong & Lunong, 2025).
Although previous studies have primarily focused on the technical development of AI advising systems, limited research has examined how students perceive and interact with these technologies in practice. Understanding student perceptions is therefore essential for designing effective AI advising tools that support and complement existing advising services (Lytridis & Tsinakos, 2025).
Design and Evaluation of an AI Academic Advisor: Insights From Student Interactions 13 Therefore, this study addresses the following research question: How do students perceive AI-based academic advising systems in terms of usability, usefulness, and motivation? In doing so, this research contributes to the literature on human–AI interaction in education in several ways. First, it examines students’ perceptions of AI-supported academic advising systems. Second, it examines the factors influencing students’ acceptance and usability of AI advising tools. Finally, it identifies key design considerations for the responsible implementation of AI-based advising systems in higher education institutions.
RELATED WORK
AI and Chatbots in Higher Education AI technologies are increasingly being integrated into educational environments to enhance teaching, learning, and Official processes. Among the most widely used AI applications in education is chatbot technology. Chatbots are Interactive software agents designed to recreate human interaction over text or voice systems (Lytridis & Tsinakos, 2025).
Chatbots can provide immediate responses to student questions, assist with course information, and guide students through university-related procedures. They’re particularly useful for answering frequently asked questions about course schedules, assignment deadlines, and graduation requirements. Since most students usually use digital platforms and mobile devices, chatbot systems can be easily integrated into university websites or learning management systems (Okonkwo & Ade-Ibijola, 2021).
Research suggests that AI chatbots can improve student engagement and reduce faculty members’ official workload by automating routine inquiries. By handling common advising questions, these systems allow academic advisors to focus on more complex student concerns (Rapp et al., 2021).
Acceptance of Academic Advising Chatbots While AI advising systems have potential advantages, student acceptance remains a key factor influencing their effectiveness. Studies investigating the adoption of advising chatbots found that perceived ease of use and social influence significantly affect students’ willingness to interact with these systems (Bilquise et al., 2023).
In surveys of university students, functional characteristics such as system usability and accessibility have been found to significantly influence behavioral intention to use advising chatbots. On the other hand, perceived usefulness and trust did not consistently affect use decisions. These results suggest that the design and user experience of AI advising systems may hold greater significance than their technical functions (Samy et al., 2026).
Human-Chatbot Interaction Another important area of research focuses on how users interact with conversational agents. Studies analysing human–chatbot interaction emphasize factors such as perceived humanness, engagement, and emotional response in shaping user experience. Researchers also highlight that conversational design, response clarity, and system reliability influence how users perceive chatbot systems and whether they trust the information provided by them.
METHOD
The AI advisor was designed as a conversational advising assistant that leverages a RAG architecture to answer student questions using institutional knowledge sources. The system integrates an LLM with a curated knowledge base of university documents, including course schedules, registration policies, prerequisite lists, and faculty information. These documents are uploaded and managed through an administrator interface, where institutional staff can add or update PDF files that the system uses as its source of information.
From a technical perspective, when a student submits a query, the system first retrieves relevant information from the stored documents and then generates a response grounded in that retrieved content (Figure 1). This design helps ensure that responses are based on verified institutional data rather than relying solely on the language model’s internal knowledge. Figure 2 illustrates the system’s administrator perspective, where staff can upload and manage knowledge base documents. These documents are then indexed and used by the AI advisor to generate responses to student queries.
To evaluate the usability and perceived usefulness of the AI advisor, a pilot user study was conducted with students from multiple colleges within the university. Participants were first given a brief introduction to the system and then asked to interact with the AI advisor for approximately 10 minutes. During this period, students were encouraged to perform exploratory advising-related tasks, such as asking questions about course availability, prerequisites, program requirements, or registration procedures.
Following the interaction period, participants completed a survey designed to assess their perceptions of the system. The survey collected demographic information and included several Likert-scale questions measuring usability, usefulness, clarity of responses, time efficiency, cultural appropriateness, and overall acceptance of the AI advisor. In addition, open-ended questions allowed students to provide qualitative feedback regarding what they liked about the system, the challenges they encountered, and suggestions for improvement.
Figure 1: An example of an interaction with the AI advisor.Figure 2: An example of managing the knowledge base from the admin’s side.
RESULTS
A total of 20 students participated in the pilot evaluation of the AI advisor system. After interacting with the system for approximately ten minutes, participants completed a survey assessing their experience with the AI advisor. Overall, the responses suggest that many students recognized the system’s potential usefulness, convenience, and accessibility, particularly for quickly retrieving advising-related information.
In terms of usability, 40% of participants (8 students) agreed or strongly agreed that the AI advisor was easy to use, while 30% (6 students) reported neutral perceptions. Similarly, 40% of students (8 participants) agreed or strongly agreed that they were able to obtain the information they needed from the system, while 25% (5 students) reported neutral responses.
Students also recognized the system’s potential efficiency benefits. Forty percent (8 students) agreed or strongly agreed that the AI advisor saved them time compared to traditional advising approaches, while 25% (5 students) remained neutral. These findings suggest that the majority of participants either experienced time savings or did not perceive any disadvantage relative to conventional advising methods.
Perceptions of response quality were generally positive. Forty-five percent of participants (9 students) agreed or strongly agreed that the responses provided by the AI advisor were clear and easy to understand, with an additional 25% (5 students) expressing neutral views. Furthermore, 50% of students (10 participants) agreed or strongly agreed that the system’s responses were respectful and culturally appropriate, reflecting generally favorable perceptions of the AI advisor’s tone and communication style.
Student attitudes toward future use also indicate encouraging potential for adoption. Forty percent (8 students) agreed or strongly agreed that they
would use the AI advisor again, and the same proportion indicated that they would recommend the system to other students. When combined with neutral responses (15% in both cases), more than half of the participants did not express negative attitudes toward continued use.
The open-ended responses provided additional context to these findings. Many students highlighted speed and convenience as key advantages, noting that the system could quickly provide answers without requiring them to search through multiple university resources or wait for an advising appointment. Several students also appreciated the simple explanations and conversational format, describing the system as helpful for clarifying course information and answering routine questions.
Meanwhile, students reported several limitations. Some participants experienced technical issues such as response delays or system lag, particularly when multiple users were interacting with the system. Others noted that the AI advisor occasionally provided general or incomplete responses, especially when asked more specific questions related to graduation requirements or student support services. A few students also mentioned that the system sometimes lacked sufficient information in its knowledge base, which resulted in answers such as “I don’t know.”
Students suggested several improvements, including expanding the system’s knowledge base with more institutional information, improving response speed, and providing more precise answers to complex advising questions. Some participants also recommended simplifying responses and improving the interface design. Overall, while students identified areas for improvement, the feedback suggests that AI-based advising systems have promising potential as a complementary tool for quickly accessing advising- related information.
DISCUSSION
The findings of this study suggest that AI-powered academic advising systems can significantly improve the accessibility and efficiency of advising services in higher education. Students generally responded positively to the system’s usability and appreciated the ability to receive instant answers to academic questions.
However, the results also highlight several important limitations: Participants noted that AI systems sometimes struggle with complex or highly specific questions. This limitation suggests that AI advisors should function as a support tool rather than a replacement for human advisors.
Accuracy and contextual understanding are important factors in influencing student trust in AI advising systems.
CONCLUSION
This study examined students’ perceptions of an AI-powered academic advising system designed to support university students’ academic planning and course selection.
The results demonstrate that AI advisors can improve access to academic information, reduce advising delays, and enhance the efficiency of student
Design and Evaluation of an AI Academic Advisor: Insights From Student Interactions 17 support services. Students generally perceived the system as easy to use and helpful for obtaining routine academic information.
However, the findings also indicate that AI advising systems must be carefully designed to ensure accuracy, contextual awareness, and culturally appropriate communication. Human advisors remain essential for complex advising situations and personalized academic guidance. Future research should explore long-term adoption of AI advising systems and examine how these technologies can be integrated effectively into university advising structures.
ETHICAL APPROVAL The study was approved by the Research Ethics Committee at Zayed University.
ACKNOWLEDGMENT
This research has been funded by Zayed University, UAE, under grant number 23014.
REFERENCES
Abdelhamid, S., Bangura, J., & Shah, S. (2025). Advisely: AI-Powered Academic Advising Using Large Language Models (LLMs). New Perspectives in Science Education. Assayed, S. K., Alkhatib, M., & Shaalan, K. (2024). A Systematic Review of
Conversational AI Chatbots in Academic Advising. Bilquise, G., Ibrahim, S., & Salhieh, S. (2023). Investigating student acceptance of an academic advising chatbot in higher education institutions. Education and Information Technologies, 29. Dawood, M. (2024). Assessing the effectiveness of Chatbots in providing personalized
academic advising and support to higher education students: A narrative literature review. Studies in Technology Enhanced Learning, 4(1). Kuhail, M. A., Al Katheeri, H., Negreiros, J., Seffah, A., & Alfandi, O. (2023). Engaging Students With a Chatbot-Based Academic Advising System. International Journal of Human–Computer Interaction, 39(10), 2115–2141. Luong, H., & Lunong, K. (2025). A Chatbot-Based Academic Advising Model for
Student in Information Technology: A Case Study. Saudi Journal of Engineering and Technology, 10(03). Lytridis, C., & Tsinakos, A. (2025). AIMT Agent: An Artificial Intelligence-Based Academic Support System. Information, 16(4), 275. Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A
systematic review. Computers and Education: Artificial Intelligence, 2(2), 100033. Rapp, A., Curti, L., & Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151, 102630. Samy, S., Maher, E., & El-Mahdy, A. (2026). Full-Duplex Relay-Aided D2D
Communication in Heterogeneous Network with Imperfect CSI. 2020 16th International Computer Engineering Conference (ICENCO).
Urbančič, M. (2024). Artificial intelligence in education: comparing the responses of different large language models. Sodobna Pedagogika/Journal of Contemporary Educational Studies. Varma, S., Shivam, S., Natarajan, S., Banerjee, A., & Roy, S. (2025). From Data
to Decisions: Enterprise-Level Domain-Specific Graph Retrieval-Augmented Generation Systems for Advanced Question Answering. Proceedings of the IEEE International Conference on Big Data and Smart Computing, BIGCOMP. Pune, India.
SEARCH THE AHFE DIGITAL BOOKSTORE BY ISBN
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2026)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.