Vanessa Mai, Caterina Neef, Anja Richert (2022). Developing an AI-based Coaching Chatbot: A Study on Disclosure as Effectiveness Factor in Human-Machine-Coaching. In: Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications. AHFE Open Access, vol. 23. https://doi.org/10.54941/ahfe100917
Coaching via chatbots can be classified as digital self-coaching. At TH Köln, an AI-based StudiCoachBot is currently being developed and implemented. The research focus is on investigating effectiveness factors in human-machine-coaching, especially in the working alliance. An interaction script for the coachbot was developed. The concept of self-disclosure was selected as an effectiveness factor, operationalized and its effectiveness compared to the self-developed concept of information disclosure. The use case is student coaching on the topic of exam anxiety. The Conversational AI Rasa was selected as a suitable system architecture. The effectiveness of the chatbot coaching was investigated in a study with ten students using a questionnaire. The results show that chatbot coaching is already well accepted in its current form. Information disclosure of a coachbot shows higher effectiveness in establishing a working alliance than self-disclosure of a chatbot. The study contributes to the effectiveness of digital self-coaching tools.
Coaching no longer takes place only face-to-face but is also practiced digitally, including self-coaching via chatbots (Kanatouri, 2020). It also represents an important didactic tool for reflecting on learning and working processes at universities (e.g. Mai, 2020). Here, it is increasingly important to make reflection processes scalable for a large number of students, such as in mechanical engineering. Feedback can be made more individualized and process-oriented through the use of digital and AI-based technologies. Such technologies can also stimulate self-coaching processes (e.g. Terblanche, 2020; Graßmann and Schermuly, 2020).
The Faculty of Process Engineering, Energy and Mechanical Systems at TH Köln/University of Applied Sciences successfully works with proven coaching formats in higher education. On this basis, reflection conversations are made scalable by implementing a coaching chatbot. An AI-based StudiCoachBot is currently being conceptualized and developed to deepen students’ self-reflection processes on learning and working strategies in a coaching process (Mai and Richert 2021). As part of a Ph.D. project the coaching processes between AI coaches and students are being investigated (Mai and Richert 2020).
RELATED WORK & RESEARCH QUESTION One of the central effectiveness factors in coaching is the development of a sustainable working alliance in the process (Künzli, 2019; Lippmann, 2013; Lindart, 2016). Therefore, the focus of this study is on investigating effectiveness factors in the working alliance between coachbot and human coachee. In particular, the concept of self-disclosure (Berninger-Schäfer, 2018) is selected as an effectiveness factor, operationalized, and compared with the self-developed concept of information disclosure (based on Gremler and Gwinner, 2008). Studies in human-machine interaction demonstrate (reciprocal) self-disclosure (Kang and Gratch, 2011; Lee et al. 2020). To assess the working alliance from the coachee’s perspective, the German short form of the Working Alliance Inventory, WAI-SR (Wilmers et al. 2008) represents an established set of items.
User-centered design research investigates the influence of chatbot personalities on user experience. Studies show that the personality of a chatbot can improve the consistency of the user experience. A chatbot’s personality can have a significant positive effect on user experience depending on context, chatbot task, and user group (Smestad and Volden, 2019).
The following research question emerges: What are the impact of a coaching chatbot’s information disclosure and self-disclosure on the student’s perceived satisfaction and effectiveness of the coaching process as well as on the perceived personality of the coachbot?
As the use case for the StudiCoachBot, the topic of exam anxiety was chosen. Based on intervention techniques of systemic coaching (Schlippe and Schweitzer, 2016) as well as oriented on coaching process steps from literature (Berninger-Schäfer, 2018), an interaction script for the coachbot has been developed. Ressource-oriented and solution-focused coaching questions were implemented (e.g. for goal setting: “How would you know your exam anxiety is gone?”). Moreover, either self-disclosure or information disclosure statements from the chatbot were inserted into the script. These were placed before each of the coaching questions. It was decided that the chatbot should either talk about its own experience with exam anxiety (self- disclosure) or reveal the same or similar information, just not from the chatbot’s perspective (information disclosure) (Mai et al. 2021).
The chatbot dialog concept was tested in a Wizard-of-Oz setting with 12 engineering students, including a chatbot coaching as well as a survey and interview. The results show that students disclosed themselves to the chatbot. Information disclosure showed even more positive effects than self-disclosure and seemed to be sufficient. The results further indicate that chatbot coaching in its current form is already well-received by students. By having the coaching done by a machine, students find it easy to open up. There is a great willingness to use it (Mai et al. 2021).
In a second step, a suitable system architecture for implementing the validated interaction script was selected (Mai et al. 2022). With Rasa (2020) a Conversational AI was chosen. The basis of Conversational AI chatbots is machine learning. Based on artificial intelligence, it enables them to learn and make connections on their own. The more data they receive, the better they perform (Brandão and Wolfram, 2018).
Experimental Design The experimental design of this study consists of a combination of chatbot coaching with the developed and programmed coaching chatbot and a survey. A questionnaire was used to capture the student’s perceived satisfaction and effectiveness of the coaching process as well as the coachbots personality (see Table 1). Ten students participated in two groups (information disclosure / self-disclosure). Coaching on the topic of exam anxiety was chosen as the use case. In both groups, the students were coached by the coaching chatbot via chat, but the coachbot in one group disclosed personal information about itself (self-disclosure) before the coaching questions, whereas in the second group it only disclosed general information.
Table 1: Categories and operationalized questions to evaluate the StudiCoachBot Category Question
Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 personality Did you feel that the coachbot had its own personality?
To assess the participants’ perceived effectiveness and satisfaction with the chatbot coaching as well as perceived chatbot personality, the questionnaire results were analyzed using statistical methods. First, the sociodemographic data of the subjects were analyzed. Second, the individual responses to the items/questions were evaluated according to their frequency. For this purpose, the categories from “rarely” to “always” were converted into numerical values from one to five and the arithmetic mean was formed. Furthermore, the statements in the comment section of the questionnaire were evaluated.
RESULTS & DISCUSSION Sample Description
Ten students participated in the experiment, five of whom interacted with the self- disclosure coaching chatbot and five of whom interacted with the information disclosure chatbot. Six trial participants were male and four were female. The age of the subjects ranged from 20 to 32 years, with an average age of 25. All subjects came from different bachelor’s and master’s degree programs, with half of the subjects enrolled in engineering science majors and the other half in social science majors.
In addition, the sample can be described according to the strength of the students’ exam anxiety. On a scale from 1 = very low to 10 = very high, the average exam anxiety of the subjects was M=5.6. The minimum of test anxiety was three and the maximum was eight. In the group with self-disclosure, the subjects had higher exam anxiety (M=6.0) than in the group with information disclosure (M=5.2).
The evaluation of the results in the category coachbot personality shows that the values for the chatbot with information disclosure are significantly higher (M=3.5) than in the group with self-disclosure (M=2.7). Thus, the subjects who interacted with the coachbot with information disclosure feel more strongly that the coachbot has a personality of its own than the subjects in the group with self-disclosure. Only on the question of the openness of the coaching chatbot (Question 3) are the scores identical in both groups (see Figure 1).
Furthermore, the subjects in the group with information disclosure are significantly more satisfied with the coaching chatbot (M=3.2) than in the experimental group with self-disclosure of the coaching bot (M=2.5).
Also with regard to the perceived effectiveness of the chatbot coaching, the chatbot group with information disclosure shows higher values (M=2.95) than the group with self-disclosure (M=2.05). What is striking is the different evaluation of the two groups with regard to the question: “Would you apply or use the coachbot’s suggestions and tips?” (Question 8). Here, the coachbot with information disclosure achieves a significantly higher agreement (M=4.2) compared to the coachbot with self-disclosure (M=2.2) (see. Figure 1).
Figure 1: Perceived satisfaction and effectiveness of the coaching process as well as perceived StudiCoachBots personality (N=10) on a scale from 1-5.
Evaluation of the Comment Area The results from the questionnaire are supported by the statements in the comment area. The test subjects state that the coaching chatbot (with information disclosure) provides assistance for self-reflection: “Yes, you can think about your problems better and get suggestions on how to improve them.” The resource-oriented coaching questions seem to have a positive influence on the subjects: “It reminds you of important points to fight anxiety because many points like positive feedback you
Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 quickly forget and only focus on negatives. The conversation makes you think back to those situations.”
However, in the statements in the comment area also criticism on chatbot coaching can be found. The main points are that the conversation seems quite “rigid” and that the coachbot cannot react individually to the user: “The chatbot seems somewhat ‘stiff’ and therefore answers like a typical computer. The questions and answers seem predetermined. However, this may also be due to the bot’s development stage.” Or: “The bot did not really respond to my answers, always ‘told’ something directly and then asked the next question.”
Discussion
About all question categories, the coaching chatbot with information disclosure performs better than the chatbot with self-disclosure: The test subjects are more satisfied with the coaching by the chatbot with information disclosure and perceive the conversation as more effective than with the self-disclosure coachbot. They also attribute more of a personality to the information disclosure chatbot than to the self- disclosure chatbot: “The easy-going nature of the bot is well received, so the feel- good factor is present.”
These findings are consistent with the results from a study by Mai et al. (2021) in that the disclosure of general information (information disclosure) seems to be sufficient and the perceived effectiveness and satisfaction is higher than when interacting with a self-disclosure coachbot. However, the results contrast with the findings of previous research in which a machine’s self-disclosure had a positive effect on the working-alliance-like constructs of copresence and social attraction (Kang and Gratch, 2011), as well as enjoyment, intimacy, and trust (Lee et al. 2020). The reason can be seen in the different construction of the test groups. In the test groups of the studies of Kang and Gratch (2011) and Lee et al. (2020), self-disclosure was compared to not disclosing any information at all. Here, self-disclosure was compared to information disclosure, which seems to build a better alliance.
Further, the results of this study show that chatbot coaching already supports students in its current form to reflect on their exam anxiety. This result is consistent with findings from technology acceptance studies on chatbot coaching. These studies are underrepresented so far. They suggest that chatbot coaching is accepted – even in early development phases – as long as robust system architectures and coaching frameworks are used in chatbot design (Terblanche and Cilliers, 2020).
The criticism of chatbot coaching, that the conversation seems too rigid and that users would have liked individual responses, can be better classified with a look at the technical implementation of the coachbot. The use of Conversational AI for chatbot coaching is very complex. Training effort and time are high and long. However, it offers many advantages for the student coaching use case once such a system is sufficiently trained. Chatbot developers can trigger the self-learning process of the chatbot system in a more individualized and process-oriented way by an increasing amount of training data. The system begins to understand the user’s free text input, learns to respond individually to user input and select different conversion
Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 paths. As one test person put it: “As soon as the chatbot has more info to point out ‘realistic’ solutions to problems, or the possibilities it has entered into the database or gained from it, the chatbot will be an even better help!”
Limitations, Future Research & Design Implications The present study is accompanied by some limitations. The sample is very small with ten test persons, so that the results only show initial tendencies and are to be regarded as explorative. In a study with a larger sample, the results need to be verified. In addition, the Conversational AI chatbot is still in an early development phase, which could have an influence on the results. This results in the following design implications: Both the technical and the conceptual development of the StudiCoachBot are central aspects that cannot happen separately, but only interdisciplinary hand-in-hand between developers, chatbot designers and coaching experts.
In the next step, the AI-based StudiCoachBot will be further trained and the results of this study are to be verified in a study with a larger sample. Further effectiveness factors in the working alliance in human-machine coaching will also be investigated (e.g., the influence of the interaction method of the coaching chatbot; results of a preliminary study can be found in Mai et al. 2022). It will also be interesting to examine which conversation paths users use. The chat histories can be examined for this purpose.
CONCLUSION
The results of this study show that the perceived satisfaction and effectiveness of chatbot coaching can be classified as rather high overall. They suggest that chatbot coaching, in its current form, already helps students reflect on their exam anxiety and is already well received by students. They obviously find it easy to open up to a machine in coaching and show a great willingness to use a coaching chatbot.
The results further indicate that the disclosure of general information seems to be sufficient and that the perceived effectiveness and satisfaction is higher than with a self- disclosing chatbot. That means, within the framework of these studies, another effectiveness factor in chatbot coaching – namely information disclosure – could be discovered and developed.
The conducted study is interdisciplinary at the interface of coaching effectiveness research and rapport/bonding research in human-machine interaction. The results are transferable to other areas of human-machine interaction. The working alliance, personality, as well as disclosure factors could also have an impact in other domains, such as in the use of care robots or other sensitive communication domains.
This journal contribution is based on a project carried out as part of the module “Research Seminar” in the master's program Mechanical Engineering at TH Köln/University of Applied Sciences. We kindly thank the master’s students Artur Fuchs, Robin Mathiak, Matej Udovicic and Sophia Wallau for carrying out this project.
REFERENCES
Berninger-Schäfer, E. (2018). Online-Coaching. Wiesbaden: Springer Fachmedien Wiesbaden. Brandão, T. K., Wolfram, G. (2018). Digital Connection: Die bessere Customer Journey mit
smarten Technologien – Strategie und Praxisbeispiele. Wiesbaden: Springer. Graßmann, C. and Schermuly, C.C. (2020). Coaching With Artificial Intelligence: Concepts and Capabilities. Human Resource Development Review, p.153448432098289. Gremler, D.D. and Gwinner, K.P. (2008). Rapport-Building Behaviors Used by Retail
Employees. Journal of Retailing, 84(3), pp.308–324. Kanatouri, S. (2020). The Digital Coach. Routledge. Kang, S.-H. and Gratch, J. (2011). People like virtual counselors that highly disclose about themselves. Studies in health technology and informatics, 167, pp. 143–148. Künzli, H. (2019). Spielstand 1:0—Die Wirksamkeit von Coaching. In Ryba, A. and Roth, G.
(eds.). Coaching und Beratung in der Praxis: Ein neurowissenschaftlich fundiertes Integrationsmodell. Klett-Cotta, pp. 102-124. Lee, Y.-C., Yamashita, N., Huang, Y. and Fu, W. (2020). “I Hear You, I Feel You”: Encouraging Deep Self-disclosure through a Chatbot. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Lippmann, E. (2013). Coaching – Angewandte Psychologie für die Beratungspraxis.
Berlin/Heidelberg: Springer. Lindart, M. (2016). Was Coaching wirksam macht. Wiesbaden: Springer Fachmedien Wiesbaden. Mai, V., Neef, C. and Richert, A. (2022). “Clicking vs. Writing”—The Impact of a Chatbot’s
Interaction Method on the Working Alliance in AI-based Coaching. Coaching | Theorie & Praxis. Mai, V., Wolff, A., Richert, A. and Preusser, I. (2021). Accompanying Reflection Processes by an AI-Based StudiCoachBot: A Study on Rapport Building in Human-Machine Coaching Using Self Disclosure. HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture, pp.439–457. Mai, V., Richert, A. (2021). StudiCoachBot an der TH Köln – Reflexionsprozesse KI-basiert
begleiten. fnma Magazin – Forum neue Medien in der Lehre Austria, [online] 1-21, pp. 21-23. Available at: https://www.fnma.at/publikationen/magazin [Accessed 18 Jan. 2022]. Mai, V. and Richert, A. (2020). AI Coaching: Effectiveness Factors of the Working Alliance in the Coaching Process between Coachbot and Human Coachee – an Explorative Study. EDULEARN20 Proceedings, [online] pp.1239–1248. Available at: https://library.iated.org/view/MAI2020AIC [Accessed 18 Jan. 2022].
Mai, V. (2020). Projektcoaching und Leadership-Coaching als integrative Elemente in der Ingenieurausbildung. cos.bibl.th-koeln.de. [online] Available at: https://cos.bibl.th- koeln.de/frontdoor/index/index/docId/926 [Accessed 18 Jan. 2022]. Rasa. (n.d.). How to Build a Contextual AI Assistant. [online] info.rasa.com. Available at:
https://info.rasa.com/how-to-build-contextual-assistant [Accessed 18 Jan. 2022]. Schlippe, A., Schweitzer, J. (2016). Lehrbuch der systemischen Therapie und Beratung I. Das Grundlagenwissen. Göttingen: Vandenhoeck & Ruprecht, Göttingen. Smestad, T.L. and Volden, F. (2019). Chatbot Personalities Matters. Internet Science, pp.170–
181. Terblanche, N. (2020). A design framework to create Artificial Intelligence Coaches.
International Journal of Evidence Based Coaching and Mentoring, [online] 18(2), pp.152–165. Available at: https://radar.brookes.ac.uk/radar/file/312d40ec-ccdf-431c- a062-2aa862166ac4/1/18_2_11.pdf [Accessed 18 Jan. 2022]. Terblanche, N. and Cilliers, D. (2020). Factors that influence users’ adoption of being coached by an Artificial Intelligence Coach. Philosophy of Coaching: An International Journal, 5(1), pp. 61-70. Wilmers, F., Munder, T., Leonhart, R., Herzog, T., Plassmann, R., Barth, J., & Linster, H.
(2008). Die deutschsprachige Version des Working Alliance Inventory – Short revised (WAI-SR) – Ein schulenübergreifendes, ökonomisches und empirisch validiertes Instrument zur Erfassung der therapeutischen Allianz. Klinische Diagnostik & Evaluation, 1(3), pp. 343–358.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8989-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional