Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

How should we design AI tools that handle personal information? Evaluating AI-generated personalized care advice based on deeply personal data

1 Data Science Design Special Team, PMSP, IMS, RIKEN, Yokohama, 2300045, Japan ↗
2 Interactive Robot Research Team, GRP, R-IH, RIKEN, Kyoto, 6190288, Japan ↗
3 Behavior Learning Research Team, GRP, R-IH, RIKEN, Kyoto, 6190288, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1007081

Abstract

AI systems typically rely on commonly available knowledge from the internet—that is, public “common sense.” However, when applying AI in personalized service domains such as healthcare and elder care, it becomes essential to incorporate deeply personal information, such as individuals’ life histories. This paper introduces a case study of an AI tool that provides personalized care advice to care workers, aiming to derive insights for designing services that focus on the individuality of each service user. We developed a prototype tool using a profile sheet constructed from real narratives. Care workers’ evaluations, analyzed qualitatively, yielded insights regarding AI usefulness, individualized care, practical applicability, advice presentation, limitations and risks, and AI use contexts.

INTRODUCTION

The knowledge possessed by generative AI is generally considered “common sense,” drawn from information available on the internet. When applying AI to services targeting individuals, such as healthcare or caregiving, it is necessary to handle not only common knowledge but also personal infor mation, including life history. However, even if AI were equipped with func tions to handle personal data, collecting and utilizing substantial personal

information that encompasses a person’s life history and values would require disclosing the individual’s life data itself. Gaining understanding and consent for the provision and use of such data is not straightforward. This study introduces a case study of an AI tool that provides personalized care advice to care workers, aiming to derive insights for designing services that focus on the individuality of each service user—an approach increasingly expected across various industrial domains.

ISSUES ADDRESSED AI and Personal Information

One advantage of using AI is its ability to present ideas from diverse perspec tives based on large datasets. Human caregivers can only generate a limited number of ideas at a time, and it is difficult for them to simultaneously con sider multiple approaches. For example, when a care recipient with dementia refuses assistance with bathing, it is challenging for a care worker to quickly generate multiple ideas for understanding the reason for refusal and suitable ways to respond.

AI, on the other hand, can propose ideas based on common knowledge, but information for empathizing with a person’s feelings, life background, and values remains limited. In healthcare and caregiving services, it is crucial to provide empathetic, individualized support, requiring AI to handle not only common knowledge through large language models but also personal information.

While using more personal information may improve the level of individu alized service, it is necessary to evaluate whether such use is valuable to both service providers and recipients. If recipients are not satisfied with services despite extensive personal information being used, they may be reluctant to provide data. Conversely, if services provide high satisfaction, recipients may recognize the value and provide more personal information. Investigating the acceptability and perceived value of AI-driven personalized services is there fore important, not only to explore their potential but also to inform ethical considerations regarding the handling of personal information.

AI in Healthcare The potential of AI in healthcare has been extensively reviewed (Rodriguez et al., 2024). There are examples of care workers evaluating ChatGPT- generated advice for dementia care (Aguirre et al., 2024). Chatbots have been proposed to support care workers (Espinoza et al., 2023), as well as tools capable of handling not only care workers’ intentions but also care recip ients’ profiles (Hasan et al., 2024). Other studies have explored AI-driven motivational effects (Sung et al., 2025). However, fundamental caregiving principles, such as respecting individual dignity and supporting independent living, must be fully considered in AI service design. Research has addressed ethical considerations for patient safety (Hoelscher et al., 2024), the necessity

How Should We Design AI Tools That Handle Personal Information 219 of generating AI support based on human feedback (Sezgin et al., 2024), enhancing the potential of persons with dementia while considering societal attitudes (Souval et al., 2025), and issues of bias, privacy, and transparency (Treder et al., 2024). Although AI applications often emphasize productivity gains, the balance between efficiency and individualized care requires careful consideration, as highlighted in discussions of the AI productivity paradox (Wachter et al., 2024).

This study investigates the acceptability and perceived value of utilizing personal information in AI, contributing to discussions on AI use in individ ual-oriented services, such as healthcare. To obtain evaluation from service users who provide personal information, it would be necessary to collect and process their data, which raises ethical concerns. Therefore, in this study, AI processes personal information provided by the experimenter, and care workers evaluate the results, allowing examination of perceived acceptability and value from the perspective of service providers.

UTILIZING NARRATIVES Challenges in Narrative Use

To provide high-quality individualized care, it is necessary to understand the narratives of each care recipient (Guendouzi et al., 2015). Narrative approaches have been proposed as effective in dementia care (Elfrink, 2023; Heersmink, 2022; Hughes, 2014; Randall, 2016). However, collecting and utilizing narratives in practice is challenging due to various factors, such as lack of rapport leading to limited self-disclosure by care recipients and insuf ficient staff skills. Narrative fragments are typically obtained during initial assessments or through daily care conversations, but they are rarely organized in ways that can directly inform care. There are currently no AI-supported services that allow care workers to reference collected narrative fragments effectively.

Narrative Utilization Ecosystem We proposed an ecosystem for collecting and using narrative fragments in caregiving settings (see Figure 1) (Ihara et al., 2025). Through this ecosys tem, collected narratives can inform care aligned with care recipients’ values, enhance rapport between care workers and recipients, encourage further self-disclosure, and facilitate the collection of additional narratives. The ecosystem consists of three components: narrative collection, AI-driven nar rative analysis, and practical utilization of narratives in caregiving settings.

Figure 1: Narrative utilization ecosystem.
Original PDF Figure 1:

Narrative collection includes not only initial assessments and daily care but also conversations during familiar area walks. Recorded conversations are transcribed using automatic speech recognition and transcription tech nologies. When daily conversation recordings are difficult, team meetings where staff share care episodes are recorded. The transcribed text is then summarized by AI into a profile sheet. In narrative analysis, AI generates care advice referencing the profile sheet, understanding not only common knowledge but also individualized requests derived from narratives, and pre senting tailored care recommendations to care workers. In practical care, care workers lacking experience or skills can provide appropriate care while referencing AI advice, allowing teams to deliver high-quality, individualized care that respects dignity and supports independent living.

PROTOTYPE OF AI CARE ADVICE TOOL Tool Specifications

To examine the effectiveness of AI-provided personalized care advice, we developed a prototype tool as a key component of the narrative utilization ecosystem. The input/output interface was determined based on prior inter views with care workers. Care workers could verbally ask questions to a notebook PC, which converts the speech into text and displays AI-generated care advice on screen. The care workers requested concise advice, so advice was limited to approximately two lines.

Advice consisted of two elements: example phrases for communication and reasoning behind the suggested approach. For instance, for a care recipi ent refusing a bath, an example phrase might be “Would you like to try just a shower? It can reduce the itchiness from atopic dermatitis,” with the reason ing being “This individual usually showers only and has a history of atopic dermatitis.” Figure 2 shows an example of advice displayed on the screen.

Figure 2: Example of advice displayed on the screen.
Original PDF Figure 2:

Profile Sheet Collecting extensive narrative fragments from actual care recipients for AI processing requires careful ethical consideration. Given limited societal understanding of these ethical issues, this study used narratives from the first author, assuming the author as a prospective care service recipient. Initial assessment interviews were conducted by a care worker according to stan dard assessment sheets, totalling 26 minutes, and casual daily-care conver sations were also recorded for 22 minutes. Both recordings were transcribed via automatic speech recognition and processed by ChatGPT to produce a profile sheet, including information on basic daily activities, health, mobility, hobbies, lifestyle, values, and care preferences and concerns.

EVALUATION Six care workers individually evaluated the prototype tool. First, they were presented with the transcribed interviews and AI-generated profile sheet, and the data collection process was explained. Next, a demonstration simulated the staff using the tool, with AI providing advice for two cases: refusal of bathing and refusal of medication. Subsequently, interviews were conducted to gather feedback, each taking approximately 15 minutes. Care workers were encouraged to suggest improvements.

ANALYSIS Evaluation comments were analyzed qualitatively using the Grounded Theory Approach(GTA). Comments were aggregated into 24 meaning units, coded through open coding (Table 1), and categorized using axial coding (Table 2). Results indicated that care workers perceived the AI tool as a sup port for supplementing their care thinking and decision-making, while being mindful of its expectations and limitations.

Table 1: Results of GTA open coding.

Table available in the original PDF paper.

No. Data (Comment) Code 1 Summarizing and organizing a large amount of conversation is impressive.

Expectation for information summarization and organization Provision of new viewpoints and ideas

2 It’s good to receive advice from diverse perspectives when staff can’t think of solutions.

Value of emotionally neutral advice

3 It’s helpful that AI provides advice without emotional involvement.

4 It’s not good to rely too much on AI. Concern about overdependence on AI Lack of depth and diversity in advice 5 The advice is too shallow / only one suggestion is given.

Interest in situation-specific advice

6 I wonder how advice changes for people with dementia.

7 It could help unify care among staff. Promoting consistency in team care 8 The tool lacks a sense of tempo; care should match the user’s pace.

Mismatch between tool tempo and real-world interaction Use as a pre-care preparation tool

9 It’s good to consult AI beforehand to prepare communication patterns.

Desire for flexible and portable use

10 Every user is unique; I’d like to use it on a smartphone.

11 It would be helpful when working night shifts alone.

Need for support in isolated situations Desire for simple operability

12 I’d like to turn it on easily only when needed.

Consideration for user privacy

13 The voice output should be heard only by the caregiver.

14 I’m not sure AI can handle angry residents.

Skepticism about AI’s practical effectiveness Expectation for personalized background information

15 It would be nice if it could tell me about the resident’s generation and favorite songs.

16 It would be great if AI could suggest countermeasures in care conferences.

Expectation for support in team discussions Expectation for preventive data- based advice

17 I want preventive suggestions for subcutaneous bleeding based on behavioral data.

Need for predictive care advice

18 I’d prefer advance advice for residents who often refuse bathing.

19 Recording should work via a wearable device.

Consideration of practical device form

20 Recorded data might be used in court; that’s a concern.

Awareness of privacy and ethical risks

21 Recording might be acceptable during reminiscence activities.

Context-dependent tolerance for recording (Continued)

Table 1: Continued.

Table available in the original PDF paper.

No. Data (Comment) Code

Desire for multiple-choice advice

22 It would be nice to have multiple advice options to choose from.

Value of individualized advice

23 It’s helpful that advice is based on the person’s background and conditions.

Recognition of AI’s advantage in personalization

24 Online searches don’t reflect individual situations, so this kind of tool is better.

Table 2: Results of GTA axial coding.

Table available in the original PDF paper.

Category Subcategory Related Codes Usefulness of AI advice Organizing information and providing 1, 2, 3

new insights Supporting consistent and reflective care 7, 9, 16

Adapting to user tempo and operability 8, 12, 19

Compatibility with real-world practice

10, 13, 21

Device form and environmental constraints 4, 11, 14

Awareness of AI’s limits and risks

Emotional, trust, and dependence concerns Privacy and ethical concerns 20

Responsiveness to individualized care

Reflecting personal traits and medical conditions

6, 15, 17, 18, 23, 24 5, 22

Expectations for advice presentation

Depth, diversity, and selection of suggestions Night shifts, meetings, recreation 11, 16, 21

Diversity of use contexts

DISCUSSIONS Care workers perceived the AI-based care advice tool as a partner that assists reflective thinking and decision-making in caregiving. Their effective ness, however, depends on conditions such as depth and diversity of advice, personalization, practical adaptability, and ethical assurance. Rather than replacing care workers, AI is positioned as a “thinking companion” that sup ports professional judgment in the person-centered practice of care.

Usefulness of AI Advice AI was evaluated positively for providing diverse perspectives and objective reasoning support for complex caregiving challenges. Care workers appre ciated not only the organization of large amounts of information but also AI’s emotion-neutral guidance, which complements advice from experienced colleagues that may be emotionally influenced.

Responsiveness to Individualized Care Care workers were particularly interested in how AI advice could be tailored to individual care recipients. Personalized recommendations based on specific symptoms, circumstances, or personal context were considered valuable. They recognized that searching online often fails to yield person alized guidance, highlighting the advantage of AI capable of individualized recommendations.

Compatibility With Real-World Practice Practical applicability was important. Demonstrations allowed care workers to realistically assess tool usability. Suggestions included recording on wear able devices, smartphone usage, and activating the tool only when needed. They also emphasized the tool’s need to present advice at an appropriate pace aligned with their adjustments during care. In fact, when a care recipient is slow to respond, care workers wait without rushing them, and adjust the speed of their own words and actions based on the care recipient’s reaction.

Expectations for Advice Presentation Care workers requested deeper advice and multiple options. While the pro totype presented single-pattern advice, future improvements could generate multiple alternatives. Advice can include both example phrases and under lying rationale, with references to specific sections of the profile sheet to provide deeper, value-informed guidance.

Awareness of AI’s Limits and Risks Care workers expressed caution regarding over-reliance on AI, particularly concerning staff training by the leader of the care team. Emphasizing under lying reasoning over superficial techniques may be more beneficial for novice care workers. Privacy and ethical concerns regarding recording were also noted. Some skepticism existed regarding AI’s effectiveness in handling chal lenging behaviors, such as care recipients prone to anger.

Diversity of Use Contexts Care workers identified the tool’s value in solo night shifts and as a sup port during team conferences. Ideas also included using AI for reminiscence therapy in recreational activities, suggesting that acceptability varies with context. In particular, AI providing appropriate care advice at conferences where care workers gather will be effective in standardizing care and raising the quality of care provided by the team.

Understanding New Care Recipients The care workers who participated in the evaluation had only met the person described in the profile sheet (the first author) once in the past and did not know their personality. Nevertheless, they gave a positive evaluation of the

How Should We Design AI Tools That Handle Personal Information 225 AI-generated care advice. This suggests that AI-generated care advice may be useful even when care workers have little prior knowledge of care recipients. In practice, the personality and values of a person who has just started using a care service are often not fully captured in the standard assessment sheet. In our experiment, AI generated a profile sheet using only two recorded con versations and used it to provide care advice. Being able to easily document, organize, and apply information about a person’s character—information that is often left out of staff-written assessment sheets—is a major advantage.

CONTRIBUTIONS AND LIMITATIONS This study contributes a case study offering insights into designing AI ser vices attentive to individual differences in service users. It illustrates narrative collection and utilization in realistic caregiving scenarios, particularly for staff facing challenges such as bathing refusal. Qualitative analysis of staff evaluations provided insights on AI usefulness, individualized care, practi cal applicability, advice presentation, limitations and risks, and potential use contexts. Limitations include evaluation using a prototype with basic func tions, and the insights are derived from a specific scenario and may not gen eralize to all individualized service designs.

CONCLUSION

This study presented a case study of an AI tool providing personalized care advice to care workers, using a profile sheet constructed from real narratives. Care workers’ evaluations, analyzed qualitatively, yielded insights regard ing AI usefulness, individualized care, practical applicability, advice presen tation, limitations and risks, and AI use contexts. Future work will include implementation of the narrative utilization ecosystem in ongoing care staff meetings to enable continuous narrative collection and further evaluation.

ACKNOWLEDGMENT

We would like to thank the employees of the nursing facilities, Living Aeru, Group Home Obisako, and Satellite Roken Aozora for cooperating in the evaluations.

REFERENCES

Aguirre, A., Hilsabeck, R., Smith, T., Xie, B., He, D., Wang, Z., Zou, N. (2024).

Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis. JMIR Aging; 7: e53019. https://doi.org/10.2196/53019 Elfrink, T. (2023). Exploring Life Story Books in Dementia Care: The potential of digital reminiscence interventions. [PhD Thesis - Research UT, graduation UT, University of Twente]. University of Twente. https://doi.org/10.3990/1.9789036558143 Espinoza, F., Cook, D., Butler, C. R. et al. (2023). Supporting dementia caregivers

in Peru through chatbots: Generative AI vs structured conversations. https://doi. org/10.14236/ewic/BCSHCI2023.11

Guendouzi, J., Davis, B., Maclagan, M. (2015). Expanding Expectations for Narrative Styles in the Context of Dementia. Topics in Language Disorders. 35, pp. 237–257. https://doi.org/10.1097/TLD.0000000000000061 Hasan, W. U., Zaman, K. T., Wang, X. et al. (2024) Empowering Alzheimer’s caregiv

ers with conversational AI: a novel approach for enhanced communication and personalized support. npj Biomed. Innov. 1, 3. doi: 10.1038/s44385-024-00004-8 Heersmink, R. (2022). Preserving Narrative Identity for Dementia Patients: Embodiment, Active Environments, and Distributed Memory. Neuroethics 15, 8. https://doi.org/10.1007/s12152-022-09479-x Hoelscher, S. H., Taylor-Pearson, K., Wei, H. (2024). Charting the Path: Nursing

Leadership in Artificial Intelligence Integration into Healthcare, Nurse Leader, Vol. 22, Issue 6, pp. 763–772, ISSN 1541–4612. https://doi.org/10.1016/j. mnl.2024.07.011 Hughes, T. (2014). Stories of professional care: Narrative analysis of accounts from people with dementia. Prof Doc Thesis University of East London School of Psychology. https://doi.org/10.15123/PUB.4063 Ihara, M., Tokunaga, H., Nakashima, T., Goto, H., Umezaki, Y., Minato, T.,

Nakamura, Y., Saruwatari, S. (2025). Narrative Utilization Ecosystem for Person- Centered Care, Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems, Vol. 160, pp. 1–11. Randall, W. (2016). Narrative Environments for Narrative Care: The Need to Honor People’s Stories. International Journal of Human Caring. 20, pp. 1–4. https://doi. org/10.20467/1091-5710-20.3.142 Rodriguez, D., Lawrence, K., Gonzalez, J., Brandfield-Harvey, B., Xu, L., Tasneem,

S., Levine, D., Mann, D. (2024). Leveraging Generative AI Tools to Support the Development of Digital Solutions in Health Care Research: Case Study. JMIR Hum Factors; 11: e52885. https://doi.org/10.2196/52885 Sezgin, E., McKay, I. (2024). Behavioral health and generative AI: A perspective on future of therapies and patient care. npj Mental Health Res 3, 25. https://doi. org/10.1038/s44184-024-00067-w Souval, H. D., Haber, Y., Tal, A., Simon, T., Elyoseph, T., Elyoseph, Z. (2025).

Transforming Perceptions: Exploring the Multifaceted Potential of Generative AI for People With Cognitive Disabilities. JMIR Neurotech; 4: e64182. doi: 10.2196/64182 Sung, G., Guillain, L., Schneider, B. (2025). Using AI to Care: Lessons Learned from Leveraging Generative AI for Personalized Affective-Motivational Feedback. Int. J. Artif. Intell. Educ. https://doi.org/10.1007/s40593-024-00455-5 Treder, M. S., Lee, S., Tsvetanov, K. A. (2024). Introduction to Large Language

Models (LLMs) for dementia care and research. Frontiers in dementia, 3, 1385303. https://doi.org/10.3389/frdem.2024.1385303 Wachter, R. M., Brynjolfsson, E. (2024). Will Generative Artificial Intelligence Deliver on Its Promise in Health Care? JAMA. 331(1): pp. 65–69. doi: 10.1001/ jama.2023.25054

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-76-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →