Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Assistive Technology System for Highly Automated Vehicles to Support People with Mild Cognitive Impairment: A Human-Centered Design Approach

1 University of Kansas, Lawrence, KS 66045, USA ↗
2 University of Florida, Gainesville, FL 32611, USA ↗
3 University of Kansas Medical Center, Kansas City, KS 66160, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1005787

Abstract

Older adults with mild cognitive impairment (MCI) experience difficulties in memory, processing speed, attention, judgment, and visuospatial skills, which may impede the ability to perform various daily activities efficiently, including driving. The emergence of highly automated vehicles that do not require human intervention may offer significant benefits to individuals with MCI as these vehicles can increase mobility and independence. However, individuals with MCI may still be required to perform higher-level activities during a ride, which can be challenging for this user group. This research is focused on designing and prototyping a system that can help during trip planning and when interacting with an automated vehicle during normal and emergency operations. The proposed assistive technology system includes a secure mobile app, a real-time traveler monitoring system, an interactive in-vehicle agent for emergencies and safety functions, and a platform integrating all sub-systems with vehicle operations via a dashboard. The initial system requirements were identified through a series of interviews and focus groups with stakeholders, such as subject matter experts and older adults with and without MCI. Iterative participatory design sessions were further conducted to establish the information architecture and create visual and interactive designs. A final evaluation session with five individuals with MCI was conducted and showed favorable results in terms of system usability.

INTRODUCTION

Mild cognitive impairment (MCI) is considered an intermediate stage between normal cognition and dementia, and is characterized by changes in cognitive functions above and beyond those anticipated due to normal age-related cognitive decline. Older adults with MCI may exhibit deficits in memory, speed of processing, attention, judgment, and visuospatial skills (Overton et al., 2019). Because of these deficits, older adults with MCI may

© 2024. Published by AHFE Open Access. All rights reserved. 2192

Assistive Technology System for Highly Automated Vehicles to Support People 2193

perform basic activities of daily living (bADL) or instrumental activities of daily living (iADL) less efficiently than done previously (Langa & Levine, 2014). Studies have shown that older adults with MCI have impaired driving skills, such as poor lane and speed control, gap judgment (Wadley et al., 2009), lane-changing performance, and increased reaction time when braking (Lamble et al., 1999). In addition, poor attention and working memory are correlated with driving cessation for more than 30 percent of this group (Pyun et al., 2018).

Recent strides in automation technologies have led to the development of automated vehicles (AVs) that promise to improve safety, mobility, and accessibility, as these can remove human errors (Fagnant & Kockelman, 2015). AVs have a wide range of automation features. At the higher end of the automation spectrum, AVs that offer full driving automation without the need for user control of the vehicle, corresponds to levels 4 and 5, as defined by the Society of Automotive Engineers (SAE International, 2021). These systems are also called Automated Driving Systems (ADS). ADS can significantly benefit older adults with MCI and offer them increased independence and mobility, as they can relieve them from the operational (e.g., speed control) or tactical (e.g., lane choice) control of the vehicle (Haghzare et al., 2021). Nevertheless, it can be challenging for travelers with cognitive deficits such as MCI to ride an ADS-equipped car as they may still be required to perform higher-level activities during a ride.

Therefore, this research aimed to design an assistive technology system for ADS specifically for people with MCI, following a user-centered design approach. The proposed assistive technology system is comprised of (1) a traveler mobile secure app that assists users in trip planning and setting up the preferred vehicle actions during emergency or normal operations; (2) a traveler monitoring and state detection system that detects the traveler’s alertness and health status in real-time; (3) an in-vehicle agent designed to interact with the traveler during emergencies, to provide reminders and alerts, operate in-vehicle safety and convenience functions, and ensure contextual awareness; and (4) a platform integrating the various sub- systems with vehicle functions through a dashboard. The in-vehicle agent receives information from the mobile app about the traveler’s preferences and their selected destinations, connects with the traveler monitoring and state detection system to obtain the traveler’s health status, triggers the appropriate actions during an emergency, and controls vehicle safety and comfort features. The in-vehicle agent is central to the proposed system, as it (1) receives information from the mobile app about the traveler’s preferences and their selected destinations, (2) connects with the traveler monitoring and state detection system to obtain the health status of the traveler and triggers the appropriate course of actions during an emergency, and (3) controls vehicle safety and comfort features. A schematic of the entire system is provided in Figure 1.

A HUMAN-CENTERED APPROACH TO DETERMINING USER NEEDS AND SYSTEM REQUIREMENTS This study used a user-centered design approach for designing assistive technology for high-level AVs, adapted to individuals with MCI. The first step of our design approach was to better understand users and their needs, which was achieved through individual interviews with seven (N = 7) subject- matter experts (SMEs) (Group 1), 11 older adults without MCI (N = 11) aged 65–79 (Group 2), and seven older adults with MCI (N = 7) and their care partners (formal or informal), aged 65–79 (Group 3). The SMEs (18 years or older) were recruited from professional organizations of driver rehabilitation specialists (DRS), rehabilitation professionals, and academics with a focus on assistive technologies, rehabilitation sciences, engineering, and inclusive design (Eskandar et al., 2022). Group 2 older adults without MCI were recruited from the Lawrence, KS community. Group 3 older adults with a clinical diagnosis of MCI or mild to moderate dementia were recruited from the Alzheimer’s Disease Research Center at the University of Kansas Medical Center. The information obtained through these interviews fed into the ideation of the system functions and its preliminary design. The study was approved by the University of Kansas Medical Center (KUMC) Institutional Review Board (IRB # STUDY00147148).

Figure 1: Assistive-technology system diagram.
Original PDF Figure 1:

The SME interviews resulted in the identification of themes used for identifying system features and user needs that would need to be addressed by the user interface. The major themes were as follows:

Assistive Technology System for Highly Automated Vehicles to Support People 2195

  • Individuals with MCI who have a care partner are highly dependent on them for transportation and decision-making while feeling safer and more comfortable in their presence.
  • Although individuals with MCI need to be trained to use ADS, they can have impaired ability to learn, along with different preferences for how they learn new technologies (e.g., experiencing it, being trained by care partners, or having assistive learning technology).
  • ADS need to provide support for individuals with MCI during unexpected and emergency events. This support can include instructing users on how to seek help, offering them solutions for solving the issue, and connecting them to their care partners or other resources.
  • Individuals with MCI may have privacy concerns about the information they share with their ADS. They may want to know how, when, and by whom their information will be used.
  • Individuals with MCI are susceptible to forgetting important information (e.g., their destination) during their trip and may require reminders.

This thematic analysis, which was further validated through the interviews with the remaining two focus groups, informed our solution and resulted in the following key features and considerations:

  • Care partners are integral in onboarding and training individuals with MCI on ADS and the proposed system. Different types of training are needed based on user preferences.
  • Different emergency scenarios were defined: 1) traveler experiences a health emergency, 2) traveler feels stress or anxiety during the trip, 3) traveler falls asleep, 4) vehicle must take a mandatory detour, and 5) vehicle experiences a mechanical issue or is involved in an accident.
  • A survey of 85 older adults was conducted as part of this study to obtain their perceptions of privacy and security, where it was found that they are uncomfortable with data being stored longer than necessary and outside parties being able to view or sell their data.
  • A reminders system that helps individuals with MCI remember specific pieces of information about their trip was created.

After establishing the initial user requirements, we conducted iterative cycles of design that included establishing the information architecture and creating the visual and interactive designs. These steps allowed for the specification of system features, organization of information content in the in-vehicle interface and mobile app, creation of simple visual designs and content that would be suitable for individuals with MCI, and focus on interaction patterns that could be used in high-stress emergency scenarios or to help facilitate training and learning of the system (e.g., in the secure application).

We conducted four 2-hour participatory design (PD) sessions with stakeholders that included an older adult without MCI, older adult with MCI and their care partner, and an SME (driver rehabilitation specialist). In addition, members from the research team with human factors, user experience, and AV expertise also participated as facilitators. All sessions

were held over Zoom and the designs and discussions were shown through an online tool (Miro) to facilitate a shared understanding of the topic. The PD sessions improved information architecture, visual design, and interaction designs for the in-vehicle agent and mobile app.

ASSISTIVE-TECHNOLOGY SYSTEM COMPONENTS A detailed description of the system components is provided in this section.

Mobile Traveler Secure App The mobile secure app was built with two goals in mind. Firstly, the app provides a convenient interface for the traveler and care partners to plan trips by scheduling them ahead of time. Secondly, the app is the primary means by which users can configure their preferences for what actions the vehicle should perform under normal and emergency scenarios, through the in-vehicle agent. Internally, the system uses the notion of a playbook of individual steps (plays) that can be taken in response to the vehicle’s behavior. For example, if the vehicle gets into an accident, a particular emergency contact can be notified by text message. Building the app around the notion of pre-set plays gives travelers and care partners the chance for significant configuration but aims to be less overwhelming than performing that configuration under time-sensitive or stressful conditions.

A key design point of the app is that it can serve as a secure repository for personal details of the user. Smartphones already contain many privacy- sensitive details that a traveler or care partner may not feel comfortable inputting directly into another device (such as a vehicle). Where possible, the app attempts to perform its task without sharing these details with any other device. For example, when the in-vehicle agent initiates a call to an emergency contact, it does so by delegating the call to the app. In this way, the vehicle can initiate an emergency call without knowing the contact details of who is being called, which is instead routed through the secure app and delegated to the phone application. This approach also reduces the learning curve of the system, since details such as the contact list and calendar of the traveler are already available through the native interface on the phone.

Traveler Monitoring and State Detection Detecting the state of the travelers was found to be quite important for the care partners. In this prototype, we focused on alertness as our target population is susceptible to motion-induced sleepiness and distraction (Ren et al., 2019). The prototype uses computer vision libraries to monitor features associated with low alertness, such as pupil size/eyelid closure/mouth opening, through a remote camera embedded within the vehicle’s dashboard. These features are combined to predict the level of alertness of the traveler in real time. We further conducted a validation study where ten older adults were recruited to record their faces while drowsy and alert (Yao et al., 2022). To account for individual differences in facial structure and facial cues that indicate low alertness, the traveler monitoring system performs a calibration step for each new traveler. The algorithm assumes that these data contain

Assistive Technology System for Highly Automated Vehicles to Support People 2197

exemplars of a high alertness state for this traveler. When the features deviate from this exemplar, the algorithm predicts a loss in alertness level, in which case, the system plays a beep sound through a sound bar in the dashboard, to remind the traveler to reposition back to the center of the screen.

To continuously measure and monitor the physiological performance of travelers we use a wearable multimodal unit, which is an off-the-shelf smart wearable vest (Hexoskin). This vest has integrated a 1-lead electrocardiogram (ECG) (256 Hz) and is equipped with chest and abdominal respiratory inductive plethysmography (RIP) sensors (128 Hz). Cardiac and respiratory data can be accessed with the Bluetooth 4.1 standard. In addition, a smart customized vibration unit was designed, fabricated, and attached to the seatbelt. This vibration unit is used as an extra communication modality with travelers, especially in emergency situations, as it tends to be more effectively perceived by users compared to visual or auditory feedback (Winkler et al., 2018). The real-time predictions from the remote camera are combined with the wearable multimodal unit predictions, which are then consolidated into a final prediction of the alertness state of the traveler. After combining the output of the two monitoring subsystems, the result is broadcast to the in-vehicle agent via the Unicast User Datagram Protocol (UDP).

In-Vehicle Agent The in-vehicle agent was designed to help the traveler complete their trips within the ADS through a user-centered in-vehicle experience that can be personalized through the mobile secure application. Understanding how to interact with an ADS during emergencies was critical for both older adults with MCI and their care partners. The agent also helps the user stay informed and engaged with the trip through map displays, entertainment, communication options, and a system of reminders that is especially useful for this user group. Visual and auditory modalities were selected for interacting with the in-vehicle agent. The personalization of the responses through the secure app also helps to build trust and provide training about the ADS. The in-vehicle agent system is displayed on a central dashboard screen placed in front of the front traveler’s seat (Figure 2). This display presents information from the in-vehicle interface, such as climate control, entertainment, and communication options and provides immediate support when the in-vehicle agent detects emergencies through visual and auditory prompts and guidance.

We defined specific emergency scenarios that are relevant to individuals with MCI and their use of ADS during trips by themselves. Each emergency scenario that was defined was also associated with actions that the vehicle can take to help the traveler respond to the situation and reduce their stress. Five scenarios were defined: 1) traveler experiences a health emergency, 2) traveler feels stress or anxiety during the trip, 3) traveler falls asleep, 4) vehicle must take a mandatory detour, and 5) vehicle experiences a mechanical issue or is involved in an accident.

Figure 2: In-vehicle integrated platform and physical prototype.
Original PDF Figure 2:

Within each emergency scenario, the in-vehicle agent’s actions were prioritized based on the severity of the emergency, the timeliness required by the response, and information needs of the travelers and their care partners. Furthermore, default actions were taken by the in-vehicle agent when no response was received from the traveler to ensure that the traveler will always have a safe response from the vehicle. Each of the in-vehicle agent actions during the emergency scenarios can be customized (i.e., enabled or disabled) in the secure app to personalize the experience, which may help users anticipate and better understand what the vehicle will do during emergencies to improve user trust.

Figure 3 presents an example where the in-vehicle agent detects that the traveler may be experiencing a health emergency. The in-vehicle agent will show a series of possible “plays” that can assist the traveler in this situation. Each play is provided in a set sequence to prioritize the safety or convenience of the in-vehicle actions, depending on the criticality of the detected emergency event. At each stage, the in-vehicle agent will ask the user if they wish to initiate the play using a simple yes or no response. If no response is received, the in-vehicle agent will progress automatically through the action tree; these default options are shown in Figure 3 as the arrow directly below an option. The timing of these automatic responses varies depending on the play. For example, the “go to a hospital”play has a time-out time of 20 seconds to provide more time for the user to respond. The health emergency scenario showcases a large variety of actions that the in-vehicle agent can take to assist the user. These include contacting services (e.g., emergency services or business services), changing the destination, providing information, or contacting and providing information to care partners and emergency contacts. The in-vehicle agent also prioritizes the agency of the user by asking for user input and asking for confirmation whenever a play is proposed. This allows the user to veto unwanted actions and feel in control of the user experience. The customization of which plays are proposed by the in-vehicle agent in the secure application provides further personalization and control for the user.

Assistive Technology System for Highly Automated Vehicles to Support People 2199

Figure 3: Health emergency scenario flowchart.
Original PDF Figure 3:

The in-vehicle agent is securely connected to the mobile app to obtain the trip information as soon as the traveler enters the ADS. Important information about the trip, such as destination address, a map with the selected route, and expected time of arrival are also displayed. The user can also access vehicle controls during the ride through the in-vehicle agent, such as windows, fan speed, lighting, or music. Users can change their destination during the ride, e.g., to add a stop at another location.

In addition, we created a reminders system that helps individuals with MCI remember specific pieces of information related to their trip that they are susceptible to forgetting at crucial times throughout the trip. Examples of reminders provided by the system include 1) safety warnings (e.g., buckle your seat belt, check your surroundings when exiting the vehicle) and reminders for belongings brought in the car at the beginning and end of the trip, 2) destination confirmations when first entering the vehicle, 3) destination orientation reminders when the vehicle arrives at the destination, and 4) wellness checks where the system asks how they feel during the trip (Eskandar et al., 2022). An example illustration of a reminder for belongings at the end of the trip is presented in Figure 4.

We created visual and interaction designs for the in-vehicle agent that reduced the complexity of information presentation and input, reduced memory burden, applied usability and design guidelines for older adults, and provided information across multiple sensory modalities through auditory prompts, and tactile feedback.

Figure 4: Traveler safety steps and belongings checking reminder.
Original PDF Figure 4:

SYSTEM EVALUATION A final evaluation session was conducted with five older adults with MCI and their care partners to obtain their assessment of the overall usability of our system and the sub-systems. The interviews were conducted online. We described the overall system and shared our screen with interviewees to show the functionality of the system in the format of short videos and then asked for feedback. All interviewees had already participated in initial interviews, so they were familiar with the concept of ADS. During the evaluation session, we introduced a persona who was interacting with the integrated system. We described and illustrated the interaction with the system during six events, including preparing for a trip, booking the trip, detouring, falling asleep, having a health emergency, and arriving at the destination. After each event, we asked some open-ended questions such as: “Would you consider using these features? Why?”, “Is there any feature that you would like to include or exclude?” or “Do you have alternative ideas that can address this need better?” Interviewees were also asked to rate the effectiveness (Q1), usability (Q2), ease of use (Q3), memorability (Q4), and satisfaction (Q5) of the functionality of the system for each event using a 5-point Likert scale. The mean ratings for the mobile app were: Q1 = 4.4/5, Q2 = 4.3/5, Q3 = 3.5/5, Q4 = 3.6/5, Q5 = 4.4/5. The mean ratings for the in-vehicle agent were: Q1 = 4.3/5, Q2 = 4.6/5, Q3 = 4.0/5, Q4 = 4.0/5, Q5 = 4.4/5. These results are presented in Figure 5.

Assistive Technology System for Highly Automated Vehicles to Support People 2201

Figure 5: Usability evaluation results.
Original PDF Figure 5:

CONCLUSION

Advanced vehicle technology, such as ADS offers potential benefits for individuals with MCI by providing independence and mobility through automation. This research aimed to design an assistive technology system for ADS tailored to people with MCI, using a user-centered approach. The proposed system includes a secure traveler mobile app, a real-time traveler monitoring system, an interactive in-vehicle agent for emergencies and safety functions, and an integrated platform connecting these sub-systems to the vehicle via a dashboard prototype. The in-vehicle agent plays a crucial role in managing traveler preferences, health status, emergency actions, and vehicle safety and comfort features. Through a series of interviews, focus groups, and participatory design sessions, the initial user requirements, system functionalities, as well as the final design of our system were gathered. A final evaluation of the system with five individuals with MCI and their care partners was conducted, which showed favorable results in terms of the usability of the mobile app and the in-vehicle agent. However, given that this evaluation was done remotely, it is recommended to conduct a follow-up in-person evaluation with a larger number of participants.

ACKNOWLEDGMENT

The authors would like to acknowledge the US Department of Transportation which provided the funding for this research through the Inclusive Design Challenge. The authors would also like to acknowledge the research stakeholders and the undergraduate students at the University of Kansas and the University of Florida who worked on this project.

REFERENCES

Eskandar, M., W. C. W. Giang, S. Motamedi, H. Devos, L. Koon, A. E Akinwuntan, A. Kondyli (2022) Designing a Reminders System in Highly Automated Vehicles’ Interfaces for Individuals with Mild Cognitive Impairment. Frontiers in Future Transportation – Connected Mobility and Automation. https://doi.org/10.3389/ ffutr.2022.854553

Fagnant, D. J., & Kockelman, K. (2015). Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77, 167–181. https://doi.org/10.1016/j.tra.2015.04. 003 Haghzare, S., Delfi, G., Stasiulis, E., Mohamud, H., Dove, E., Rapoport, M. J.,

Naglie, G., Mihailidis, A., & Campos, J. L. (2021). Can automated vehicles be useful to persons living with dementia? The perspectives of care-partners of people living with dementia. The Gerontologist. https://doi.org/10.1093/geront/gnab174 Lamble, D.; Kauranen, T.; Laakso, M.; Summala, H. (1999) Cognitive load and detection thresholds in car following situations: Safety implications for using mobile (cellular) telephones while driving. Accid. Anal. Prev. 31, 617–623. Langa, K. M., & Levine, D. A. (2014). The diagnosis and management of mild

cognitive impairment: a clinical review. JAMA, 312(23), 2551–2561. https://do i.org/10.1001/jama.2014.13806 Overton, M., Pihlsgård, M., & Elmståhl, S. (2019). Prevalence and incidence of mild cognitive impairment across subtypes, age, and sex. Dementia and Geriatric Cognitive Disorders, 47(4–6), 219–232. https://doi.org/10.1159/000499763 Pyun, J. M., Kang, M. J., Kim, S., Baek, M. J., Wang, M. J., & Kim, S. (2018). Driving

cessation and cognitive dysfunction in patients with mild cognitive impairment. Journal of Clinical Medicine, 7(12), 545. https://doi.org/10.3390/jcm7120545 Ren P, Anderson AJ, McDermott K, Baran TM, Lin F. (2019). Cognitive fatigue and cortical-striatal network in old age. Aging (Albany, NY), 11(8), 2312 SAE International. (2021). Taxonomy and definitions for terms related to driving

automation systems for on-road motor vehicles (J3016_202104). https://www.sa e.org/standards/content/j3016_202104 Wadley, V. G., Okonkwo, O., Crowe, M., Vance, D. E., Elgin, J. M., Ball, K.

K., & Owsley, C. (2009). Mild cognitive impairment and everyday function: an investigation of driving performance. Journal of Geriatric Psychiatry and Neurology, 22(2), 87–94. https://doi.org/10.1177/0891988708328215 Winkler, S., Kazazi, J., & Vollrath, M. (2018). How to warn drivers in various safetycritical situations–Different strategies, different reactions. Accident Analysis & Prevention, 117, 410–426 Yao, H., S. Motamedi, W. C. W. Giang, A. Kondyli, E. Jain (2022) In-vehicle Alertness

Monitoring for Older Adults. https://doi.org/10.48550/arXiv.2208.08091

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →