Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

UX research for autonomous vehicles: Focusing on interaction with mobile device

1 Samsung Electronics, Suwon, Gyeonggi, South Korea ↗
2 Hongik University, Graduate School of Industrial Arts, Service Design, Seoul, South Korea ↗
Publication year
Digital Object Identifier10.54941/ahfe1004335

Abstract

Since starting to launch the autonomous driving vehicle service to the public, customer experience has also emerged as an important factor for the services. Beyond simply considering the interaction between users and machines, this study seeks to simultaneously study another element that cannot be ruled out., with the aim of providing a satisfactory experience for the customer in terms of service. Self-driving taxis are a new service, but users will expect a continuous experience based on the experience they have learned from similar services. The existing similar experience is based on three services or technologies: shared vehicle service, on-call taxi service, and semi-autonomous driving experience. One touchpoint that cannot be ruled out from these three previous experiences is the link to the mobile phone. Customers who experience cutting-edge technology are all power users of mobile phones and shared vehicle services and on-call taxi services are based on mobile phone applications and location information. Most drivers carry mobile phones while driving, connect mobile phones with vehicles, and obtain information and content from them. This study aims at natural UI linkage between mobile phones and self-driving cars for superior customer experience.2. Research Methods The study includes three main steps; first, interview and observe iOS and Android users. Each user's observations detail the user's journey from vehicle entry to departure and organize the features. At this time, the main goal of the analysis is to separate only the user's journey, which is equivalent to an autonomous driving scenario, and to check which content subjects are mainly used by iOS' CarPlay services and Android Auto. Experiment with two major mobile phone platform users to avoid being biased toward a specific mobile phone platform.The second step is to experiment with fast and dirty prototyping techniques. The previous analysis defines and embodies the user's journey to the future automotive experience. The subjects experience this prototyping as it is and analyze the process. Intensive research has been conducted into the subject's mobile phone interworking process, but this part is not specified directly to the subject. This is to make sure that users naturally expect to connect to their mobile phones. It is analyzed what subjects who are familiar with the existing user experience of cars and mobile phones expect and expect from self-driving vehicles.The final stage is co-creation. Its goal is to create an interface that will satisfy customers regarding situations in which mobile phones and vehicle systems are linked by predicting their experiences. Assuming that the subject is connected to a terminal and a self-driving car, it writes the contents of the vehicle system directly. Based on the results of this process, we will derive a UI that will satisfy the users.3. Results The experiment was conducted on users who were familiar with driving cars and using mobile phones. To increase the likelihood of accepting future technologies, limit subjects to those familiar with existing experiences.The first test was able to analyze the experience of cars linked to mobile phones. Subjects naturally accept and use mobile phones as an integral part of their car-related experiences.In the second experiment, we were able to analyze the initial feelings of users related to autonomous vehicles. The subjects had a vague sense of uneasiness about cutting-edge technology that they had yet to experience in real life. Under these circumstances, they wanted to get additional information from their mobile phones and offset their anxiety.The third experiment resulted in the user experience that current drivers hoped for in future cars. In the context of the vehicle's software system interworking with the mobile phone, what factors users value was identified and the rules of customer experience focused on these factors were derived.4. Conclusions This paper aimed to make sure that users naturally expect mobile phone links in future car situations and to suggest optimized interfaces when linking mobile phones. This is not just a definition of a machine-to-person interface, but a basic study to design the customer experience in terms of the future automotive service industry.Subjects were found to expect future services based on similar existing experiences, and anxiety about the technology of autonomous driving was also found in common. On the service side, subjects reacted sensitively to the processing of personal information in shared vehicle situations. The study has an academic implication that it has drawn a major direction in how mobile communication devices, the most cutting-edge technology for current users, should be linked to self-driving vehicles, the most advanced technology in the future.

INTRODUCTION

Research Background Following the advancement of the technology itself, the autonomous driv- ing vehicle is spurring various fields such as infrastructure, legislation, and design. Since launching the autonomous vehicle service to the public, cus- tomer experience has also emerged as essential for the services. Beyond simply considering the human-machine interface (HMI) between the user and the vehicle, this study includes another element that cannot be ruled out from the user journey. User journeys for autonomous-driving-based taxi services are shown in Fig. 1 (Kim et al., 2020). This study indicated that mobile phone communicates with the vehicle on behalf of the user over several stages.

A study on user experiences (UX) of autonomous vehicles pointed out that it is crucial to enhance user’s trust at the current moment when autonomous driving technology is being introduced (Lee & Lee, 2020). And then, it also drew eight touchpoints to increase trust, and mobile phone and mobile appli- cations are included. A study about vehicle information systems indicates that; A growing trend is accessing portable smartphone-based systems (e.g.,

© 2023. Published by AHFE Open Access. All rights reserved. 378

UX Research for Autonomous Vehicles: Focusing on Interaction With Mobile 379

CarPlay and Android Auto). These systems support an expansion of various in-vehicle infotainment system features and functions. (David et al., 2019)

Figure 1: User journey map and issue points of autonomous taxi.
Original PDF Figure 1:

The enhanced interworking of mobile phones and vehicles in future auto- motive services is a customer experience that can be predicted. In this study, we derive touchpoints that interact with mobile phones when using autonomous vehicles (AVs). Subsequently, essential directions are established to provide an excellent customer experience at each touchpoint.

Research Goal This study analyses UX related to an autonomous vehicle by limiting the scope to case users move by autonomous driving taxi services.

Each stage of this study t has the following specific objectives:

1. Through user observations, researchers predict the use scenarios of mobile phones in future autonomous taxi services. 2. Researchers implement the low-fidelity prototype that corresponds to the scenario based on the previous stage. 3. In a co-creation workshop, researchers simulate the autonomous taxi service with participants, then discuss them in various aspects.

RESEARCH METHODS Research Process

This study involves three main stages. First, researchers observe the drivers and interview them. From boarding to getting off the vehicle, researchers monitor the entire process and list what they do. Among these derived user journeys, researchers filter the processes involved in autonomous driving. Additionally, researchers define key customer touchpoints in these processes. The second step is to make a test environment with quick and dirty prototyp- ing techniques. Based on previous analysis, researchers construct a prototype that embodies the future automotive experience. The final step is co-creation. Its goal is to upgrade the UX of the prototype, which is made in 2nd step. The members of the co-creation workshop may mean experts in other fields or non-design experts (Jung & Kim, 2018). Through the workshop, researchers compensate for the shortcomings of prototypes and get insights to refine the UX.

380 Lee and Lee

Figure 2: Three-steps of the study process.
Original PDF Figure 2:

User Research The first step, the user research, is conducted with observation techniques and interviews of four drivers. When participants drive to work, the researcher observes and records all performances. It does not tell the participant the research item, mobile phone, in advance. It is to understand whether the mobile phone is interoperated or not naturally. In addition, iOS-based mobile phone users and Android-based mobile phone users are selected equally to avoid being biased toward specific mobile phone platforms.

Table 1. Participants’ driving condition.

Table available in the original PDF paper.

No. Vehicle model Commute time OS of phone Model of phone Connection 1 BMW Mini S 2 Min. Android Galaxy S20+ Not connected 2 Hyundai Grandeur 47 Min. Android Galaxy S10 Wired 3 BMW M340i 30 Min. iOS iPhone 11 Pro Max Wireless 4 Genesis GV70 26 Min. iOS iPhone XR Wired

Shadowing in User Research: As a UX study technique, shadowing is a qualitative research method that observes a subject’s behavior without inter- vention. UX professional says, “Asking users to adopt new behaviors or even modify existing behaviors is very, very hard.” Shadowing lets researchers understand existing behaviors so that they can adopt new designs to those behaviors. (Interaction design foundation, 2021) Therefore, this research is consistent with the purpose of this study to optimize the experience of customers familiar with traditional taxi services and vehicle sharing services when they use an autonomous driving taxi. After user shadowing, researchers interview participants and define user journey, which is related to mobile phones.

Create Prototypes for a Co-Creation Workshop The Wizard of Oz (WOZ): The second step is quick and dirty prototyping. In the previous step, we defined points where a mobile phone is used in future taxi services. We build prototypes of an autonomous driving taxi service based on them. The laboratory setting was designed following the Wizard of Oz (WOZ) prototyping method. WOZ is a methodology in which one or more ‘wizards’ simulate part or all of the system while interacting with users who think they are using a real system (Bernsen et al., 1994).

UX Research for Autonomous Vehicles: Focusing on Interaction With Mobile 381

A Co-creation Design: Co-creation refers to any act of collective creativity, i.e., creativity that two or more people share. Experts can participate in this co-workshop for user-centered design with an ‘expert perspective’ approach (Sanders & Stappers, 2008). In this study, a co-creation workshop is con- ducted by participants with work experience in autonomous vehicles. This process makes up for the weaknesses of limited prototyping. Furthermore, through this workshop, we derive critical insights in terms of UX and service design.

The four subjects who participated in the previous user shadowing phase equally participate in this workshop. They directly provided the basis for designing mobile phone usage on a vehicle. So then, they can evaluate and supplement prototypes directly. As experts, so that they could present com- plementary points, developers and designers working for mobile software (SW) were selected. The additional participants are three professionals in the field of the digital cockpit and autonomous vehicles. With one researcher conducting and participating, a total of eight people participated in the co-design.

Table 2. Profile of participants in co-creation workshop.

Table available in the original PDF paper.

No. Category Field of job Position Company Career Mobile device UX designer Electronics company 7 Years

1 Who participated in user shadowing

11 Years

Project manager Automotive AI S/W company

2 Mobile device SW engineer Electronics company 9 Years 3 Mobile device SW engineer Electronics company 13 Years 4 E-Book platform UX designer E-Book service 15 Years 5 Professionals in the future automotive field

Autonomous Driving Simulation S/W UX designer Electronics company 7 Years

6 Digital Cockpit Solution

UX designer Electronics company 8 Years

7 Digital Cockpit Solution

8 The researcher Mobile device Product planner Electronics company 16 Years

RESULTS

Analyzation of User Shadowing and Interview Researchers observed four participants going to work in the morning with- out interference. And then, they recorded the behavior of using a mobile phone while driving. After riding, researchers interviewed about unusual things during observation.

Group of Android users: Both were users of Samsung’s Galaxy brand. The first participant. She is using a mobile phone, Galaxy S20+, and her travel time is 12 minutes. She does not connect her mobile phone to the vehi- cle, but she turns on the mobile application and checks the map information. Her user journey is as follows:

1. Bring the key from the usual place (near the front door). 2. Move to the vehicle.

382 Lee and Lee

3. Press the button on the key fob to unlock the vehicle. 4. Open the door of the vehicle. 5. Get in the vehicle. 6. Change right foot shoe to driving shoe which is in the vehicle. 7. Turn on the engine. 8. Turn on the headlight. 9. Start driving and move the car. 10. Fasten the seat belt where the vehicle stops for the first time. 11. Run the map application on the mobile phone (TMAP). 12. Arrival 13. Parking 14. Turn off the engine. 15. Get off the vehicle. 16. Close the door of the vehicle. 17. Lock the vehicle by pressing the key fob button.

What is unusual is that the phone’s map application is launched, but the destination is not set. She does not check her mobile phone while driving. When asked why she turned on the map application even if she knew the route well, she replied, “To check the speed section or road information.”

The second participant: The second man uses the Galaxy S10 as his mobile phone and travels 47 minutes. After boarding the vehicle, the phone is wired directly to the vehicle. While driving, he runs Android Auto, a vehicle-linked software provided by Android OS, and then uses the mobile phone’s map and music applications. His user journey is as follows:

1. Move to the vehicle. 2. Take out the key fob from the bag. 3. Press the button on the key fob to unlock the vehicle. 4. Open the door of the vehicle. 5. Get in the vehicle. 6. Turn on the engine. 7. Plug the cable connected to the vehicle into the mobile phone. 8. When a mobile phone is connected, the icon of Android Auto on the vehicle’s head unit is activated. The driver runs the enabled Android Auto. 9. When the driver runs the Android Auto app, the navigation app is already running. At this time, select the “Latest destination” menu, and set the destination. 10. Fasten the seat belt. 11. Start driving and move the car. 12. The driver runs a music application inside the Android Auto through the vehicle’s head unit’s user interface (UI) and then plays music (Spotify). 13. Arrival 14. Parking 15. Turn off the engine. 16. Get off the vehicle. 17. Lock the vehicle by pressing the key fob button.

UX Research for Autonomous Vehicles: Focusing on Interaction With Mobile 383

What is uncommon is that an additional HUD is installed and used. He said that navigation information is acquired through HUD when driving and that the UI of the head unit is mainly used to set the location or control multimedia.

Group of iOS users: Two iPhone users used CarPlay when their phones were connected to the vehicle.

The third participant: His phone is the iPhone 11 Pro Max, and he drives to work for 30 minutes. The vehicle and the mobile phone are automati- cally connected wirelessly after the driver is on board. The user journey is as follows:

1. Move to the vehicle. 2. Take the mobile phone to the handle, and then the vehicle door is to be unlocked. 3. Open the door. 4. Get in the vehicle. 5. Fasten the seat belt. 6. Place the phone in the smartphone tray of the vehicle. 7. Press the start/stop button to turn on the engine. 8. Tap the menu of CarPlay on the vehicle’s head unit. 9. Run the map application in CarPlay of the head unit (TMAP). 10. Select the “Latest destination” menu, set the destination. 11. The driver runs a music application in CarPlay of the head unit and plays music (Apple Music). 12. Start driving and move the car. 13. Arrival 14. Parking 15. Turn off the engine by pressing the start/stop button. 16. Get off the vehicle. 17. Turn off the engine. 18. Take the phone to the handle, and then the door is to be locked.

Most notably, the use of mobile digital keys. For some BMW car models, users can store digitized car keys on their iPhones and perform the same functions with a key fob on their mobile phones like lock/unlock and engine start/stop. At this time, wireless communication between a vehicle and a mobile phone is via NFC.

The fourth participant: She uses an iPhone XR and travels for 26 minutes in a Genesis-branded GV70. As shown in Fig. 3, the mobile phone application is used via the vehicle’s head unit. The user journey is as follows:

1. Move to the vehicle. 2. Take out the key fob from the bag. 3. Press the button on the key fob to unlock the vehicle. 4. Open the door of the vehicle. 5. Get in the vehicle. 6. Turn on the engine. 7. Plug the cable connected to the vehicle into the mobile phone. 8. Tap the menu of CarPlay on the vehicle’s head unit.

384 Lee and Lee

9. Run the map application in CarPlay of the head unit (TMAP). 10. Select the “Latest destination” menu, set the destination. 11. Fasten the seat belt. 12. Start driving and move the car. 13. The driver runs a music application in CarPlay of the head unit and plays music (Melon Music). 14. Arrival 15. Parking 16. Turn off the engine. 17. Get off the vehicle. 18. Lock the vehicle by pressing the key fob button.

She answered that she does not prefer the navigation SW provided in the vehi- cle, so by using CarPlay, her preferred navigation application can be available through the vehicle’s head unit.

The interoperable points between a mobile phone and a vehicle are a key, comprehensive SW provided by mobile operating systems and mobile appli- cations that run in the system. Focusing on these elements, we define the role of mobile phones in autonomous taxi services in the following process.

Figure 3: User shadowing of participant 4 - she connects the mobile phone and the vehicle wired, runs carPlay on the vehicle’s head unit for navigation and music apps.
Original PDF Figure 3:

Create Prototypes for a Co-Creation Workshop The quick and dirty prototype is prepared to assume an autonomous driving taxi service. The first preparation is a taxi service concept vehicle, and the second preparation is an application for this taxi service. Participants down- load a mock-up of the application to their mobile phones. Assuming that the vehicle operates unmanned, a task scenario is designed. According to the scenario, the researcher acts as a wizard and performs interactions between participants, mobile applications, and vehicles.

We define the autonomous driving-based taxi service user journey as shown in Fig. 4 and then prototype a mobile application corresponding to tasks A∼E. The mock-up of the app consists of 6 mobile app screens,

UX Research for Autonomous Vehicles: Focusing on Interaction With Mobile 385

like Fig. 5. The transaction step for unlocking the car before boarding and locking the car after getting off requires user authentication. IOS and Android have different biometric authentication methods, providing different mock- up screens to each group. The wizard guides seven participants to simulate this taxi service using the prototype on their mobile phones. Participants use mobile mock-ups to board, move, and get out of the vehicle. All participants experienced every step of the prototype and then proposed a scenario for each journey in the co-creation workshop. They designed the improvements of the mobile application for themselves and proposed new functions for this service.

Figure 4: User journey for prototyping and test.
Original PDF Figure 4:
Figure 5: Six app screen mock-ups corresponding to step A∼E.
Original PDF Figure 5:

Matching between Passenger and Car: The process of first boarding needs to be examined more carefully. The vehicle must track and follow the location of the passenger, which must comply with traffic regulations. It is also part of the design of seamless mutual recognition between the vehicle and the pas- senger. The technology to look at here is the Ultra-Wideband (UWB). UWB is a radio technology that transmits large amounts of information over a wide range of bandwidths. It has the advantage of being able to accurately mea- sure the distance and location between devices, which has drawn attention again along with the Internet of Things (IoT) service. According to a study in 2021, the chips that implement UWB standards are already deployed in a

386 Lee and Lee

recent high-end model of mobile phones and the advanced automotive indus- try (Singh et al., 2021). In other words, vehicles can measure sophisticated locations and provide corresponding interactions after the passenger’s mobile phone enters the range.

User authentication when boarding: ID certification for passengers is a critical security issue because it is an unmanned system. This role can be performed primarily by mobile phones, and it is also linked to the role of a mobile digital key. The digitized car key is installed on a mobile phone. When attempting to unlock the car, it requires user authentication provided by the mobile phone. Secondary certification can be carried out directly through the vehicle. However, since the service provider cannot hold authentication information for all customers, indirectly, the car verifies that it matches the phone owner. Biometric data can be used for additional functions to check the conditions suitable for driving, such as whether drunk or not.

Digital Key: Internalization of car keys into mobile phones is standardized through the Car Connectivity Consortium (CCC). This standardization indi- cates that users can share their car keys remotely on mobile cross-platforms like iOS and Android. In April 2021, CCC announced the Digital Key Release 3.0 specification, including Passive Keyless Entry and Engine Start using UWB (The Car Connectivity Consortium®, 2021). This announcement means that if both vehicle and phone are equipped with a UWB chipset, the car will be automatically unlocked upon close access by the user.

Interworking with Mobile Phone: All the experts who participated pre- dicted that the interoperability of mobile phones and vehicles would be more diverse than it is today. All seven also expected the phone’s screen to be mirrored through the vehicle’s large display. Five of them preferred that the information obtained from the vehicle and the phone’s application be synced. The idea of receiving road conditions through drones was also presented.

Consideration to multi-passengers: UWB is also a notable factor for this agenda. A study by Zhang D. shows that the vehicle can calculate the sophis- ticated location of the mobile phone, thus identifying which seat the owner of the mobile is in Zhang D., 2021. In other words, it is possible to choose which of the many passengers to connect their mobile phones and vehicles. Alter- natively, it is possible to design a structure that mirrors multiple terminals on multiple displays in vehicles.

Service experience connecting before and after boarding: The idea that the vehicle’s interior should be streamed through mobile phones to check cleanliness and safety before boarding was also derived. The payment result and management of data (e.g., personal boarding records) should verify the results on a mobile phone. The link between users and service providers for lost and found management is also the mobile phone.

CONCLUSION

This study verified that mobile phones are an indispensable factor in autonomous vehicles and will be linked in an advanced method for increas- ingly diverse functions. When studying UX for future technologies, WOZ approach was able to complement the limitations of the test environment.

UX Research for Autonomous Vehicles: Focusing on Interaction With Mobile 387

Through a co-creation workshop between engineers and UX designers in the mobile phone and automotive fields, we predicted user scenarios of autonomous taxi services from various aspects and derived optimized UX. One limitation of the study is the prediction of user accounts. Many services experienced by this service collide with mobile platforms and vehicle plat- forms. This is a common issue for various stakeholders, including vehicle manufacturers, mobile platforms, service providers, and users.

REFERENCES

Bernsen, N. O., Dybkjær, H. & Dybkjær, L. (1994). Wizard of oz prototyping: How and when. Proc. CCI Working Papers Cognit. Sci./HCI, Roskilde, Denmark. Car Connectivity Consortium. (2021/4/21). Digital Key Release 3.0 Specifica-

tion. https://carconnectivity.org/press-release/car-connectivity-consortium-delive rs-digital-key-release-3-0-specification David L. Strayer, Joel M. Cooper, Madeleine M. McCarty, Douglas J. Getty, Camille L. Wheatley, Conner J. Motzkus, Rachel M. Goethe, Francesco Biondi, William J. Horrey. (2019). Visual and Cognitive Demands of CarPlay, Android Auto, and Five Native Infotainment Systems, Vol. 61 (pp. 1371–1386). https://doi.org/10.1177/0018720819836575 Jung, Y. & Kim, S. (2018). Study on the Place Branding from the Co-design point

of view -Based on Abandoned Railroad Urban Park. Journal of Communication Design, Vol. 64 (pp. 141–154). Kim, S., Chang, J. J. E., Park, H. H., Song, S. U., Cha, C. B., Kim, J. W., & Kang, N.

(2020). Autonomous Taxi Service Design and User Experience. International Jour- nal of Human–Computer Interaction, 36(5), 429–448. https://doi.org/10.1080/ 10447318.2019.1653556. Lee, M. & Lee, Y. (2020). UI Proposal for Shared Autonomous Vehicles: Focusing on Improving User’s Trust. In H. Krömker (Ed.), HCI in Mobility, Trans- port, and Automotive Systems. Driving Behavior, Urban and Smart Mobility (pp. 282–296). Springer International Publishing. https://doi.org/10.1007/978-3- 030-50537-0_21. Sanders, E. B. N. & Stappers, P. J. (2008). Co-creation and the new landscapes of

design. Co-design, 4(1), 5–18. Shadowing in User Research - Do You See What They See? | Interaction design foundation. https://www.interaction-design.org/literature/article/shadowing-in-user- research-do-you-see-what-they-see, last accessed 2021/05/30. Singh, M., Roeschlin, M., Zalzala, E., Leu, P. & ˇCapkun, S. (2021, June). Secu-

rity analysis of IEEE 802.15. 4z/HRP UWB time-of-flight distance measurement. In Proceedings of the 14th ACM Conference on Security and Privacy in Wireless and Mobile Networks (pp. 227–237). Zhang, D. (2021). Ultra-Wideband Ranging for In-Vehicle Smartphone Positioning (Master’s thesis, Schulich School of Engineering).

Cover of the proceedings volume Application of Emerging Technologies

Explore the complete digital volume

Application of Emerging Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-91-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →