Xiaohua Sun, Jinglu Li, Weiwei Guo (2023). A design approach of proactive HMI based on smart interaction. In: Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems. AHFE Open Access, vol. 69, pp. 77–86. https://doi.org/10.54941/ahfe1002823
As AI advances, intelligent systems are gaining the ability to collaborate with humans to accomplish everyday tasks proactively. In proactive HMI design, the accuracy of the user intention prediction model in the mechanism becomes the key to affecting the quality of the proactive HMI experience. However, there are three issues that caused the lack of effective ways to improve the prediction accuracy of user prediction models. In this paper, we analyze the Information for improving user prediction accuracy, the Intervention stage, and the required contents for smart interaction. Then, we develop an approach of the proactive HMI based on smart interaction, which is the method that robots learn from the users through interactions. We propose the elements, the framework, and the guidelines. This paper also provides how to use this approach in design case. With this approach, the accuracy of user intention prediction of proactive HMI can be improved and then can be achieved the goal of improving the design effect and the user experience of proactive HMI can be achieved.
Compared with traditional “passive”human-computer interaction, proactive interaction is initiated by the system after predicting the user’s intention (Nothdurft et al., 2015). In the automotive area, some researchers used a pro- active recommendation system as a method to solve diver’s problems, such as distraction (Bader, 2013) and the information overload (Bader et al. 2011). However, due to the limitations of the vehicle proactive human-machine inte- raction (HMI) prediction model and rules, the results of the prediction model are often biased, which affects the results of predicting user intentions and may lead to failed interaction process and poor user experience. Vehicle proa- ctive HMI requires personalized intention prediction for different users. Still, it is often difficult to establish a prediction model applicable to users from the very beginning, so the user prediction model should have the ability to improve the human-computer interaction process (Senft et al., 2017; Cakmak et al., 2010). First, this paper summarized the mechanism and issues of pro- active HMI. Second, provided the smart interaction gets into proactive HMI stages and required contents. Then, offered an approach including elements and a design framework to improve the user experience in proactive HMI by smart interactions to obtain the valid information needed to improve the
accuracy of predictions. Last, provided how to use this approach in design cases.
PROACTIVE HMI Proactive HMI means that the vehicle system can timely perceive the status and changes of the user and the environment, accurately predict the user’s intention and behavior, and take the initiative to respond to the user’s needs. Its essence is to embed the computing, communication, and perception capa- bilities into the vehicle so that the interaction initiated by the vehicle results from data collection, analysis and processing, and decision-making instead of relying entirely on the user’s direct input.
Mechanism of Proactive HMI As shown in Figure 1, the core part of this mechanism is how to construct and learn about the models of user behavior and intention prediction by AI engine. To achieve this target, there are two stages: offline learning and online prediction.
Issues of Proactive HMI Constructing prediction models and rules requires a large amount of diverse domain knowledge and data. Currently, there are mainly the following problems:
First, the initial training samples of the model have limitations, which are difficult to cover all actual objective conditions, such as missing sample data, a small amount of data, or incorrect data. Second, it is difficult to accumulate the personalized data required by the prediction model. Last, Since the driving situation is complex and changeable, errors will get if the prediction rules and models are fixed, which cannot be constantly adjusted to the changing environment. The prediction performance will be poor or even ineffective.
INFORMATION FOR IMPROVING USER PREDICTION ACCURACY
1. User habits and behavioral preference information. User habits are a routine of behavior repeated regularly in the car or dri- ving process, including using the in-car media, driving, and travel-related
A Design Approach of Proactive HMI Based on Smart Interaction 79
behaviors. This information is related to establishing the user knowledge graph model and is often directly related to the prediction rules or models. Behavioral preference refers to the user’s tendency when it comes to judgment and decision, with uncertainty, such as taste preference, con- sumption preference, brand preference, etc. 2. Contextual information. It refers to the performance of the current situation, including the user’s current context and state, as well as the context of the environ- ment. Real-time data and important interrelated information are needed to supplement the environment, vehicle, and human knowledge graph model. 3. Feedback data from users. The user feedback on the system results and supplements the missing information. The annotation information from users.
INTERVENTION STAGE AND REQUIRED CONTENTS Smart interaction is flexible and aims to obtain information to improve the accuracy of system prediction. Therefore it can be involved in every stage of Proactive HMI. In different stages of Proactive HMI, the purpose, the target information, and the way to get information are all different, as shown in Table 1.
APPROACH OF SMART INTERACTION IN PROACTIVE HMI DESIGN Smart interaction is the interaction process between humans and intelligent systems (such as a social robot or intelligent vehicle active system), which is designed to improve the performance of intelligent systems by obtaining the knowledge required by intelligent systems. This is an effective means to enhance robot performance in the intelligent robots field.
Smart interaction is a way of active learning (Taylor et al., 2021). This process refers to learning from users in an interactive, natural, and low- interference way (Gonzalez-Pacheco et al., 2018) during the interaction between humans and robots. By accumulating critical information needed for model optimization and improvement, the model and its parameters can be appropriately modified to support the model’s continuous learning and modification and realize the system’s progress and evolution.
To improve the ability of model prediction and reasoning decision-making of proactive HMI in intelligent vehicles, a smart interaction module can be added to obtain critical information naturally needed by the system through direct interaction with users. In this way, the current problems of proa- ctive HMI technology mentioned above can be solved, such as low accuracy of model prediction and weak environmental adaptability, to optimize the prediction results of the prediction model.
As Figure 2 shows, the framework of proactive HMI based on smart inte- raction is composed of three grey areas numbered 1-3: data input, user’s intent prediction model, and initiate action. The design method of proactive
Table 1. The content comparison of smart interaction is intervened in stages of proactive HMI.
Table available in the original PDF paper.
Stage Purpose Target info How to get
whether to initiate
Cause Confirmation
Context information
Before initiation
The preference
what’s the good time to initiate
Confirmation of execution
Keyword selection
The emotions of users
what’s the good content to initiate
User status Confirmation
The urgency of tasks
The changes of user behavior
Context information confirmation
Context information
Confirmation of execution
The preference
Initiation
Whether the initiation point is appropriate
Confirmation of preference
The user decision model
User input
User type learning
Whether the initiation way is appropriate
Personal information
Update the information in real-time
Supplement the missing content of proactive interaction Confirmation
Whether the proactive HMI content align the user’s intention
Learn user preferences to next time serve
Get users’ comments on the content of this proactive interaction
Cause Confirmation
User usage Habits
Follow-up behaviors
Confirmation of execution
User personalized selection
Access to effective information to optimize the latter proactive HMI
Keyword selection
Cause of user
Learn the preference
User type learning
Behavior
User input
The user decision model
Confirmation of contents
User emotion
HMI based on smart interaction is to introduce smart interaction in speci- fic scenarios to obtain the information needed to improve the accuracy of the proactive interaction prediction model. There are two ways to realize smart interaction between human and vehicle systems: feedback and asking for learning.
A Design Approach of Proactive HMI Based on Smart Interaction 81
Figure 2: The framework of proactive HMI based on smart interaction.
Feedback, that is, obtaining the user’s input on the forecast results, and the system modifies the model according to the feedback results, such as telling the machine whether it is right or wrong or choosing the result from the multiple options offered by the model prediction. Asking for learning, that is, learning by asking the user. Through self-assessment, the machine judges the missing, fuzzy, and wrong information, takes this as the learning objective to ask the user and collects relevant information to learn the model.
The design method of active interaction based on smart interaction inclu- des the factors and design framework of smart interaction.
Elements of Smart Interaction Design Three elements need to be designed in smart interaction: the content of inte- raction(what to interact), the way of interaction(how to interact), and the point of occurrence(when to interact).
1. The content of the interaction It refers to what information and actions will happen in smart interaction design. To obtain the system’s missing, fuzzy, and error information and get the user’s feedback and evaluation of the system prediction results. For example, in smart interaction design to obtain information about a user’s preferences for news types, the content asks about the most exci- ting news content of the day. In the smart interaction design, to obtain whether the predictive reminder is correct, its content is the user’s feed- back after comparing the actual situation with the predictive reminder result. 2. The way of interaction It refers to selecting the appropriate forms of appearance and interaction ways according to what influence will have in the current situation from the interaction content. It can be chosen according to the medium in the multi-modal environment in the car. For example, when asking users about their favorite types of news during the day, we can choose their leisure time in the car after the day’s trip and through voice interaction
Figure 3: The design framework of Smart Interaction.
and screen interaction. If the user’s behavior changes compared to the usual, the system interacts with the user by voice in the scene where the change occurs and in the idle state where driving is safe. 3. The time point of interaction occurrence It refers to the context and time point after the trigger condition. For example, the smart interaction design to obtain the user’s news type pre- ference information appears when the user has used relevant services on the same day and initiates smart interaction when the user is in a safe, private state after the trip.
The Design Framework of Smart Interaction Design The proactive HMI design framework based on smart interaction (as shown in Figure 3) can be used in specific situations.
In the upper solid line frame of the figure is the proactive HMI part, which may be a type of proactive HMI function or a fragment of pro- active HMI. Based on the initiated content of the proactive HMI, we can infer what information is needed to improve its prediction accuracy. To obtain this information, we should design the corresponding smart interaction.
A Design Approach of Proactive HMI Based on Smart Interaction 83
The use process includes: First, define the purpose of smart interaction design. The purpose of smart interaction design is determined according to the information needed for the proactive HMI to improve the accuracy of prediction, including the missing target information, the information on the threshold to be defined, and the information on user feedback error.
Then, finish the smart interaction canvas. (1) It is necessary to clarify the stage of smart interaction involved in proactive HMI. The intervention sta- ges of smart interaction are determined to identify the types of information to be concerned with and the practical means to obtain it. (2) a complete smart interaction should include the way, content design, and time point of interaction occurrence to achieve the purpose of this learning process.
Finally, self-check whether the purpose of smart interaction can be achieved.
The Guideline of Smart Interaction
Low disturbance. The task of smart interaction can be decomposed into multiple situations and states. The initiation is highly relevant to the cur- rent situation, reducing the burden of a single task and preventing users from experiencing too many interaction processes at a time.
The helper humanoid metaphor. Smart interaction design should be done in a humanlike and polite way to get close to the communication logic between people naturally. In the specific scenes related to efficiency, it conforms to the helper humanoid metaphor, whose feature positioning is to provide convenient and efficient related services.
The voice interface design in smart interaction should be highly relevant to the context. Context is continuously tracked with critical corpora to make it easier for the user to understand the content of the dialogue so that the interaction with the system can be more efficiently understood and processed.
DESIGN CASE In the design case, we choose the scenario when the electronic vehicle with a low battery provides appropriate charging suggestions and effectively alleviates users’ range anxiety.
To achieve the above goals, smart interactive intervention learning is nee- ded, including the threshold of the emergency level of the car battery and the user’s personal preferences for charging, such as the brand and charging station. As figure 4 shows, two kinds of information will improve predi- ction accuracy: the preference for range anxiety (low battery anxiety) and the preference for charging, including brand, location, price, etc.
There are two phases in the smart interaction design to obtain the user’s range anxiety preference. In the first phase: since there is no knowledge graph about preferences, the system must be aware of users’ behaviors, collect as much data as possible, and verify and mark the data through intelligent interactive design after data collection. The dialogue design:
Figure 4: The case design framework of the low battery anxiety.
Vehicle: “Hi, are you worried about the battery level?”
User: “Hmm…” Vehicle: “The current battery can drive 110km. Are you worried about reaching your destination?”
Suppose the user answers “Yes” or some other affirmative answer. (If the answer is no. Then design to defuse embarrassment) Vehicle: “According to the current traffic conditions, you can reach the destination.”
The user’s expression is relaxed, and anxiety is relieved. Through this dia- logue, the intelligent system can know that the user will have anxiety when
A Design Approach of Proactive HMI Based on Smart Interaction 85
Figure 5: The case design of proactive HMI after smart interaction learning.
the battery is around 110km. When the system provides information about reaching the destination, the user’s anxiety will be relieved.
In the second phase: the system learned some of the user’s behavior. If the user has a private charging port, the battery level threshold is in the charging post each time. In addition, each time the user charges outside, the battery level threshold will be confirmed by smart interaction before the user reaches the point again. As shown in figure 5.
Vehicle: “Hi, it’s a good time to charge.”
If the current trip is urgent, the vehicle will calculate whether it can reach the destination according to the line traffic conditions and then recommend the most suitable charging scheme according to whether there is time pressure.
There are also two phases in the smart interaction design to obtain the user’s preference for charging services.
The first phase is when the system is unfamiliar to the user. Smart inte- raction appears in the finishing charging, asking for the service feelings and feedback.
The second phase is when the system is familiar to the user. Users often charge at fixed-brand charging stations. When the same behavior happens again, ask the user why. If the current user behavior differs from the usual behavior, the intelligent system confirms the reason to the user. One day the user went to another brand charging pile to charge and got in the car to ask the user why.
CONCLUSION
Whether the predictive model accurately predicts the user’s intention deter- mines the quality of the experience of the proactive HMI. Introducing smart interaction as a design approach to the proactive HMI can effectively solve this problem.
According to the proactive HMI design method based on smart interaction, the information needed to improve accuracy is taken as the design goal. fol- lowing the framework of this approach, the content, the way, and the time point can be appropriately designed. This approach can help future design
researchers to design a proactive HMI scheme with high prediction accuracy and better user experience.
REFERENCES
Bader, R., n.d. Proactive Recommender Systems in Automotive Scenarios 348. Bader, R., Woerndl, W., Prinz, V., n.d. Situation Awareness for Proactive In-Car Recommendations of Points-of-Interest (POI) 10. Cakmak, M., Chao, C., Thomaz, A.L., 2010. Designing interactions for robot active
learners. IEEE Transactions on Autonomous Mental Development 2, 108–118. Gonzalez-Pacheco, V., Malfaz, M., Castro-Gonzalez, A., Castillo, J.C., Alonso, F., Salichs, M.A., 2018. Analyzing the impact of different feature queries in active learning for social robots. International Journal of Social Robotics 10, 251–264. Nothdurft, F., Ultes, S., Minker, W., n.d. Finding Appropriate Interaction Strategies
for Proactive Dialogue Systems—An Open Quest 8. Senft, E., Lemaignan, S., Baxter, P.E., Belpaeme, T., 2017. Leveraging Human Inputs in Interactive Machine Learning for Human Robot Interaction, in: Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human- Robot Interaction, HRI ‘17. Association for Computing Machinery, New York, NY, USA, pp. 281–282. https://doi.org/10.1145/3029798.3038385 Taylor, A.T., Berrueta, T.A., Murphey, T.D., 2021. Active Learning in Robotics: A Review of Control Principles. Mechatronics 77, 102576. https://doi.org/10.1016/j.mechatronics.2021.102576
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-45-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional