To investigate the effect of a multimodal interaction balance model on improving the emotional driving experience when users perform a high load driving task in a negative driving emotion context. The questionnaire analysis was used to obtain the main negative emotions and the corresponding driving situations. Combined with the STAR interview method to understand the user's interaction task in specific contexts, the user's cognitive load was assessed by the SWAT subjective load assessment technique to obtain a high cognitive load task, and the VACP model was used to establish a balanced model of interaction task and interaction modality. Simulated multimodal interaction physiological experiments were conducted to analyze the impact of the multimodal balance model on participants' physiological data when they performed high cognitive load tasks with different emotions, and the physiological data were analyzed to assess participants' emotional experience. The emotional experience design of an intelligent vehicle robot is used as an example to validate the method. The results show that the multimodal interaction balance model can effectively reduce the user's cognitive load and improve the pleasantness of the interaction experience, and find a breakthrough for the development of intelligent vehicle-mounted robots in emotional experience.
With the development of intelligent cars, the interaction between humans and vehicles has increased and become more diversified. The interaction logic and user experience in driving scenarios deserve serious consideration, and it is an important proposition to realize efficient, safe, and natural human-computer interaction and improve users’ emotional experience. In the face of complex tasks and diverse interactions, driver cognitive overload is inevitable, and cognitive overload can easily lead to driver mood swings, which can lead to some dangerous driving behaviors if drivers drive with negative emoti- ons, affecting user driving safety and driving pleasure. Although China has adopted strict legal regulations to control drivers’ dangerous driving beh- avior in its driving system, there is a lack of quantitative identification of drivers’ dangerous driving control, especially the driving problems caused by
A Multimodal Interaction Experience Design Approach 741
negative emotions, which are somewhat complex and ambiguous, and driving emotion elimination has become a popular research topic. Therefore, in the face of complex and diverse driving situations and drivers’ negative emoti- ons, how to improve the impact of drivers’ negative emotions during driving through more natural interaction will be the development direction of future innovation of smart car products.
Emotional experience is the core of user experience. Users obtain their attitudes and evaluations of products from their subjective experience of the product interaction process and form motivations to influence their cognition and behavior. Regarding research on affective experience, PrEmo affective measure can be used to measure emotions induced by various products and other stimuli to assess the impact of products on emotions (Desmet P M A, 2004). Dormann C conducts affective assessment of human-computer intera- ction from a consumer behavior domain approach, emphasizing the importa- nce of users’ affective characteristics (Dormann C, 2003). In terms of research on the influence of emotions on driving, the analysis of influencing factors for negative emotional driving work of bus drivers quantified the relationship between the influencing factors and the degree of influence on driver work through structural equation modeling (Qiusheng Tang et al, 2021). The dri- ving load state was evaluated by collecting driving physiological data, and the results showed that considering initial emotions can effectively eliminate the influence of driver load in the resting state (Jing Huang and Mengting Yang, 2021). In terms of emotional experience evaluation, objective data were provided for further understanding of user experience by showing specific differences in cognitive involvement and emotional response changes during user interaction (Lee S et al, 2020) A set of user affective experience asses- sment methods based on the PAD affective model is provided for the need of fast, accurate and highly adaptable user affective experience assessment in the product design process (Ni Jiang, 2021). Na Lin analyzed the positive and negative effects of stimuli on emotional responses in design by studying the emotional factors in product design and applying them to the process of emotional experience design (Na Lin, 2011).
Multimodal interaction is an important means of achieving natural inte- raction between humans and the intelligent body, which directly affects the emotional experience of users. Regarding research on multimodal interaction design, Naumann A B et al found that multimodal interfaces with touch, voice, and motion control outperformed unimodal interactions in terms of efficiency, robustness, and user satisfaction (Naumann A B et al, 2010). Takashi Yamauchi et al investigated the enhancement of people’s emotional experience through visualization of haptics by interacting The case of plants shows that haptic interfaces facilitate the generation of emotions (Takashi Yamauchi et al, 2018) A multimodal human-computer intelligent interaction method for smart home device control was proposed to achieve efficient and accurate contactless smart home human-computer interaction (Gangli Shao et al, 2021) Lu Weihua et al proposed a multimodal human-machine data-driven service design method to solve the problem of poor civil aviation check-in experience (Weihua Lu et al, 2022). With the development of arti- ficial intelligence technology and the continuous optimization of in-vehicle
742 Wang and Wei
Figure 1: Research technical framework.
interaction systems, drivers are gradually shifting from facing traditional sin- gle driving tasks to facing multiple parallel task situations as well. Therefore, it is especially critical to screen the elements of poor user experience caused by cognitive load and evaluate the impact of cognitive load on user experience.
To address the problem of cognitive overload during driving, the cognitive resource occupation of drivers can be reduced through natural interaction. Based on the above background, the article focuses on the impact of multi- modal interaction channels on the emotional experience of negative driving when users perform driving tasks in the context of negative driving emotions, so as to verify that the multimodal interaction balance model can improve negative driving emotions and enhance driving pleasure. The purpose is that when users can improve their cognitive ability in the face of complex situati- ons, emotions and interaction tasks in driving scenarios, they can reduce the cognitive load and enjoy the pleasure of the interaction process.
RESEARCH TECHNICAL FRAMEWORK The multimodal interaction experience study of negative emotion driving is divided into four main parts, and the specific process framework is shown in Figure 1.
(1) Through questionnaire survey and SPSS data analysis, we obtain the main negative driving emotions of users, summarize the reasons affecting users’ emotions and driving contextual tasks, and obtain contextual data of negative emotions. (2) Combined with STAR interview method and SWAT scale, the situational tasks were sliced and analyzed to obtain high cognitive load tasks. (3) Analyze the context-specific tasks according to the VACP standard scale and propose hypotheses for solutions using the multimodal balance model. (4) A multimodal interaction physiological experiment was constructed to analyze the effect of the multimodal balance model on the affective expe- rience of users performing contextual tasks under high cognitive load in a negative driving situation, and the participants’ affective experience was evaluated by comparing physiological data and subjective scales.
A Multimodal Interaction Experience Design Approach 743
METHODS
Task and Participants A questionnaire was designed to obtain a breakdown of the types of nega- tive emotions generated by users during driving and their causes, and the correlation between negative emotions and their causes was analyzed by SPSS. The significantly correlated negative emotions and their causes were used as the eliciting conditions for the physiological experiment to improve the authenticity and credibility of the experiment.
For this study, we screened young drivers in Xiamen, recommended sui- table candidates through the research company, and confirmed the final candidates based on their age, gender, income, and driving status. Next, for the selected respondents, we conducted one-on-one in-depth interviews. Finally, we confirmed that 20 respondents participated in this study, including 7 females and 13 males. During the interviews, STAR analysis was used to ask the participants to recall the experience of their best sensory experience in their daily life, so as to obtain the interaction modality category of the best user experience. Next, to understand how participants usually eliminate negative emotions, the interaction modality category of negative emotions was obtained. Finally, participants were asked to recall the process of their best driving experience and the process of their worst driving experience, so as to obtain the typical scenarios and interaction factors that affect driving emotions.
Experimental Design The analysis of SPSS shows that anxiety and anger are the main negative emotions of driving. Anger is mainly influenced by the driving situation out- side the car and environmental reasons, being forced by the car on the left and right and driving too slow are positively correlated with anger. Anxiety was mainly influenced by environmental reasons and road conditions. Com- muting to and from work and being affected by bad weather were positively correlated with anxiety. An experiment with 1 cognitive load (high cogni- tive load) * 2 negative emotions (anxiety, anger) was designed to analyze the participants’ physiological data and subjective scale evaluations during driving.
We used E-Prime psychology experimental manipulation software to pre- sent the driving scenario and time control of the experiment. Physiological polysomnography was used to record participants as they performed diffe- rent cognitive load tasks, provide multimodal balance feedback by way of the Wizard of Oz test, and observe participants’ mean heart rate and maximum skin electrical signal data.
Measurements and Procedure To measure the cognitive load in the experiment, the SWAT scale was used for evaluation. We also used a standard scale of VACP model to test the effect of considering the multimodal balance with and without multimodal balance on users’ affective experience. VACP model is evolved based on multi-resource
744 Wang and Wei
Figure 2: VACP standard scale and case display diagram.
theory for predicting workload. VACP denotes four channels: visual, audi- tory, cognitive, and motor, respectively. There are 0-7 scores in each channel, and higher scores represent higher occupied channels. The modality with high occupied channels can be identified through task analysis and the corre- sponding complementary channels can be derived to construct a multimodal balance model based on interaction contexts. the VACP standard scale and case demonstration are shown in Figure 2.
According to the results of SWAT scale analysis, driving at higher speeds (greater than or equal to 40km/h) during the peak commuting period is a high cognitive load task. Combined with the VACP model predicts that under high cognitive load, the proportion of anger emotion channel occupancy caused by left and right congestion VACP is 5.4, 2, 6.8, and 4.6, respectively. It can be seen that visual and cognitive channel occupancy is higher, and auditory is occupied less and can play more space. Combined with the comprehen- sive analysis of users’ negative emotion elimination modality in the interview information, it is proposed to balance the anger emotion through the modal combination of voice conversation and scent placement. Similarly, the bala- ncing modality for anxiety under low cognitive load is dynamic expression and music playing.
The experimental procedure was divided into three main parts, (1) Partici- pants were first introduced to the process and equipment of the experiment, and learned to adapt to the data collection of the experiment through simple manipulation. (2) After formally starting the experiment, participants were first induced to produce anger or anxiety through experimental material sti- muli, and multimodal balance models were provided and not provided while participants operated the high cognitive load interaction task, respectively. (3) Changes in participants’ emotional experiences were judged by analyzing physiological data at the end of the experiment.
A Multimodal Interaction Experience Design Approach 745
Table 1. LSD-t test.
Table available in the original PDF paper.
Multiple Comparisons Std. Error Sig. 95% Confidence Interval
Dependent Variable
(I) group (J) group Mean Difference (I-J)
Lower Bound
Upper Bound 12.90000* 2.37126 0 8.1772 17.6228
Heart rate before anger
after anger 5.00000* 2.37126 0.038 0.2772 9.7228 before anxiety after anxiety
0.315 0.19149 0.104 -0.0664 0.6964
GSRmax before anger
after anger .47500* 0.19149 0.015 0.0936 0.8564 before anxiety after anxiety
* The significance level for the difference in means is 0.05.
RESULTS
The heart rate and maximum electrodermal signal of the users were recorded during the experiment, and the LSD-t test was performed on the sample data using SPSS software. The chi-square test yielded that the significance of heart rate and GSRmax were P = 0.911>0.05; P = 0.265>0.05, respectively, indi- cating that the variance data were valid, and the significance was obtained by one-way ANOVA were less than 0.05, indicating that the data were very significantly different.
A post hoc LSD-t test was conducted to explore the correlation between the data in two pairs, and a significant difference was obtained between the heart rate measurements, and a specific analysis revealed that multimodal balance was significant for both anger and anxiety calming during the high cognitive load task. While between the GSRmax measurements, only anxi- ety contrasted significant values of 0.015<0.05, and anger values were not significant. This indicates that multimodal balance had a significant effect on the calming of anxiety during the high cognitive load task and a non- significant effect on the calming of anger. It is possible that the time of the experiment may have influenced the participants’ psychological states to flu- ctuate resulting in changes in the values. Overall, the multimodal balance had an ameliorating effect on negative emotions in the high cognitive load driving task. As is shown in Table 1.
DISCUSSION AND CONCLUSIONS There were significant differences in the effectiveness of multimodal bala- ncing models for improving negative driving emotions. Overall, in the context of high cognitive load, the user’s mental state and physiological data indi- cators change significantly, and the balancing of interaction load through appropriate interaction can achieve the effect of improving the experience of negative emotions. For anger, dialogue combined with scent can effecti- vely interfere with the user’s attention and disengage from the anger in time.
746 Wang and Wei
For anxious emotions, dynamic expressions combined with music can help users translate their attention instead of keeping their eyes on the vehicle in front of them.
Due to the limitations of the COVID-19 pandemic, this study still has limi- tations. First, the sample size is not very adequate, which may cause biased effects on the experimental data. Second, the results of our study were done through a simulation experiment, which could not fully simulate a real dri- ving scenario. Finally, only 2 variables were used in the participants’ negative emotions, can more fine-grained emotions be introduced for analysis? These questions are subject to further study in the future.
ACKNOWLEDGMENT
The authors would like to acknowledge Shaanxi Provincial Innovation Capability Support Program Funding Project (2021PT-025) and Shaanxi University “Youth Outstanding” Talent Support Program Project.
REFERENCES
Bangli Shao, Yin Zhu, Run Zhu, et al. (2021). Multi-modal man-machine intelligent interaction approach for smart home equipment control. Journal of Forestry Engineering, 6(4), 7. Desmet, P. M. A. (2004). Measuring emotion: development and application of
an instrument to measure emotional responses to products. Kluwer Academic Publishers. Dormann, C. (2003). Affective experiences in the home: measuring emotion. Hoit. Jing Huang, Mengting Yang. (2021). Initial Emotion-based Evaluation of the Personalized Driving Load State. China J. Highw. Transp. Lee, S., Ryu, H., Park, B., & Yun, M. H. (2020). Using physiological recordings for
studying user experience: case of conversational agent-equipped tv. International Journal of Human-Computer Interaction, 36(9), 1–13. Naumann, A. B., Wechsung, I., & Hurtienne, J. (2010). Multimodal interaction: a suitable strategy for including older users?. Interacting with Computers, 22(6), 465–474. Ni Jiang, Ran Li, Chunyao Liu, et al. (2021). Application of PAD Emotion Model in
User Emotional Experience Evaluation. Packaging Engineering, 42(22), 8. Na Lin, Bing Jiang. (2011). Emotional Experience Design of Homogenization Era: With Mobile Telephone as an Example. Journal of Beijing Institute of Technology (Social Sciences Edition), 13(6), 5. Qiusheng Tang, Lingbo Sun, Shichao Liu, et al. (2021). Analysis of influencing
factors of bus drivers’ working state under negative emotional driving based on structural equation model. Science Technology and Engineering. Weihua Lu, Guanyue Jiang, Yi Xue, et al. Multi-modal man-machine data-driven service design for check-in experience. Computer Integrated Manufacturing Systems. Yamauchi, T., Seo, J., & Sungkajun, A. (2018). Interactive plants: multisensory
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-23-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional