Cao Liang (2024). Bridging the Cognitive Gap: Optimizing Gesture Interaction Design for the Elderly. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 783–792. https://doi.org/10.54941/ahfe1004543
With the advancement of technology, individuals are on the verge of embracing an era of artificial intelligence (AI). While the utilization of innovative products enhances convenience in people's lives, it often brings forth new challenges and problems. The elderly, in particular, face additional obstacles in using AI products due to physical and cognitive aging. However, gestural interaction, as a new means of interaction, has the potential to address this issue.In this paper, we employ qualitative research methods to explore the guidelines for gesture interaction between AI systems and elderly users. By conducting focus group interviews, we identify a suitable set of gesture commands and analyze the factors that impact the elderly's comprehension of gestures. The results of our study provide valuable research methods and guidelines for future gesture command design.
The global demographic landscape is significantly shifting due to an aging population, posing distinct challenges and opportunities in healthcare, social services, and economic planning. To tackle these challenges, various sectors are actively exploring Artificial Intelligence (AI) solutions. Researchers are investigating how aging-related normative changes impact technology inter- actions and are dedicated to designing technology that better caters to the elderly demographic (Czaja & Lee, 2007; Rogers, Stronge, & Fisk, 2006). However, the elderly often face challenges related to age-related changes, such as cognitive impairments or mobility disabilities. To address these issues, gesture technologies have emerged as a promising solution, offering a combi- nation of enjoyment, naturalness, ease of use, intuitiveness, and learnability (Van Beurden et al., 2012).
In various Human-Computer Interaction (HCI) systems, such as touch- screens, mobile devices, and hands-free interfaces, experts devise user ges- tures, often driven by convenience or alignment with the technology rather than user-centric considerations. This recognition is evident in both design literature (Huang et al., 2018) and the broader HCI domain (Piumsomboon et al., 2013). To address this challenge, researchers have introduced inno- vative approaches. Nielsen et al., (2008) proposed a method for creating a
tailored gesture vocabulary for hands-free computer interaction in ubiqui- tous computing, considering factors like user intuition, learning speed, and ergonomics. Building on this foundation, Wobbrock et al., (2005) devel- oped the “guessability study methodology,” initially focused on enhancing symbolic input predictability but later expanded to include broader user- generated gesture elicitation. In this setup, users suggest gestures based on provided references, with all propositions considered valid, employing the “Wizard of Oz” approach for confirmation (Lee and Billinghurst, 2008). This methodology, commonplace in user-elicited studies, significantly con- tributes to advancing user-centric gesture design. Apart from user-elicited studies, Villarreal-Narvaez et al., (2022) argue that theoretical analysis can notably enhance the early-stage design of future user interfaces. In this arti- cle, I will integrate this perspective with findings from an empirical gesture elicitation study.
This article aims to engage elderly individuals in the gesture design pro- cess, temporarily separating technical concerns from the design of gesture commands, with a primary focus on accommodating the cognitive and usage habits of the elderly. The qualitative research conducted in this study explores suitable gesture control methods for individuals aged 65 and older in China, with a key emphasis on identifying gestures accepted by the elderly and easy for them to quickly learn. This work contributes significantly in two aspects: 1) proposing guidelines for designing gestures that cater to the elderly’s needs and exploring design principles applicable beyond China, and 2) developing gesture commands aligned with the habits of elderly individuals in China while analyzing factors influencing gesture recognition in this population.
With the integration and advancement of AI in elderly care, gesture inter- action is poised to become the future trend of human-computer interaction. My work not only focuses on exploring the needs of gesture interaction for older adults but also provides design guidance for other scenarios involving gesture-based controls.
METHODOLOGY
While extensive research and user studies have been conducted on gesture recognition, a critical gap exists in understanding gesture vocabulary for the elderly. Many studies have adopted Jacob O. Wobbrock’s 2005 “guess- ability study methodology,” but user-generated gestures present challenges, especially for seniors dealing with cognitive decline. To bridge this gap, I’ve adapted my research methods for the elderly. I’ve introduced a predefined gesture set inspired by sign language and existing research. Using a multi- round interview approach with closed-ended questions, I aim to explore how the elderly recognize and use gestures and identify appropriate gesture com- mands. This approach effectively avoids the dilemma of users struggling to design new gesture commands.
Before initiating the study on natural-gesture control for the elderly, I meticulously crafted a hand gesture list, blending natural body language with device control functions (refer to Figure 1). These gestures account for age- related changes and impairments, prioritizing simplicity, easy recognition, and rapid learning.
Bridging the Cognitive Gap: Optimizing Gesture Interaction Design for the Elderly 785
Figure 1: Predefined gesture commands set.
This research encompasses three rounds of interviews, with two primary objectives: a. To gain insights into the characteristics and variations of elderly cognition. b. To identify the most suitable gesture command set for older users in a home environment. To accomplish these goals, the research will be structured into three distinct stages, each emphasizing specific phases, as outlined below: Phase one: Elderly Cognition Investigation
– Conducting one-on-one interviews utilizing the card sorting method to categorize and collect data on various predefined gesture commands. – Formulating hypotheses based on participant performance to unveil the underlying structure and associations users attribute to these gestures.
Phase two: Optimal Gesture Commands Identification – Evaluating participants’ comprehension of preset gestures by comparing results with Phase One. – Participants rate their confidence in associating each gesture on a scale of 1 to 5, allowing researchers to assess users’ certainty. – Screening gestures based on comprehension and certainty ratings, retain- ing clear and consistent gestures while refining or eliminating ambiguous ones.
Phase three: Feasibility Assessment – Assessing the feasibility of selected gestures by observing and recording the time participants require to learn each gesture. – Initial demonstration of gestures in picture form, followed by guided participant imitation. – For challenging gestures, step-by-step demonstrations are provided, with researchers recording the learning time. – Utilizing user performance data to inform the selection and optimization of the final set of gesture commands.
786 Cao
RESULTS AND ANALYSIS This study employs the grounded theory methodology to analyze the recog- nition of gesture commands among the elderly population. Through open coding, axial coding, and constant comparison, I derived insights and iden- tified patterns that contributed to the development of a grounded theory on gesture recognition and classification among older users.
Table 1. Coding.
Table available in the original PDF paper.
Open code Axial code Quote Code Sub-Category Category
Consensus in Gesture Recognition
Shared Gesture Understanding
Elderly and young participants reached a consensus on 14 out of 18 gestures.
Consensus on Gesture Recognition Concentrated Choices of Young
Young participants’ choices were more concentrated compared to the elderly.
Age-Related Cognitive Variability
Cognitive Differences between Young and Elderly
The elderly were more likely to hesitate in making gesture choices.
Hesitation among the Elderly Influence by Apple mac
Young users’ choices were influenced more by prior experience with smart devices.
Impact of Cultural Background and Experience
Cultural Influence on Gesture Interpretation Interface gestures influenced the understanding of 3D gestures.
Interface Gesture Impact Gesture like volume control knob in the car Influenced by the use of physical buttons
OK is familiar sign language Sign language Choose “Unknown” by elderly “Unknown” Decreased comprehension/ Technophobia
Incomprehension in the test
Challenges in Technology Understanding Some elderly participants found gesture control incomprehensible even after multiple explanations.
Few gestures allow all older testers to agree on a choice
Seniors’ choices often vary Uncertainty in Switch and Adjust choice Adjusting or Switching
Aging and stiffness of the finger joints
Fingers are not flexible
Decreased strength and flexibility in the elderly
Physical Limitations and Gesture Interaction Gestures in the air for long periods of time cause fatigue
Feeling tired when doing this action
Shared Gesture Understanding with cultural influence: In this study, I tested eighteen gesture commands, and the results showed that the recog- nition of most gestures reached a consensus between the young and elderly participants. Specifically, fourteen out of the eighteen gestures were recog- nized similarly by both groups. The gestures in Figures 2 achieved high consensus in the test, with their understanding deeply influenced by cul- tural backgrounds. Many gestures have their origins in sign language, which is widely used across the world. For instance, the gesture “OK” (Figure 2)
Bridging the Cognitive Gap: Optimizing Gesture Interaction Design for the Elderly 787
was initially popularized in Western society, but its meaning has transcended cultural boundaries, as both young and elderly Chinese individuals now com- prehend its significance. Furthermore, during the process of recognizing a gesture, individuals unconsciously link the gesture command to gestures used in their daily life. This unconscious association is significantly influenced by their living environment, resulting in users from similar cultural backgrounds reaching a consensus in gesture recognition easily.
Figure 2: Gesture (14) recognition result between the young and elderly.
Age-Related Cognitive Variability and Challenges in Technology Under- standing: While the young and elderly tend to agree on many things, cognitive decline with age becomes evident during testing. The elderly participants took an average of 747 seconds to complete the test, more than twice the time taken by the younger participants (average 326 seconds). This time gap highlights cognitive differences between the age groups.
Additionally, the results showed that younger participants had more con- sistent perceptions of gestures and were more certain in their choices, likely due to their greater exposure to smartphones and the internet, which fos- ters a shared understanding. In contrast, the elderly, with less technological exposure, displayed more uncertainty in their gesture interpretations. This variability can be attributed to “Technophobia,” a fear or aversion to tech- nology, predominantly observed in the elderly. Their unfamiliarity with modern digital devices and technological apprehension can hinder technology comprehension and make them anxious or resistant to using gesture-based technology, affecting their gesture recognition choices.
Cultural Influence on Gesture Interpretation and Challenges in Technol- ogy Understanding: This study examined the impact of two factors on elderly users: educational background and prior experience with smart products. When using the criterion of whether the elderly had a high school educa- tion, 78% of cases in the gesture test revealed significant differences between the groups (see Figure 3). Interviews with three elderly individuals with no education highlighted difficulties in understanding gesture techniques, often leading to incomplete tests and obstacles in using gesture-based technology.
788 Cao
Furthermore, considering prior experience with smart products, 72% of test results differed between user groups (see Figure 3). Participants with smart product experience, especially in interface gestures, exhibited better profi- ciency and certainty in gesture recognition. Their familiarity with digital interfaces and gesture interactions enabled more decisive choices.
In summary, prior smart product experience and education significantly influence the elderly’s ability to comprehend gesture commands. Designing inclusive interfaces and considering users’ tech exposure can enhance the usability and accessibility of gesture-based systems across age groups.
Figure 3: Educational factor on gesture recognition; Prior experience factor on gesture recognition.
Physical Limitations and Challenges in Technology Understanding: Phys- ical limitations among the elderly also significantly influence gesture inter- action and recognition. As individuals age, they may experience decreased strength and flexibility, which can impact their ability to perform cer- tain gestures effectively. During the interviews, it was observed that some elderly participants faced challenges in physically executing specific gestures like Figure 4, leading to difficulties in accurately conveying their intended commands.
The physical limitations experienced by the elderly can manifest in various ways during gesture-based interactions. For instance, gestures that require precise fine motor control or complex hand movements may be more chal- lenging for individuals with reduced dexterity. Likewise, gestures that involve large arm motions or reaching might pose difficulties for those with limited mobility or joint stiffness. As a result, elderly participants may struggle to per- form certain gestures, leading to inconsistent or unclear gesture recognition outcomes.
Furthermore, the physical limitations may intersect with challenges in technology understanding, amplifying the difficulties faced by some elderly participants. Feeling unable to execute gestures fluently due to phys- ical constraints can contribute to a lack of confidence and increased technophobia, further hindering their engagement with gesture-based technology.
Bridging the Cognitive Gap: Optimizing Gesture Interaction Design for the Elderly 789
Figure 4: Gesture usability testing.
In conclusion, our grounded theory analysis illuminated the complexi- ties of gesture recognition among young and elderly users. Understanding the cognitive differences and identifying factors influencing gesture under- standing can inform the design of more intuitive and inclusive gesture-based interfaces for older users in various environments.
Based on the above insights we propose a guideline as follows: Context-Sensitive Gestures: Designing gestures that align with the context of use is essential in creating a seamless and intuitive interaction experience for the elderly. Taking into account the various contexts in which gestures will be performed, such as sitting or standing, can greatly enhance the effectiveness and usability of gesture-based technology for this demographic.
Localization Design: Insights from interviews highlight the substantial impact of cultural background on users’ judgment in gesture interactions. When designing gesture-based systems for a specific population, it’s crucial for designers to consider cognitive abilities, living environment, and cultural background. Understanding the users’ cultural context aids in identifying and incorporating gestures aligned with their cognitive patterns, enhancing user experience and engagement with the technology.
Simplicity and Clarity: With age, cognitive decline and memory loss pose challenges, particularly for the elderly. Designing gesture interactions requires prioritizing simple and clear gestures to ensure a smooth interaction expe- rience. Complex or unclear gestures may cause confusion and frustration, hindering effective technology interaction. Incorporating intuitive gestures enhances the overall user experience, facilitating easy navigation and use of smart devices for the elderly.
Large and Precise Gestures: Designing gestures that require large, deliber- ate motions is essential to accommodate the reduced dexterity often expe- rienced by the elderly. Such gestures should be tailored to their physical capabilities, allowing them to interact with technology comfortably and accurately.
Exertion Management: By streamlining the gestures needed to perform common tasks, designers can alleviate physical fatigue that may arise during use of the technology. The elderly, in particular, may face challenges in main- taining repetitive and complex gestures, making it essential to optimize the interaction process for efficiency and ease of use.
790 Cao
Feedback and Confirmation: Providing visual and auditory feedback to confirm successful gesture recognition is a crucial aspect of designing user interfaces for the elderly. These feedback mechanisms serve as reassuring cues, validating that the gesture input has been accurately recognized by the sys- tem. Visual feedback, such as highlighting the selected gesture or displaying an animation, can offer a clear indication of successful interaction, instill- ing confidence in the elderly users’ actions. Concurrently, auditory feedback, such as a subtle sound or a gentle tone, can reinforce the acknowledgment of their gestures, creating a multisensory experience that further solidifies their interactions.
Customization Options: Allowing users to customize gestures to match their personal habits can further enhance the usability and user satisfaction of gesture-based interactions. Each individual may have unique preferences and accustomed movements, making customizable gestures a valuable feature. By providing this flexibility, designers empower users to create gestures that align with their natural gestures and movements, resulting in a more intuitive and personalized experience.
The guideline contribute to enhancing the user experience and facilitating seamless interactions between older adults and smart devices. Following these guidelines and three rounds of assessment, we have introduced a customized set of recommendations for elderly users in their home environment.
In the first phase, we tested the consensus level of each gesture among the elderly and visually presented it in the form of a bubble chart (Figure 5). The larger the bubble, the more easily the gesture command is recognized by the majority of elderly individuals.
Figure 5: Gesture comprehension among elderly users.
In the second phase, we tested the users’ certainty regarding gesture com- mands using a five-point scoring mechanism (Figure 6), where a score of five indicates a high level of confidence in understanding the gestures, and a score of one indicates uncertainty about the meaning of the gestures.
Based on our analysis of data from the initial two rounds, we identified a set of gestures that are highly intuitive and well-suited for elderly users’
Bridging the Cognitive Gap: Optimizing Gesture Interaction Design for the Elderly 791
effective command control. In the study’s final phase, we conducted exten- sive usability testing, considering user performance and the time needed for learning. This comprehensive assessment resulted in the selection of gesture commands (see Figure 7) aligned with elderly users’ learning capabilities. Our emphasis was on clarity and enhancing the overall user experience, ensuring a seamless and empowering interaction with technology at home.
Figure 6: Score of conformity.Figure 7: Final gesture set (open, close, adjust, switch).
CONCLUSION
The gesture interaction mode significantly enhances the elderly’s quality of life by providing a seamless, intuitive way to use smart devices, eliminating the need for extensive learning. However, cognitive differences due to aging can present challenges. With limited existing research, this paper investigates elderly cognitive characteristics related to gestures and suggests design guide- lines. This research serves as an initial step in this direction, shedding light on the complexities of designing gesture interactions for the elderly and identify- ing potential advantages in future gesture interaction development for older users.
At its essence, this study underscores the urgency for gesture interaction to evolve, embracing inclusivity, accessibility, and a compassionate under- standing of the cognitive changes accompanying aging as integral aspects
792 Cao
of innovative design. By transcending age-specific design constraints and embracing the diversity of user experiences, we can pave the way for a tech- nological landscape that genuinely caters to and empowers individuals of all ages.
ACKNOWLEDGMENT
I would like to especially thank my major professor Christopher Arnold and two other committee members, Shea Tillman and Clark Lundell, School of Industrial and Graphic Design at Auburn University. They gave many valu- able comments and help during my research. Thanks to my parents, they have always supported my study financially and emotionally. Thank my friends, who have been with me through the difficulties. Because of these people’s help, I complete the research in the past years successfully.
REFERENCES
Czaja, S. J., & Lee, C. C. (2007). Information Technology and Older Adults. In A. Sears & J. A. Jacko (Eds.), The Human-Computer Interaction Handbook. Lawrence Earlbaum Associates: New York and London. Czaja, S. J., Charness, N., Fisk, A. D., Hertzog, C., Nair, S. N., Rogers, W. A.,
& Sharit, J. (2006). Factors predicting the use of technology: Findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychology and Aging, 21(2), 333–352. Huang, J., Jaiswal, P. & Rai, R. 2018. Gesture-based system for next generation natural and intuitive interfaces. AI EDAM, 1–15. Lee, M. & Billinghurst, M. A Wizard of Oz study for an AR multimodal interface.
Proceedings of the 10th international conference on Multimodal interfaces, 2008. ACM, 249–256. Nielsen, M. Störring, T. B. Moeslund, and E. Granum, “A procedure for developing intuitive and ergonomic gesture interfaces for HCI,” in Gesture-Based Commu- nication in Human-Computer Interaction (Lecture Notes in Computer Science), vol. 2915. Berlin, Germany: Springer-Verlag, 2004, pp. 409–420. O. Wobbrock, H. H. Aung, B. Rothrock, and B. A. Myers, “Maximizing the guessabil-
ity of symbolic input,” in Proc. CHI Extended Abstracts Human Factors Comput. Syst., 2005, pp. 1869–1872. Piumsomboon, T., Clark, A., Billinghurst, M. & Cockburn, A. User-defined gestures for augmented reality. IFIP Conference on Human-Computer Interaction, 2013. Springer, 282–299. Van Beurden, M. H. P. H., IJsselsteijn, W. A., & Kort, Y. (2012). User Experience
of Gesture Based Interfaces: A Comparison with Traditional Interaction Meth- ods on Pragmatic and Hedonic Qualities. In Lecture Notes in Computer Science (pp. 36–47). https://doi.org/10.1007/978-3-642-34182-3_4 Villarreal-Narvaez, S., Sluÿters, A., Vanderdonckt, J., & Mbaki Luzayisu, E. (2022).
Theoretically-Defined vs. User-Defined Squeeze Gestures. Proceedings of the ACM on Human-Computer Interaction, 6, 73–102.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional