Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Interaction preferences of human body's natural gestures and gestures in smart Large displays scenarios

1 School of Design, Hunan University, Lushan Road (S), Yuelu District, Changsha, Hunan Province, China ↗
2 State Key Laboratory of Advanced Design and Manufacturing for Vehicle Body, Lushan Road (S), Yuelu District, Changsha, Hunan Province, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1003446

Abstract

With the rapid development of smart Large displays and motion sensing technology in the past ten years, gesture interaction based on smart Large displays has become more and more popular in home and public environments. The freehand interaction of smart large screen can be applied to traditional desktop displays and input Situations where equipment is unsuitable or insufficient for information presentation and task manipulation.This paper discusses the influence of natural body posture factors and interaction distance on the gesture interaction of the home smart Large displays. The user completes the specified task by customizing gestures in the home smart screen in a natural posture. The gestures displayed by the user are counted and the smart Large displays is designed. A set of user-defined interactive gestures in household scenarios. The perceived usability evaluation experiment of the user-defined interactive gesture set is conducted, the perceived usability and workload of the user gesture set are recorded, and the best gesture command preference is screened out.The results showed that the same natural posture had a significant impact on the application variation of gestures, with significant differences in interaction performance, perceived usability, and workload between sitting and lying postures. This study provided implications that are likely to improve the design and deployment of large display-based freehand interaction techniques.

INTRODUCTION

In the past decade, due to the rapid development of smart screens and motion sensing technologies, free-hand interaction based on smart screens has become more and more popular in personal and public environments (Hamed S. Alavi, Himanshu Verma, 2017). The free-hand interaction of the smart large screen can be applied to situations where traditional desktop displays and input devices are not suitable or not sufficient for information presentation and task operation. Compared with traditional device-assisted desktop interaction, freehand interaction based on smart large screen has many advantages, such as long-distance interaction, free space navigation, easy hand interaction, and the device is not connected to the user, and the

© 2023. Published by AHFE Open Access. All rights reserved. 851

852 Li and He

smart large screen provides information visualization. Better visibility and more space (Nuno, Samuel, 2018). Previous studies have made multiple stu- dies to develop various freehand gesture interaction gestures (Banerjee, A., Burstyn, J. 2021). However, there are also many limitations in freehand interaction with large home screens. For example, the screen environment may make interaction, comfort, and usability ratings low during freehand interaction, accompanied by high workload (Da Tao, Xiaofeng Diao, 2021).

To solve the above problems, we must first define the use environment of using freehand interaction on large displays as a home smart home scene.

However, the role of body posture in large-display-based freehand intera- ction has not been investigated in previous reports. Similar to factors such as body posture, the effect of interaction distance on freehand interactions with large displays has not been well studied so far.

The purpose of this study is to test whether the body posture will affect the gesture interaction of the user operating the smart large screen when opera- ting the smart large screen at home, and to find suitable user-defined gestures in the smart home scene.

STUDY AND METHOD Participants

In the experiment, 24 right-handed participants (12 males and 12 females) were recruited at a local university. We included participants who: (1) had neither neck and shoulder injuries nor a history of significant neck and shoul- der pain; (2) had used a home smart TV for at least 3 years and used it for more than 1 h per day; these criteria were designed to ensure Partici- pants had similar skills in using smart TVs at home.The mean body height and whole-arm-length of the participants were 155.7 cm (SD = 39.3) and 69.8 cm (SD = 17.8), respectively. All participants reported that they were right-handed, had normal or corrected-to-normal vision and normal color vision, and had no known motor impairments. The study was approved by the Institutional Review Board of HUNAN University.All participants signed an informed consent form, and the experimental procedures were clarified before the study began.

Apparatus The smart big screen is Huawei ideahub board (China Huawei Techno- logies Co., Ltd.). The screen size of this smart large screen is 65 inches (1650mm × 1050mm, resolution 1980×1080) and the windows10 opera- ting system is an all-in-one computer as an experimental display. This choice of monitor size is used because it is widely used in many educational, comme- rcial, and home environments. The display is set at the default height with its center store 135cm off the ground. Created a computer application program developed by Visual Basic to simulate the actual usage scenario of the user. Using 3 Canon cameras to record user gestures, it is less likely to lose user gesture tracking.

Interaction Preferences of Human Body’s Natural Gestures and Gestures 853

Experiment In the selection of test control instructions, we based on previous research (R.- D. Vatavu, 2012). From the 12 most commonly used commands for smart TV control, 4 control commands are selected as the initial required commands for our user test. The 4 selected commands are: turn on the TV, turn off the TV, turn up the volume, turn down the volume.

In the selection of test postures, we refer to the positioning and descri- ption of standard human static postures in anatomical literature, and select the most important 6 postures: x Upright sitting position: 90-degree semi- upright sitting position y Fowler sitting position: 30 to 45 3 degrees of semi-upright sitting position z Left lying position: horizontal position, left side facing down { Right lying position: horizontal position, right side facing down | Supine position: face up horizontal position } Prone position: face down horizontal position (Fig. 1).

Figure 1: Positioning and description of human static posture (adapted from Anatomi- cal position: definitions and illustrations, 2018).
Original PDF Figure 1:

The experiment adopted a three-factor (2×2×1) subject design with 6 body postures (x upright sitting, y Fowler sitting: z left lying position, { right lying position, | supine position, } prone position), 4 An operational task (turn on TV, turn off TV, turn up the volume, turn down the volume) as an independent variable. According to the observed free distance range (i.e. 1.72m-3.35m), the subject was at the same height as the center of the screen when sitting upright.

Dependent variables include task performance indicators (i.e., task com- pletion time, error rate, and number of errors) and user perception indicators (i.e., perceived availability and workload). Task completion time is the time it takes for participants to complete the pointing or dragging task. The error rate is calculated as the proportion of tasks with errors.

Experimental Procedure The first step: debug the equipment and arrange the testes to be in place. Read the test instructions. 2. The experimenter adjusted the seat to 90

  • , and the sitting posture of the tested person was adjusted to sit upright. Watch the sitting posture norms and follow the guidance to turn on the TV, turn off the

854 Li and He

TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera positions are respectively videotaped and recorded. 3. The experimenter adjusted the seat to 120

  • , and the sitting posture of the tested person was adjusted to Fowler sitting. Watch Fowler’s sitting posture specification and follow the guidance to turn on the TV, turn off the TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera positions are respectively videotaped and recorded. 4. The experimenter adjusted the seat to 180
  • , and the sitting posture of the tested person was adjusted to lie on the left side. Watch the left-lying posture specification and follow the guidance to turn on the TV, turn off the TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera positions are respectively videotaped and recorded. 5. The experimenter adjusts the posi- tion of the pillow, and the sitting posture of the tested person is adjusted to lie on the right side. Watch the right side lying posture specification and fol- low the guidance to turn on the TV, turn off the TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera positions take video and audio recording respectively. 6. The experimenter adjusts the position of the pillow, and the sitting posture of the tested person is adjusted to supine. Watch the supine posture specification and follow the guidance to turn on the TV, turn off the TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera posi- tions are respectively videotaped and recorded. 7. The experimenter adjusts the position of the pillow, and the sitting posture of the tested person is adju- sted to prone. Watch the prone posture specification and follow the guidance to turn on the TV, turn off the TV, turn up the volume, and turn down the volume. The experimenters record according to the behavior analysis form, and the three camera positions are respectively videotaped and recorded. 8. Conduct semi-structured interviews with testers (gesture switching during the experiment, gesture comfort, etc.), and end the experiment.

A 10-minute interval between tasks to avoid fatigue. To prevent interfere- nce, all electronics were turned off except for experimental equipment.

RESULTS

Data statistics: In this experimental test, 24 subjects were tested in 6 different postures: sitting upright, Fowler sitting, lying on the left side, lying on the right side, lying on the back, and prone. 576 gestures operated by commands.

576 gestures are grouped together. The grouping principles are as follows: x The left and right hands are summarized as the same gesture; y The same gesture with multiple frequencies is summarized as the same gesture; z The different angles of the palm of the same gesture are summarized as the same gesture. { Only the posture and movement of the hand are considered, and other parts such as the elbow are not considered.

The obtained gesture set will be used as the option setting of the multi-choice questionnaire design, and the 1–2 gestures with the highest

Interaction Preferences of Human Body’s Natural Gestures and Gestures 855

frequency will be screened out through the questionnaire as the final test gesture set.

Verify whether the user’s posture changes have an impact on the interaction between gestures and smart TVs, and create gestures suitable for turning on and off the TV and adjusting the volume under multiple postures. Two-way ANOVA by comparing the frequency of gesture changes, take the gestures of the tested individual in the sitting posture as the control group, count the frequency of gesture changes, and verify whether the posture has an impact on the gestures. By comparing the consistency rate of the sample gestures with the gesture consistency rate of the same subject completing 4 tasks under 6 body postures, the influence of posture on gestures can be obtai- ned. The frequency and matching degree of gestures. The occurrence and frequency of gestures and the degree of matching between gestures and tasks in the questionnaire survey are obtained by users, and the general gestures are obtained.

According to the scale of gesture evaluation, user gestures were classified and the same gestures were marked.

The consistent rate of gestures of individual users in different gestures when turning on TV, turning off TV, turning up the volume and turning down the volume were calculated respectively. The individual agreement rate of each task was summarized, the mean value was calculated, and the speci- fic influencing factors of posture on gesture agreement rate were compared and analyzed (Table 1). Single factor analysis of variance and Chi-square analysis were carried out to determine whether the task had an effect on the gesture agreement rate (Table 2). Combined with specific experimental data

Table 1. Objective quantified two-factor analysis of variance.

Table available in the original PDF paper.

Table 2. Subjective quantified two-factor analysis of variance.

Table available in the original PDF paper.

856 Li and He

and subsequent gesture design judgment, all data were subject to subjective quantification, that is, with TV as the axis, hand movement changes and movement path changes were regarded as gesture changes.

Consistency rate calculation formula: 2 |Pi| X

Ar = |Pr| Pi⊂Pr

A total of 96 user-task consistency rate data were obtained by summarizing the consistency rate of 24 subjects performing four tasks. The mean value of data samples was 0.63, the maximum value was 1, and the minimum value was 0.17. As a result, the overall consistency degree of users was greater than 0.6, which was quite high (Table 3). The consistency rate is relatively concentrated below 0.4 and 1, among which 32 samples are less than 0.4 and 36 are the maximum value 1. A total of 56 samples had data less than 0.6, with a low consistency rate, accounting for 58.33% of the total data. Although the overall consistency is high, the data dispersion analysis shows that users with high consistency rate are more likely to choose simple gestures with one hand and wrist movement than those with high consistency rate. This kind of gesture is less affected than complex gestures in different gestures, because wrist movement can be easily used when the arm is restricted (Table 4).

The minimum expected count is 16, and the value of progressive significa- nce (bilateral) - Pearson chi-square is 0.03, less than 0.05. There is a difference between turning on TV and turning up the volume on hand gestures. The consistency rate of turning on TV is 0.66, and turning up the volume is 0.60 (Table 5). To explore the reasons, it can be seen that the task of increasing the volume is a task requiring displacement in the user’s cognition, while in the position of lying on the side and supine, the user has a high probability of changing the gesture due to the limitation of the arm. Therefore, in the task

Table 3. Consistency rate processing results of sample gestures.

Table available in the original PDF paper.

Interaction Preferences of Human Body’s Natural Gestures and Gestures 857

Table 4. Consistency rate processing results of sample gesture.

Table available in the original PDF paper.

Table 5. Square card test.

Table available in the original PDF paper.

of displacement cognition, the use of wrist and arm should be considered in gesture creation.

Limitations and Future Work Our study only evaluated short-term task execution behavior. However, if a large workplace-based display-based hand-drawn interaction to daily tasks, users would use it for hours. The effects of posture and distance effects may change in this long-term use case and warrant further study. Second, practi- cing unaided interaction for either short or long periods of time has physical demands and fatigue, which are important indicators of well-being. The way we use subjective scales to measure physical demand and fatigue may may be influenced by self-report bias. Further research work could be conducted to compare different types of unassisted hand-to-hand interactions. By doing so, we can better understand the advantages of various types of unassisted interactions as an alternative to interacting with large displays. The advanta- ges and disadvantages of various unassisted interactions as an alternative to interacting with large displays.

858 Li and He

For future research, it is hoped that a complete set of user-defined gestures can be started, where users perform gestures based on operational commands, to summarize the gestures for somatic interactions that are consistent with user perceptions, rather than based on existing gestures. In fact, it is hoped that a multimodal form of evaluation can be introduced so that gestu- res with good performance in both physiological and psychological aspects of somatosensory interaction can be identified, making future gestures for somatosensory interaction sufficiently user-friendly.

CONCLUSION

The user gestures of each posture task were classified according to the scale of gesture evaluation, and the same gestures were marked. Calculate the fre- quency of the same gestures in the same task with the same body posture, select the most frequent 2–4 gestures and shoot them into GIFs.

The set questionnaire collects the gesture data that users think best matches the task under this gesture, and each gesture has corresponding data, namely the matching degree. Based on the corresponding analysis of the matching degree and frequency of gestures, the highest group of gestures is the one with the highest applicability under multiple gestures, and three groups of gestures are obtained.

The change of the user’s posture has a significant effect on the gesture interaction with the smart TV. Users subjectively believe that the consistency of gestures is not affected by the difference in the use of palms, wrists and arms with the TV as the axis. One-handed, simple hand movements are more suitable for use in multiple postures.

Task difference has no significant effect on air distance gesture interaction, but when the task involves displacement path, the consistency rate of the task with displacement path is lower. When it comes to the gestures of displace- ment path, wrist and arm gestures can be used at the same time, which is more suitable for the use of situations in multiple positions. According to the frequency and matching degree, switching TV - palm open contraction frequ- ency is the largest, and switching TV - remote control is the largest matching degree. Adjust volume - Palm up and down for maximum frequency and matching.

Limited consideration should be given to the design of gestures in the multi- posture situation. In simple hand movements, the wrist and arm can be moved simultaneously and can be switched easily without cognitive stress.

REFERENCES

António Teixeira, Nuno Almeida, Carlos Pereira, Miguel Oliveira e Silva, Diogo Vieira, and Samuel Silva. 2017. Applications of the Multimodal Interaction Arch- itecture in Ambient Assisted Living. In Multimodal Interaction with W3C Stan- dards - Toward Natural User (1 ed.), Deborah Dahl (Ed.). Springer International Publishing, 271–291. Banerjee, A., Burstyn, J., Girouard, A., Vertegaal, R., 2012. MultiPoint: comparing

laser and manual pointing as remote input in large display interactions. Int. J. Hum. Comput. Stud. 70 (10), 690–702.

Interaction Preferences of Human Body’s Natural Gestures and Gestures 859

D. Vatavu, “User-defifined gestures for free-hand tv control,” in Proceedings of the 10th European Conference on Interactive TV and Video, 2012, pp. 45–4. Da Tao, Xiaofeng Diao, Tieyan Wang, Jingya Guo, Xingda Qu, Freehand intera-

ction with large displays: Effects of body posture, interaction distance and target size on task performance, perceived usability and workload, Applied Ergonomics, Volume 93, 2021, 103370, ISSN 0003–6870. Gabriela Amaral Araújo de Oliveira, Raphael Winckler de Bettio, and André Pimenta Freire. 2016. Accessibility of the smart home for users with visual disabilities: an evaluation of open source mobile applications for home automation. In Procee- dings of the 15th Brazilian Symposium on Human Factors in Computer Systems. ACM, 29. Google. {n d.}. Google Assistant. ({n. d.}). https://developers.google.com/assistant/sd

k/overview Hamed S. Alavi, Himanshu Verma, Michael Papinutto, and Denis Lalanne. 2017.

Comfort: A Coordinate of User Experience in Interactive Built Environments. In IFIP Conference on Human-Computer Interaction. Springer, 247–257. Nuno Almeida, Samuel Silva, António Teixeira, and Diogo Vieira. 2017. Multi- Device Applications Using the Multimodal Architecture. In Multimodal Intera- ction with W3C Standards - Toward Natural User (1 ed.), Deborah Dahl (Ed.). Springer International Publishing, 367–383. Nuno Almeida, Samuel Silva, António Teixeira, Maksym Ketsmur, Diogo Guima-

rães, and Emanuel Fonseca. 2018. Multimodal Interaction for Accessible Smart Homes. In Proceedings of the 8th International Conference on Software Develo- pment and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI 2018). Association for Computing Machinery, New York, NY, USA, 63–70. Paolo Baggia, Daniel C. Burnett, Jerry Carter, Deborah A. Dahl, Gerry McCobb, and Dave Raggett. 2009. EMMA: Extensible MultiModal Annotation markup language. (2009). http://www.w3.org/TR/emma/ Paulo A Condado and Fernando G Lobo. 2015. A system for controlling assisted

living environments using mobile devices. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM, 33–38.

Cover of the proceedings volume Ergonomics In Design

Explore the complete digital volume

Ergonomics In Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-53-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →