Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

The Design of Mid-Air Ultrasonic Haptic Interfaces Based on the Perception of Lines

1 VEMI Lab, The University of Maine, Orono, ME 04469, USA ↗
2 Toyota Research Institute, Los Altos, CA 94022, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1003586

Abstract

Mid-air ultrasonic feedback is a new form of haptic stimulation supporting mid-air, touch-free user interfaces. Functional implementation of ultrasonic haptic (UH) interfaces depend upon the ability to accurately distinguish between the intensity, shape, orientation, and movement of a signal. This user study (N = 15) investigates the ability to non-visually perceive two ultrasonic lines with varying lengths (3, 5, and 7 cm) and orientations (vertical and horizontal) using the palm of the hand. Key results showed that: (1) the orientation of the lines had no effect on a user’s accuracy when determining their relative lengths, (2) line length distinction significantly improved when the length difference was at least 4 cm, and (3) a clear learning curve was evident when evaluating a new user’s ability to perceive ultrasonic signals. The capabilities of UH technology identified and discussed within this study will help engineer user-friendly and functional mid-air haptic interfaces for future applications.

INTRODUCTION

Mid-air haptic feedback is a new form of technology that utilizes ultrasonic waves to create acoustic radiation forces. These forces stimulate mechanore- ceptors within the skin, which in turn, create an ultrasonic signal that can be perceived tactually without physical touch (Carter et al. 2013). Using this technology, a mid-air tactile point can be moved along a path to pro- duce lines, curves, and shapes, which can be non-visually perceived as haptic (touch-based) sensations on users’ hands and implemented within touch-free haptic interfaces (Hajas et al. 2020).

This study investigates the ability for a user to non-visually perceive two ultrasonic haptic (UH) lines with varying lengths (3, 5, 7 cm) and orien- tations (vertical and horizontal). Assessing the ability for users to perceive basic geometric elements (e.g., line length and orientation) is essential to consider before developing more complex touch-free haptic interfaces, spe- cifically ones that incorporate sliders, shapes, or other inputs along the

© 2023. Published by AHFE Open Access. All rights reserved. 18

The Design of Mid-Air Ultrasonic Haptic Interfaces Based on the Perception of Lines 19

X, Y, and Z axes. These touch-free haptic interfaces require accurate perce- ption of rectilinear components which represent the building blocks of many graphical displays and user interactions (Chongyang et al. 2019). Although the literature suggests early success of this promising technology, further user studies investigating the perception of basic geometrical elements are needed to optimize UH interfaces for commercial use.

RELATED WORK

Recent user studies have aimed to investigate human perception of UH sensa- tions, with particular interest in establishing the boundaries of what stimulus parameters and patterns are best perceived. These studies have begun to elu- cidate what specific orientations, frequencies, shapes, and movements of the UH signals are most salient and lead to the most accurate performance.

One study examined the difference in perception of UH movement along the vertical, horizontal, and oblique axes of the hand. This led to the fin- ding of the “oblique effect”, which demonstrated that humans have enhanced haptic perception (and greater accuracy) of ultrasonic stimuli that move along the horizontal and vertical axes of the hand, compared to oblique movement (Perquin et al. 2021). While the current study aims to further characterize the effect orientation has on the perception of length, this finding focused our research to the vertical and horizontal axes.

Investigation into the perception of one UH line showed that stimuli shor- ter than 3 cm were more error prone and not well perceived (Rakkolainen et al. 2019). These results motivated the use of line lengths of 3, 5, and 7 cm. Focusing on more than one line, as we do here, is important when conside- ring designing usable interfaces that often include complex line combinations and orientations along the X, Y, and Z axes. While several other studies have examined the accuracy of a user’s perception of varying ultrasonic feedback patterns, to our knowledge, no study has assessed a user’s ability to perceive the difference in length between two (or more) lines, and how the axis on which they are drawn affects this perception. As accurate perception of the extent of oriented lines is fundamental to understanding graphical content, the results of the current study will provide important insight as to how this new haptic interface can be optimized to support the building blocks of many mid-air non-visual interactions.

Our approach and experimental hypotheses were informed by traditional haptic studies investigating perception of embossed physical lines. Based on results that haptic perception improves with increased line length (Palani et al. 2020), we predicted that haptic lines would be more perceptible beyond the 3 cm threshold. Furthermore, it was hypothesized that vertically oriented lines would demonstrate increased accuracy vs. horizontal line perception based on the horizontal-vertical illusion, which consistently shows that horizontal lines are perceived as shorter than vertical lines (Surber et al. 2017).

METHODS

Participants Fifteen participants (NMale = 10, MAge = 22.8, SDAge = 4.9) were tested on their perception of the relative length of two UH lines. Participants, all of

whom were right-handed, were asked to use their dominant hand throughout the duration of the experiment. No participants self-reported any sensation loss in their dominant hand. All participants gave informed consent, and the study was approved by the Institutional Review Board at The University of Maine. Participants were blindfolded throughout the experiment to ensure that vision was not biasing their results.

Procedure This study implemented a within-subjects design with two conditions: esti- mating lengths of two parallel lines oriented along the (1) horizontal axis and (2) vertical axis. Line lengths of 3, 5, and 7 cm were used within each condition. Each orientation condition consisted of 18 trials, totaling 36 tri- als in the experiment for each participant. In each trial, two UH lines were presented one after another on the participant’s right palm with the varying lengths above. The apparatus used to render the UH stimuli was an Ultra- Leap STRATOS array (www.ultraleap.com). The framework [A.1] prevents the movement of the array and ensures the participant’s hand is stable throu- ghout the experiment. A speaker was placed next to the framework to play white noise throughout the experiment to mask any auditory cues from the device (Howard et al. 2019).

The experiment employed two phases for each orientation: the practice phase and the experimental phase. While each experimental phase was imme- diately preceded by its corresponding practice phase, the block order of vertical or horizontal line presentation alternated between participants, with line length trials randomized within each block.

Practice Phase Participants began by feeling the full rendering pattern [A.2] while viewing the user-interface that displayed the route of the pattern in real time. Partici- pants were then asked if both ultrasonic lines fit comfortably on their hand and if they understood the rendering pattern. If the signal did not fit com- fortably on their hand, they were instructed to reposition their hand for best perception. Once participants indicated that they could readily perceive both lines and that they understood the rendering pattern, they were instructed to pull down their blindfold to start the criterion test. This criterion test was performed to ensure the participant could non-visually distinguish a line ren- dered on the left vs. right side (or top vs. bottom) of their hand and vice versa. Additionally, the test was a form of training, allowing the participants to fur- ther familiarize themselves with the system before the experimental trials. In the criterion test, an ultrasonic line was rendered on one side of their palm. The test consisted of 10 trials, with an accuracy score requirement of 70% to move on. If a participant failed, the experiment would have concluded — however, all 15 participants passed the criterion test. The experimental phase began once participants passed the criterion test.

Experimental Phase Each orientation consisted of 18 trials, totaling to 36 trials per participant.

The Design of Mid-Air Ultrasonic Haptic Interfaces Based on the Perception of Lines 21

For each of the 18 trials, two lines were alternately presented, centered on the palm along the specified axis. The length difference setting, which was randomly selected for each trial, determined the lengths of the lines (3, 5, or 7 cm). All lines had a width of 0.86 cm, which was dictated by the har- dware’s cone of ultrasonic radiation. The appropriate UH line setting was presented on the participant’s palm using the specific rendering pattern [A.2]. Once the rendering pattern concluded, participants were asked which of the presented lines was longer, or whether they were the same length. The rese- archer recorded the participant’s response in a Qualtrics survey, along with the accurate line lengths for that trial. There was no time limit for the partici- pant to feel the stimulus and answer the prompt. Following the participant’s assessment of the relative lengths, the next pair of lines were presented. Upon completing the first orientation condition, the participants were prompted to remove their blindfold for a break before the next orientation practice and experimental phase.

RESULTS

Four dependent measures were evaluated as a function of the two-condition experiment: Total accuracy, orientation accuracy, length differentiation accu- racy, and quartile accuracy.

Total Accuracy The average accuracy for all 15 participants, independent of orientation, line length difference and quartile, was 50.9%, meaning the average number of correct trials per participant was approximately 18 out of 36. A one-sample t-test (p < 0.0001) was performed to determine whether participants perfor- med better than chance, which signified that participants’ haptic perception of the ultrasonic signal enabled them to clearly perceive the line length dif- ference with a significantly better accuracy than random guessing (or 33% accuracy). However, the total accuracy throughout this study showed that participants can only perceive the length comparison between two UH lines correctly approximately half the time, illuminating the need for further design improvements of the UH system.

Orientation Accuracy Each orientation dataset included 270 trials, 18 trials across 15 participants. The total number of correct trials within the horizontal orientation condition was 139, representing a total average accuracy of 51.5%. The total number of correct trials within the vertical orientation condition was 136, representing a total average accuracy of 50.4%. These findings suggest that line orien- tation, i.e., whether the UH lines were rendered vertically or horizontally on the hand, had no statistically reliable effect on accurate perception of line length. This null hypothesis was assessed using a chi-square test (χ2 [1, N = 540] = 0.27, p = 0.61), providing empirical support that orientation along the vertical or horizontal axis has no statistically reliable differential effect on human perception of line extent, at least for the lengths evaluated in this study.

Length Differentiation Accuracy It was hypothesized that performance would improve as the difference in line length between the two lines increased. Data shown in Table 1 supports this hypothesis. As the length difference of the two lines increases, the accuracy at which participants can accurately perceive these differences correspondingly increases. A chi-square test (χ2 [2, N = 540] = 23.51, p < 0.001) validated the hypothesis that length difference magnitude reliably influences human perception of the relative lengths of two UH lines. Specifically, performance reliably improved when the length difference magnitude was 4 cm, compared to a 2 cm and 0 cm length difference.

Quartile Accuracy Learning effects have often been found with interactions between humans and novel interfaces (Plaza et al. 2010). In support of this historical trend, our findings, shown in Table 2, demonstrate that participants’ performance improved as the experiment progressed through each quartile, or in other words, each 9 trials. The learning effect, and by extension cognitive effort,

Table 1. Average accuracy values for various orientation and length difference combinations.

Table available in the original PDF paper.

Table 2. Average accuracy values for the four quartiles of the experiment.

Table available in the original PDF paper.

The Design of Mid-Air Ultrasonic Haptic Interfaces Based on the Perception of Lines 23

decreased on average for each quartile of the experiment. A chi-square test (χ2 [3, N = 540] = 11.40, p = 0.01) indicated that there was a clear learning effect in this study, and that participants’ performance improved, independent of the line length or orientation, as they progressed through the study.

DISCUSSION

The integration of ultrasonic haptics into commercial applications is reli- ant upon the ability for users to accurately and intuitively interact with the system. By designing interfaces with effective guidelines in mind, this tech- nology has the potential to benefit the automotive (Harrington et al. 2018, Shakeri et al. 2018), healthcare (Ma et al. 2011), accessibility (Bhardwaj et al. 2020), and entertainment industries (Shen et al. 2022). For example, the use of gesture control in vehicle infotainment systems has been propo- sed as a solution for reducing visual demand, and therefore, reducing the risk of accidental collision. One challenge for these systems is that they have been found to alter users’ intended gesture controls, likely owing to the abse- nce of physical tactile feedback (Georigiou et al. 2017). The implementation of intuitive ultrasonic haptic feedback within gesture-controlled dashboards can ease the visual demand required and allow the driver to keep their eyes on the road. UH interfaces have also been proposed as a solution for redu- cing the risk of cross-contamination with public touchscreens (Battista et al. 2021). By developing effective touch-free displays, users are able to interact with these systems with a reduced risk of acquiring or transmitting contagi- ous diseases, therefore promoting improved public health. Furthermore, the entertainment industry is another area where this technology can promote innovation. UH technology can provide a new level of immersion and inte- raction within video games and movies. For example, the user could feel a pulse or vibration when their in-game character is hit, or audio tactile stimuli can be integrated into movies to enhance the viewing experience by allowing the viewer to feel the action and excitement of the film (Ablart et al. 2017). While these potential use-cases for UH technology are exciting, they are cur- rently constrained by the lack of perceptual accuracy by users. The guidelines below, derived from the results of this study, will encourage functional and user-friendly UH interfaces, specifically for ones that employ basic line-based geometric primitives.

Line Orientation Results from this study indicate that the orientation of ultrasonic haptic lines has no effect on a user’s ability to accurately determine their relative lengths. In other words, UH lines can be oriented in either the vertical or horizon- tal orientation without sacrificing perceptual salience. This guideline opens further design customizability within UH interfaces. For example, touch-free automotive dashboards can implement haptic lines to signify the magnitudes of different settings like audio volume or climate temperature. The finding that orientation has no effect on salience enables designers to give the user the option to decide whether they want the lines along the horizontal or vertical

axis, depending on the user’s preference. Giving users the ability to customize their preferred line orientation will further promote user-friendly interfaces and the adoption of this new technology.

Line Length Distinction Results from this study indicate that line length distinction significantly improved when the length difference between two lines is at least 4 centi- meters. This least noticeable difference is essential to consider if an interface requires users to differentiate between the lengths of geometric building blocks. This guideline will ensure that users do not misinterpret the repre- sentation of a UH signal, further promoting functional UH interface design. By innovating upon current ultrasonic haptic technology to allow for greater intensity of UH signals, the least noticeable length difference has potential to decrease. When participants were asked if a more intense signal would improve their performance, 80% of participants agreed, giving rise to an avenue for further research.

Learning Curve A clear learning curve was evident when evaluating a new user’s ability to perceive ultrasonic signals in this study. Other studies with novel haptic interfaces have shown more time to learn is needed compared to traditional physical touch or vision (Palani et al. 2020). We posit that the novelty of these UH interfaces gave rise to increased cognitive load and attentional demands for the user to master than would be expected from the same tasks using tra- ditional embossed tactile graphics. Although participants were trained before the experiment, the learning effect throughout the study suggests that more robust training is needed. Rigorous training tutorials should be implemented within UH interfaces to eliminate the presence of a steep learning curve and further promote user intuitiveness.

CONCLUSION

Functional UH user-interfaces (UIs) and their applications have the potential to impact several industries. The integration of this technology relies upon further research into the human perception of UH lines, a basic building block of many geometrical elements and UI components. The purpose of this study was to investigate users’ perception of the relative length of two UH lines presented on the user’s palm. Evidence indicated that: (1) the ori- entation of the lines had no effect on a user’s accuracy when determining their relative lengths, (2) line length distinction significantly improved when the length difference between those lines was at least 4 cm, and (3) a clear learning curve was evident when evaluating a new user’s ability to perceive ultrasonic signals. These findings yield a set of parameters that promote greater design customizability, improved haptic perceptibility, and increa- sed intuitiveness of interfaces that utilize UH technology. The capabilities and limitations of UH technology identified and discussed within this study will help engineer user-friendly and functional touch-free haptic interfaces for

The Design of Mid-Air Ultrasonic Haptic Interfaces Based on the Perception of Lines 25

commercial applications and inform future research for this promising new haptic interaction style.

ACKNOWLEDGMENT

Special thanks to the staff at the VEMI Lab for supporting this research. Toyota Research Institute provided funds to support this work, which was also partially funded by NSF grant CHS-1910603 to N.A. Giudice.

REFERENCES

Ablart, D., Frier, W., Limerick, H., Georgiou, O., Obrist, M., 2019. Using Ultrasonic Mid-air Haptic Patterns in Multi-Modal User Experiences. pp. 1–6. Ablart, D., Velasco, C., Obrist, M., 2017. Integrating Mid-Air Haptics into Movie

Experiences, in: Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video, TVX ‘17. Association for Computing Machinery, New York, NY, USA, pp. 77–84. Battista A, Nicolaides C, Georgiou O. Modelling disease transmission from touchscreen user interfaces. R Soc Open Sci. 2021 Jul 28. Bhardwaj, A., Chae, J., Noeske, R. H., Kim, J. R., 2021. TangibleData: Interactive

Data Visualization with Mid-Air Haptics, in: Proceedings of the 27th ACM Sym- posium on Virtual Reality Software and Technology, VRST ‘21. Association for Computing Machinery, New York, NY, USA, pp. 1–11. Carter, T., Seah, S. A., Long, B., Drinkwater, B., Subramanian, S., 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces, in: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST ‘13. Association for Computing Machinery, New York, NY, USA, pp. 505–514. Chongyang, S., Nai, W., Xiaoying, S., 2019. Tactile sensitivity in ultrasonic haptics:

Do different parts of hand and different rendering methods have an impact on perceptual threshold? Virtual Reality & Intelligent Hardware 1, 265. Hajas, D., Pittera, D., Nasce, A., Georgiou, O., Obrist, M., 2020. Mid-Air Haptic Rendering of 2D Geometric Shapes With a Dynamic Tactile Pointer. IEEE Transactions on Haptics 13, 806–817. Harrington, K., Large, D. R., Burnett, G., Georgiou, O., 2018. Exploring the Use of

Mid-Air Ultrasonic Feedback to Enhance Automotive User Interfaces, in: Proce- edings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ‘18. Association for Computing Machinery, New York, NY, USA, pp. 11–20. Howard, T., Gallagher, G., Lécuyer, A., Pacchierotti, C., Marchal, M., 2019. Investigating the Recognition of Local Shapes Using Mid-air Ultrasound Haptics, in: 2019 IEEE World Haptics Conference (WHC). Presented at the 2019 IEEE World Haptics Conference (WHC), pp. 503–508. Ma, M., Zheng, H., 2011. Virtual Reality and Serious Games in Healthcare, in:

Brahnam, S., Jain, L. C. (Eds.), Advanced Computational Intelligence Paradigms in Healthcare 6. Virtual Reality in Psychotherapy, Rehabilitation, and Assessment, Studies in Computational Intelligence. Springer, Berlin, Heidelberg, pp. 169–192. Palani, H. P., Fink, P. D. S., Giudice, N. A., 2020. Design Guidelines for Schematizing and Rendering Haptically Perceivable Graphical Elements on Touchscreen Devices. International Journal of Human–Computer Interaction 36, 1393–1414. Perquin, M. N., Taylor, M., Lorusso, J., Kolasinski, J., 2021. Directional biases in

whole hand motion perception revealed by mid-air tactile stimulation. Cortex 142, 221–236.

Rakkolainen, I., Sand, A., Raisamo, R., 2019. A Survey of Mid-Air Ultrasonic Tactile Feedback, in: 2019 IEEE International Symposium on Multimedia (ISM). Pre- sented at the 2019 IEEE International Symposium on Multimedia (ISM), IEEE, San Diego, CA, USA, pp. 94–944. Shakeri, G., Williamson, J., Brewster, S., 2018. May the Force Be with You:

Ultrasound Haptic Feedback for Mid-Air Gesture Interaction in Cars. pp. 1–10. Shen, V., Shultz, C., Harrison, C., 2022. Mouth Haptics in VR using a Headset Ultrasound Phased Array, in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ‘22. Association for Computing Machinery, New York, NY, USA, pp. 1–14.

APPENDIX

A.1 Hardware Apparatus

Figure 1: This figure displays the hardware apparatus used by the participant to receive the ultrasonic haptic signal.
Original PDF Figure 1:

A.2 Orientation Rendering Pattern

Figure 2: This figure displays the rendering technique of the ultrasonic signal in the vertical and horizontal orientation from the bottom view. The first line is presented twice followed by the second line being presented twice. Within each iteration, pul- ses were implemented to ensure the participant could perceive the ends of the lines. The first pulse, indicating the start of the rendering, pulsed once. The second pulse, indicating the end of the rendering, pulsed twice.
Original PDF Figure 2:
Cover of the proceedings volume Human Factors and Systems Interaction

Explore the complete digital volume

Human Factors and Systems Interaction

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-60-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →