Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Towards Reliable Tactile Mid-Air Interfaces: Analysis of Influencing Factors of the Perception of Tactile Mid-Air Feedback

Chemnitz University of Technology, Chair for Ergonomics and Innovation, Erfenschlager Straße 73, 09125, Chemnitz, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1002760

Abstract

Human-machine interfaces require an efficient and reliable interaction under various con-ditions. Especially under conditions with high cognitive workload which require rapid situa-tion assessment, interfaces should reliably support the human perception. Here, research addresses gesture-based interfaces as a possible interface to enable intuitive interactions based on ingrained daily routines. Using spatial commands executed by the bare hand, one can recreate real world interactions like pushing a knob, turning a controller, or using a ges-ture as an input command. In comparison to real input devices gesture-based interfaces lack haptic feedback for the user. Tactile feedback is important as it indicates for example interaction borders (e. g. edge of the mouse pad) or provides feedback of a successful in-teraction (e. g. sensation of the pushed button). This helps to increase the usability of mid-air gestural systems. For the realisation of mid-air tactile feedback two technologies can be considered. Vortex-generators and ultrasound-based feedback utilise the bare hand and need no device attached to the hand. However, they provide weak feedback which can be influenced by airflow, hand posture, clothing, workload or other factors. To achieve the aforementioned benefits, users have to reliably perceive the tactile feedback – and to do so, perception of tactile mid-air feedback needs to be researched in more detail.We present a method for the analysis of influencing factors on the perception. A driving simulator was the basis for a standardised apparatus in which tactile feedback was pre-sented via a vortex-generator. For each influencing factor, participants were asked to do a driving task (Lane Change Task - LCT) and detect tactile stimuli in parallel. By the help of the method of constant stimuli, a psychometric function for each influencing factor was derived. On this function detection, thresholds of 50%, 90%, 95% and 99% were chosen to represent the most important values in terms of human-machine interaction. In compari-son to widely used methods like staircase procedures, this approach promises to give a better insight into the effects of the influencing factors as the whole psychometric func-tion can be analysed instead of one distinct value.Two experiments (N= 80; 31) were conducted to apply the approach and analyse the influ-ence of workload by a variation of speed and secondary task type. Also, one experiment (N= 16) investigated the influence of hand temperature on the perception of the feedback. Results show that increased speed and the addition of a secondary tasks significantly in-crease the perceived workload. Regarding the perception of tactile stimuli, slight differ-ences for different workload conditions and a cut-off for high workload conditions were found. Furthermore, the effect of temperature on the perception on tactile feedback could be shown. Based on the studies, advantages and disadvantages of the proposed approach are dis-cussed. Also, the impact of workload and temperature in terms of design recommenda-tions for human-machine interaction are examined. The presented approach suggests a promising method to investigate the impact of influencing factors on specific design ele-ments for human-computer interaction. Further studies should investigate the eligibility for other modalities and applications.

PERCEPTION OF MID-AIR TACTILE FEEDBACK Whereas first research papers on mid-air tactile feedback had no specific field of application or could be considered within game-based applications, research states that possible safety critical applications can profit a lot by them. For example, in-vehicle interfaces, medical applications or control rooms will comprise critical decision tasks in complex situations. Here, the combination of visual and tactile feedback is highly efficient and can be supe- rior in comparison to visual- and acoustic feedback (Burke et al., 2006). However, requirements in terms of robustness of interaction are higher for these applications, thus the impact of possible influencing factors should be considered.

© 2022. Published by AHFE Open Access. All rights reserved. 419

420 Bernhagen and Bullinger

For the reintegration of mid-air tactile feedback into gesture-based intera- ction, two common technologies can be considered. Vortex-generators (Sodhi et al., 2013) and ultrasound-based feedback (Friedrich et al., 2022) utilize the bare hand and need no further hardware attached to the hand. The more sophisticated technique, ultrasound feedback, causes noticeable vibrations of the human skin by oscillating the skin itself. One disadvantage is the limited interaction distance and the limited ability to provide a complete coverage of the gestural input space (Rakkolainen et al., 2020). Vortex-generators pro- duce vortex rings which travel to the target and produce a noticeable impulse on the skin. These systems can follow spatial hand movements and provide feedback for distances above one meter (Sodhi et al., 2013). Disadvantages of current systems are loud noise, delay between the contraction of the spea- ker membranes and the impact on the hand as well as reliability of the target deviation.

In comparison to ultrasound feedback, vortex-generators do not produce steady but discrete sensations. Both techniques have in common that they generate weak feedback which is comparable to a whiff. For the perception of these tactile signals, multiple mechanoreceptors within the human skin get stimulated (Lederman, Klatzky, 2009). Each one is specialized on different parameters of the feedback (e.g. strength, frequency and area size). Vortex- generators produce a low frequency feedback which is perceived by Meiss- ner’s corpuscles (SA I). Ultrasound otherwise incorporate Pacinian corpuscles (FA II) for the detection of high frequency vibrations (40 – 400 Hz).

In order to ensure the effectiveness of the mid-air tactile interaction, dete- ction thresholds under real tasks need to be analyzed. Here, tactile feedback needs to be perceived under diverse conditions. Especially temperature and the angle of impact for vortex-rings might be influencing factors for the perce- ption of mid-air tactile feedback. As a characteristic for safety critical decision tasks, workload might affect the perception of tactile feedback as well. Tactile feedback promises to beneficial for heavy workload tasks and so a variation of task type and number of tasks should be taken into consideration for an impact analysis.

To study whether one of the factors influences the perception, psychophy- sical methods are an established approach (Treutwein, 1995). To estimate thresholds, just noticeable differences or psychometric functions three meth- ods can be applied. In application of the method of constant stimuli, well distinguishable points on the stimulus domain are presented repetitively. By statements of the subject whether the signal was recognized or not, the pro- bability of the signal perception can be derived. For the method of limits the experimenter increases or decreases the stimulus level in small steps. At the predefined limits of a stimulus interval, they reverse the steps. For each sti- mulus the participant has to state whether it was smaller than, equal to or larger than the standard. Finally, the method of adjustment also sets limits for the stimulus variation. However, here the change of direction of stimulus values occurs when the participant is close to the point of subjective equiva- lence (PSE) to repeatedly lower the range between not perceivable and barely perceivable stimuli. Deficits of these original methods are the loss of control of intraindividual decision criteria, bias and data economy. To overcome these

Towards Reliable Tactile Mid-Air Interfaces: Analysis of Influencing Factors 421

drawbacks several adaptive methods were developed (e.g. Linschoten, et al., 2001; Snoeren & Puts 1997; Lesmes et al., 2006). However, most of the adaptive methods have been used in settings for visual and acoustic cues (Linschoten, 2001; Hatzfeld, 2013) and only few researchers adopted it to haptic or tactile perception. They base on assumptions on user behavior in a highly controlled laboratory settings with a minimum of environmental stimuli, which are not validated for applied research.

To answer the question of the impact of influencing factors on perception the complete psychophysical function is of interest. Especially in terms of human factors research, not only the point of subjective equality is of inte- rest, also does the human technology-interaction require highly perceivable stimuli. Therefore, perception rates of 50%, 90%, 95% and 99% should be compared for each influencing factor. As a result, we chose the method of constant stimuli to derive the necessary data.

METHOD

To analyze the influence of workload by a variation of the secondary task type, one study with 32 participants (13 females, 19 males) was conducted. Here, the average age was 28,03 years (SD = 4.83; range: 21-38) and the mean driving milage 8398.44 km (SD = 8676.24). A second study (N = 15; 5 females, 10 males) investigated the influence of hand temperature on the perception of the feedback. In comparison to the first study the average age of the participants was lower (M = 23.4 years; SD = 5.91; range: 20-38) with a mean annual driving milage of 9864 km (SD = 11627.09).

Participants had to conduct the lane change task (Mattes, 2003) within a driving simulator as a main task. The LCT was developed to serve as a reliable, valid and objective test to measure cognitive workload and driver distraction. Participant’s task is to drive on a straight with three lanes. Signs on constant distances indicate which lane the driver has to follow until the next sign appears. After a straight with a predefined length, a 90-degree bend ends a section and the next straight with signs appears. Within the bends no tactile signals were presented to ensure a constant workload. Participants were asked to say yes, if they noticed a tactile stimulus.

For the production of the mid-air tactile signal a self-made vortex- generator was build. The design is based on the works of Sodhi and colleagues (2013). Here, audio speakers displace air within an encased 3D-printed body (Figure 1, right). Using speakers allows for a fast response of the system and a high degree of control over feedback frequency and strength. The generator base, air chamber and nozzle were 3D-printed using PLA filament. We used a DC motor driver and the Arduino Uno platform to generate the signal for five air displacing speakers. Using a DC signal with slow rising flanks, allo- wed us to remove two audible sounds of the speaker membrane actuation, leaving only one audible pop. Due to the air compression when the spea- ker actuates, the only exit of the air is the nozzle of the generator. Here, at the circular end, an air vortice is created. The semi-stable circular vortice is able to travel a distance of at least one meter until it dissolves. To use the maximum available force and limit the sound effects the generator is placed

422 Bernhagen and Bullinger

Figure 1: Driving simulator (le.) and vortex-generator (re.).
Original PDF Figure 1:

close to the hand on a constant distance of 7.5 cm. Air vortices were directed on the middle of the hand’s palm.

During the experiment participants wore headphones to prevent the reco- gnition of noises of the generator, which may give hints to the participants when a feedback signal was generated. For the generation of a randomized feedback, we used the software MathWorks MATLAB R2020a. The program generated the signals and pauses in a randomized order for each participant and condition. Afterwards an audio file was generated and transferred to the speakers. The second important apparatus of the study was the mobile dri- ving simulator, which consists of a fully adjustable chair, instrument cluster, wheel, pedals and a 60 inch in diameter screen (Figure 1, left).

In the first experiment we used a 2x8 factor design. Participants were told to drive with the maximum speed of 90 km/h. They experienced two diffe- rent conditions within two temporal separated sessions. In the first session of study one, participants conducted the lane change and the perception task (Cond A). In session two an additional task was given, where participants had to read out a number which was displayed on a screen within the instru- ment cluster (Cond B). To measure the perception rate, eight different (six in study two) feedback intensities were presented eight times per session. For the second study a 3x6 factor design was used to compare different hand tempe- ratures. By the help of warming and cooling pads we changed the hand skin temperature to: cold (M = 24.23 °C, SD = 1.2 °C) and warm (M = 37.52 °C, SD = 1.43 °C). The regular temperature (M = 31.13 °C, SD = 1.55 °C) was used as the baseline. Participants had to conduct the same lane change task as described in study one (without an additional task). Every factor appea- red in randomized order. Also, participants were asked to always start with a baseline drive to get to know the lane change task and the driving simu- lator. Thus, learning affects could be reduced and a baseline for participants workload levels could be generated.

For the measurement of mental workload, the questionnaire Driving Acti- vity Load Index (DALI) was used (Pauzié, 2008). This questionnaire bases on the NASA TXL questionnaire and is adopted to the requirements of driving

Towards Reliable Tactile Mid-Air Interfaces: Analysis of Influencing Factors 423

Table 1. Parameters and confidence intervals for the fitted psychometric curve of study one.

Table available in the original PDF paper.

α α - 95% CI β β - 95% CI SSE R2 Cond A 1.461 1.404, 1.518 0.3015 0.2482, 0.3549 8.433 0.7585 Cond B 1.592 1.517, 1.668 0.3449 0.275, 0.4147 12.15 0.6944

Table 2. Human factors related thresholds for task with medium and high cognitive workload.

Table available in the original PDF paper.

50% 90% 95% 99%

Cond A [mN] 1.461 2.124 2.349 2.844 Cond B [mN] 1.592 2.351 2.609 3.178

studies. On a five-point likert scale, participants can assess their attention, auditive and temporal demand, visual, stress and the interference. Immedia- tely after the completion of each condition the questionnaire was handed out to the participants. To limit the influence of the factor age on the tactile perception task (Woodward, 1993; Godde et al., 2018), we only invited participants with a maximum age of 40 years. Furthermore, possible modera- ting factors like profession, free time activities, known diseases and personal driving milage were asked within a questionnaire.

Before each study started, a pre-test with three participants was conducted. Here, the noise level of the vortex-generator was validated to ensure that no acoustic signals give additional hints. Also, stimulus interval and stimuli steps were estimated and set.

RESULTS

Both studies were analyzed using MathWorks MATLAB R2020a including the curve fitting tool. The following equation and the method of nonlinear least squares was used to fit the data and represent a logistic function (Treu- twein, 1995). Parameter α is the position parameter (PSE) and β presents the slope, respective the spread.

Mathematical equation reproduced from the original PDF
Equation reproduced from the original PDF.

Table 1 and figure 2 show the results of the nonlinear fit and visualizes the psychometric functions for the influencing factor workload in study one.

In the figure the fitted functions for the conditions separate and show a different course. However, the confidence intervals overlap especially until the PSE. The corresponding percentiles for each function are presented in table 2. Data bases on the fitted functions without respecting confidence bounds.

To show whether the additional tasks impact the workload, a univa- riate analysis of variants has been conducted for the comparison of the DALI-scores of three workload conditions (test drive, Cond A and Cond B).

424 Bernhagen and Bullinger

Figure 2: Psychometric functions with 95% confidence interval for driving task (Cond A) and driving plus secondary task (Cond B).
Original PDF Figure 2:
Figure 3: Psychometric functions for regular, warm and cold skin temperature.
Original PDF Figure 3:

The item effort of attention showed significant differences over all three con- ditions (test drive M = 2.64, Cond A M = 3.30 and Cond B M = 4.42; F(2.64) = 46.87, p < .001, partial η2 = .59). Also, Bonferroni-corrected pai- rwise comparisons showed significant differences between each condition. Finally, the item interference evaluates the added disturbance by additional tasks (here perception and number recognition task). Due to the absence of interference in the test drive, the calculations were done between condi- tion A (M = 2.45) and B (M = 4.15). Results from Huyn-Feldt-test show a significant difference (F(1.00,32.00) = 92.24, p < .001; partial η2 = .74).

For study two we utilized the same logistic function and fitted the data by the help of the nonlinear fit. For better visibility no confidence bounds are shown for psychometric functions in Figure 3. Table 3 shows the correspon- ding parameters.

Towards Reliable Tactile Mid-Air Interfaces: Analysis of Influencing Factors 425

Table 3. Parameters and confidence intervals for the fitted psychometric curve of study two.

Table available in the original PDF paper.

α α - 95% CI β β - 95% CI SSE R2 Regular 1.437 1.400, 1.473 0.1345 0.09859, 0.1705 3.859 0.6047 Warm 1.405 1.361, 1.448 0.1548 0.1107, 0.1989 4.779 0.5346 Cold 1.502 1.449, 1.556 0.2026 0.1431, 0.2620 4.574 0.4781

Table 4. Human factors related thresholds for tasks with medium and high cognitive workload.

Table available in the original PDF paper.

50% 90% 95% 99%

Regular [mN] 1.437 1.733 1.838 2.096 Warm [mN] 1.405 1.745 1.860 2,116 Cold [mN] 1.502 1.945 2.098 2.437

Whereas all three functions show only minor deviations for the space below the PSE, the cold condition has noticeable lower detection thresholds. As a result, especially the values relevant for human computer-interaction differed between regular/warm and cold skin temperatures. In Table 4 these thresholds for the three conditions are shown.

DISCUSSION AND CONCLUSION Results in the first study show perception thresholds for the point of sub- jective equality of around 1.5 mN, which is similar to the findings of other researchers (Burdea, 1996; Cascio et al., 2008). Figure 2 shows that confi- dence intervals of both functions overlap especially for the lower part until approximately the PSE. Especially interesting is that for values important to human factor related questions, workload seems to affect the perception negatively. Data of the DALI questionnaire show significant differences for the perceived workload between both conditions and supports the finding that the lower perception rates of mid-air feedback was caused by high wor- kload. Overall, data has a high noise on the intraindividual level. In summary, the findings show that the detection of perception thresholds in an applied setting with a medium workload in condition A is on a similar level as stated by other researchers. Increasing the workload by a secondary task (condition B) decreases the perception thresholds. Further studies should evaluate the scope of the influencing factor for more than two workload levels.

Study two showed PSEs in the same range of study one. Here, results sup- port findings reported by other authors, as warm and regular conditions had similar psychometric functions (Gescheider et al., 1997; Wada et al., 1996). A visible difference could be found for cold conditions in comparison to regu- lar and warm. Hence, participants had worse perception rates when the skin was cooled down. However, in comparison to study one the slope (β-value) differed much. This effect is due to insufficient data acquisition for the upper perception levels. Even though the amplitude interval was set within the pre- test, participants rarely achieved perception levels above 90%. Due to that,

426 Bernhagen and Bullinger

the coefficients of determination are low compared to study one and describe only about 50% to 60% of the deviation.

Overall, the presented studies show that it is possible to replicate fin- dings for laboratory perception tasks within an applied research setting. Even though the methodological approach is old and not state of the art in terms of efficiency, it gives meaningful insights for applied human machine-interaction research. Especially, if it is combined with another study topic (e. g. usability study), a medium number of participants (in our research approx. 30) can produce a well fitted psychometric function including individual differences. Referring to the addressed research on mid-air tactile feedback systems, it can validate the perception thresholds of the system. Simultaneously studies on the signal design or the effectiveness in high workload conditions can be con- ducted with validate data of the perception. However, more emphasis should be placed on the number of trials for each stimulus value. As a result, intrain- dividual differences will be reduced and therefore individual comparisons can be conducted. By that, normal deviations for the perception performance of different participants can be reliably taken into consideration within data analysis. Following studies will use the updated methodology and compare the quality standards of threshold estimations: bias, precision and efficiency.

ACKNOWLEDGMENT

This research was partially supported by the Federal Ministry for Economic Affairs and Climate Action (project ‘KARIFF’, 16KN055773). The sponsor had no role in the study design, the collection, analysis and interpretation of data, the writing of the report, or the submission of the paper for publication.

REFERENCES

Burke, J. L., Prewett, M. S., Gray, A. A., Yang, L., Stilson, F. R., Coovert, M. D., ... & Redden, E. (2006). Comparing the effects of visual-auditory and visual- tactile feedback on user performance: a meta-analysis. In Proceedings of the 8th international conference on Multimodal interfaces (pp. 108–117). ACM. Burdea, G. C. (1996). Force and touch feedback for virtual reality. John Wiley &

Sons, Inc. Cascio, C., McGlone, F., Folger, S., Tannan, V., Baranek, G., Pelphrey, K. A., & Essick, G. (2008). Tactile perception in adults with autism: a multidimensio- nal psychophysical study. Journal of autism and developmental disorders, 38(1), 127–137. Friedrich, M., Bernhagen, M., Widmann, L., & Bullinger, A.C. (2022). Get in

touch with mid-air haptic feedback. Human centered design of feedback pat- terns. In GfA e.V. (Ed.), Technologie und Bildung in hybriden Arbeitswelten. 68. Frühjahrskongress der Gesellschaft für Arbeitswissenschaft e.V. GfA-Press. Geschneider, G., Thorpe, J. M., Goodarz, J., & Bolanowski, S. J. (1997). The effects of skin temperature on the detection and discrimination of tactile stimulation. Somatosensory & motor research, 14(3), 181–188. Godde, B., Bruns, P., Wendel, V., & Trautmann, M. (2018). Effects of age and

individual experiences on tactile perception over the life span in women. Acta Psychologica, 190, 135–141.

Towards Reliable Tactile Mid-Air Interfaces: Analysis of Influencing Factors 427

Hatzfeld, C. (2013). Experimentelle Analyse der menschlichen Kraftwahrnehmung als ingenieurtechnische Entwurfsgrundlage für haptische Systeme. Dr.-Hut-Verlag. Lederman, S. J., & Klatzky, R. L. (2009). Haptic perception: A tutorial. Attention,

Perception, & Psychophysics, 71(7), 1439–1459. Lesmes, L. A., Jeon, S. T., Lu, Z. L., & Dosher, B. A. (2006). Bayesian adaptive estimation of threshold versus contrast external noise functions: The quick TvC method. Vision research, 46(19), 3160–3176. Linschoten, M. R., Harvey, L. O., Eller, P. M., & Jafek, B. W. (2001). Fast and accurate

measurement of taste and smell thresholds using a maximum-likelihood adaptive staircase procedure. Perception & psychophysics, 63(8), 1330–1347. Mattes, S. (2003). The lane-change-task as a tool for driver distraction evaluation.

Quality of Work and Products in Enterprises of the Future, 57(60), 7. Pauzié, A. (2008). A method to assess the driver mental workload: The driving activity load index (DALI). IET Intelligent Transport Systems, 2(4), 315–322. Rakkolainen, I., Freeman, E., Sand, A., Raisamo, R., & Brewster, S. (2020). A survey

of mid-air ultrasound haptics and its applications. IEEE Transactions on Haptics, 14(1), 2–19. Snoeren, P. R., & Puts, M. J. (1997). Multiple parameter estimation in an adaptive psychometric method: MUEST, an extension of the QUEST method. Journal of Mathematical Psychology, 41(4), 431–439. Sodhi, R., Poupyrev, I., Glisson, M., & Israr, A. (2013). AIREAL. ACM Transactions

on Graphics (TOG), 32(4), 1–10. Treutwein, B. (1995). Adaptive psychophysical procedures. Vision research, 35(17), 2503–2522. Wada, C., Ino, S., & Ifukube, T. (1996). Dependency of the Skin Temperature on

the Threshold and the Quality of the Tactile Sensation. A Basic Study for Tactile Aids. Transactions of the Society of Instrument and Control Engineers, 32(11), 1566–1568. Woodward, K. L. (1993). The relationship between skin compliance, age, gender, and tactile discriminative thresholds in humans. Somatosensory & motor research, 10(1), 63–67.

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-44-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →