Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Development of Interactive System for the Visually Impaired using Tactile Sensations and Gestures

Shibaura Institute of Technology, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe100592

Abstract

Even though previous studies have confirmed the superior abilities of the visually impaired to identify tactile sensations, few systems use such abilities. In this study, we developed and evaluated an interactive system that utilizes the superior abilities of the visually impaired by adding active elements of gestures to increase their enjoyment. We employed the TECHTILE toolkit as a device to give tactile sensations to players and Kinect for Windows as a device to recognize their gestures and developed an interactive system with different types of tactile sensations and corresponding gestures. We experimentally evaluated our system.

INTRODUCTION

We previously developed game systems for the visually impaired (Ishii 2006, Takahashi 2011), one of which uses the TECTILE toolkit to generate tactile sensations (Ito, 2012). However, since strong demand exists for active physical exercise for the visually impaired, we developed a more active system by incorporating gestures into it. Our interactive system utilizes the superior abilities of the visually impaired with additional active elements of gestures to increase their enjoyment. This article describes the construction and evaluation of our system.

SYSTEMS DESIGN We used a Kinect sensor as an input device for PCs because it can detect human gestures using infrared sensors. We employed tactile sensations and sound for the feedback of the game information. We used the TECHTILE toolkit for the tactile sensations (Minamisawa). When using the system, players stand in front of the Kinect sensor and the speakers and hold a hand-made device in their left hand for the tactile sensation output. The following is our system flow:

1. The PC outputs tactile sensations thorough the TECHTILE toolkit. 2. The PC outputs sound through speakers. 3. A player makes a gesture that corresponds to the tactile sensation. 4. The Kinect sensor detects the player’s joint positions. 5. The PC receives information from the Kinect sensor and processes it.

Fig. 1 Diagram of system

Table 1 Enemy sounds and relevant gestures Game situation Tactile sensation Relevant gesture Pot seems to fall Shaking Raise right hand Device is attacked by A ball turns inside Put right hand on the device Floor is wobbly A thing is grind Raise right knee Ghost is near Air bubbles are broken Squat

Game Content The game content is shown in Fig. 2.

Fig. 2 Game’s flowchart

Preliminary Experiment We made a prototype and experimentally evaluated it with six sighted participants. We took log data during the game and asked the following questions:

・Did you enjoy this game?

・Do you want to play it again?

・Are the tactile sensations and gestures related naturally?

Our analysis of our experimental results confirmed that our participants enjoyed the game. Fig. 3 shows the relation between the tactile sensations and the gestures. The sensation which shows that the device was attacked by enemy was judged natural, but not the others. Fig. 4 shows the game’s degree of difficulty. The participants felt that it was difficult.

Fig. 3 Evaluation of naturalness between tactile sensations and relevant gestures

Fig. 4 Evaluation of naturalness between tactile sensations and relevant gestures

Improvement of System 1 Based on the results of our preliminary experiment, we made the following improvements:

・Revised the relations between tactile sensations and game situations ・Added exercise time ・Carefully designed the tutorials

We revised the relations by changing the tactile sensation from “Air bubbles are broken” to “Cellphone vibration.” We also changed the game situation from “The floor is wobbly” to “A monster is crawling on the floor” and “The pot seems to fall from the top right corner direction.”

Table 2 Revised tactile sensations and relevant gestures

Evaluation Experiment 1 We experimentally evaluated our system by employing eight visually impaired participants. We took log data during the game and asked the following questions:

・Did you enjoy this game?

・Would you want to play this game again?

・Are the tactile sensations and gestures related naturally?

Figure 5 shows the evaluation results about whether the game was enjoyable. No subjects felt that it was unenjoyable. Fig. 6 shows the evaluation results of the question about difficulty. Many thought that it was difficult. Fig. 7 shows the evaluation results of the question about the relations between the tactile sensations and the situations they represented. The ratio of the answers showing naturalness increased for “The ghost is near” and “A monster is crawling on the floor.”

Fig. 5 Evaluation of enjoyability

Fig. 6 Evaluation of difficulty

Fig. 7 Evaluation of naturalness of high knee to relevant enemy sound

Improvement of System 2 Based on the results of evaluation experiment 1, we made the following improvements:

・Changed the game so that the number of tactile sensations increases one by one. ・Added background music and sound effects. ・Added a physical strength parameter.

If a player fails the gesture, her physical strength decreases. When her physical strength is depleted, the game is over.

Fig. 8 New game’s flowchart

Evaluation Experiment 2 We made a prototype and experimentally evaluated it by employing eight sighted participants(eight teenage boy). We took log data during the game and asked the following questions:

・Did you enjoy this game?

・Do you want to play it again?

Figure 9 shows the evaluation results about whether the game was enjoyable. No subjects felt that the game was unenjoyable. We also obtained the following opinions: ・The game’s progress was monotonous.

・This game is too short.

・I had difficulty distinguishing the differences among the tactile sensations.

Fig. 9 Evaluation of enjoyability

Improvement of System 3 Based on the results of evaluation experiment 2, we made the following improvements:

・Revised the content in which two tactile sensations were added to the last cycle.

・Edited the sound sources to increase the loudness of the tactile sensations.

Evaluation Experiment 3 We experimentally evaluated our system by employing six sighted and eight visually impaired participants. We took log data during the game and asked the following questions:

・Did you enjoy this game?

・Do you want to play it again?

Figure 10 shows the evaluation results about whether the game was enjoyable. No subjects felt that it was unenjoyable. Fig. 11 shows the evaluation results of the question whether they wanted to play it again; many do. Fig. 12 shows the evaluation result of the question about the tutorial’s length. We found a difference between the sighted and the visually impaired.

Fig. 10 Evaluation of enjoyability

Fig. 11 Evaluation of want to play game again

Fig. 12 Evaluation of tutorial length

CONCLUSIONS

We developed a game system for the visually impaired that utilizes gestures and tactile sensations. We evaluated it on the basis of experiment results and obtained the following: ・Our game system is enjoyable for both the visually impaired and the sighted.

・The visually impaired accepted longer tutorials than we expected.

Since we only did evaluation experiments on young generations, we need to improve the game for wider generations.

REFERENCES

Ikegami, Y., Ito, K., Ishii, H. and Ohkura, M.: Development of a Tracking Sound Game for Exercise Support of Visually Impaired, Human-Computer Interaction, Part II, HCII2011, LNCS 6772, 31/35, Orlando, July.12, 2011.

Cover of the proceedings volume Advances in Affective and Pleasurable Design

Explore the complete digital volume

Advances in Affective and Pleasurable Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2109-8

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →