Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Verification of an alternative wheelchair control in a virtual environment

Hochschule Bremen – University of Applied Science, Bremen, 28199, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1005488

Abstract

This work presents a training simulator that is being developed in collaboration with electric wheelchair users and medical professionals. The main function of the software is to serve as a development platform for alternative control systems that enable people with paralysis to control a wheelchair themselves. The project developed two prototype control systems using technologies such as eye tracking and a brain-computer interface.

INTRODUCTION

The focus of the research presented here is an alternative electric wheelchair control. The control is verified and improved in a virtual environment (VE) in close collaboration with the target group. People with a spinal cord injury in the neck area (locked-in syndrome) are unable to control a wheelchair in the traditional way. Simple motor tasks, such as operating a joystick, are hardly or not at all possible for them.

The first alternative control developed for the project consists of a combination of several sensors: brain-computer interface (BCI), eye tracking (ET), voice control (VC) and obstacle detection (OD). The second is an eye- controlled virtual joystick. Before a real wheelchair is constructed using one of these or another technique, the controls mentioned are implemented in software and iteratively developed to take the specific needs of the target group into account. This software is meant to be a simulation where the controls can be tested and developed and later, it serves as a training environment.

In order to obtain meaningful test results, the VE contains a realistic residential area. This represents accessible houses, streets and sidewalks. In this way, the control can be tested in various everyday situations. The software operator has the option to activate obstacles, pedestrians or cars in the VE.

After the software is completed, the test environment serves as a training simulator. It should be used before a wheelchair user learns how to control the real wheelchair. The development takes place on a participatory basis

© 2024. Published by AHFE Open Access. All rights reserved. 289

290 Bulk and Tannert

with future users of the software. The focus is equally on wheelchair users and the software operators.

CHOOSING A CONTROL TECHNOLOGY In order to find a suitable control technology that takes into account the special requirements of the target group, various technologies are compared in this project and their usability is examined. The BCI procedures, eye- tracking and voice control in particular require a closer look.

There is already various research into BCI controls for wheelchairs. Different BCI measurement methods are used in this research: Steady-state Visual Evoked Potential (SSVEP) (Bastos, 2011; Chen et al., 2022), Motor Imagery (MI) (Bastos, 2011; Palumbo et al., 2021) and Face Expression (FE) (Rabhi, Mrabet and Fnaiech, 2018). What they all have in common is that they have the potential to control electric wheelchairs. With the help of these controls, people with a complete cross-section can gain the opportunity to control their wheelchair themselves.

The P300 technique is another BCI method that offers higher accuracy, less fatigue, and more commands than SSVEP (Puanhvuan et al., 2017). The method by Puanhvuan et al. (2017) uses the P300 method when obstacle avoidance and other safety features are activated. This leaded to an accuracy of 83.42% with the developed control. The latency was around 250 to 500 milliseconds. This accuracy and latency could be insufficient for safe movement on the road. In addition, an extra monitor must be mounted on the wheelchair, which the user has to look at to control it, which also applies to SSVEP. This could make it difficult to observe the surroundings.

The MI-BCI attempts a different approach, but has some drawbacks that make it unsuitable for this project. The process needs a lot of training and user experience to work as a controller. Users need to have the ability to do certain mental activities, such as picturing moving an arm, that match the wheelchair’s control commands. Not all users can acquire the necessary control over such a controller. Even experienced users who get along well with the procedure have highly fluctuating and therefore inaccurate success rates when executing mental commands. The resulting delays could pose a high safety risk in road traffic. In addition, MI-BCIs put a lot of cognitive strain on users. This can lead to difficulties when users need to manage multiple tasks at the same time, such as navigating a complex environment (Palumbo et al., 2021).

BCI also allows facial expressions to be recorded. Facial expressions are natural and intuitive to implement. They can be implemented without much explanation from users and require no effort. Facial expressions also offer the advantage that they can be quickly recognized and interpreted. This could lead to responsive control like steering or stopping the wheelchair (Lin and Jiang, 2016).

The same applies to alternative controls based on eye tracking methods (Bai et al., 2016; Cojocaru et al., 2019; Luo et al., 2021; Wanluk et al., 2016; Wästlund et al., 2015). These also show that wheelchair users can carry out the desired control commands even though they suffer from a

Verification of an Alternative Wheelchair Control in a Virtual Environment 291

spinal cord injury. The most common method is to point the wheelchair in the appropriate viewing direction. This control proves to be responsive and reliable, but also requires a mechanism that prevents unintentional steering maneuvers when users simply want to look around.

Similar results were achieved with voice command based controls (Abdulghani et al., 2020; Cao et al., 2021; Jayakody et al., 2019; Karim et al., 2022).

Voice control is probably suitable for tasks that are not urgent, such as adjusting the backrests or leg supports (Abdulghani et al., 2020; Shinde, 2023). Activating or deactivating various functions could also be implemented via voice. However, time-critical commands such as stopping or sudden steering could take longer than necessary in the event of danger. It is important to remember that not only speaking itself takes time, but also recognizing and processing the spoken words (Dearsley, 2024). In noisy environments, speech recognition can also be more difficult (Dua, Akanksha and Dua, 2023). These possible disadvantages could be minimized by sensible positioning of the microphone and modern ambient noise filters.

In the work of Taher, Amor and Jallouli (2016) BCI, eye-tracking and voice control techniques are combined. It uses an algorithm to determine the most likely control command from the user. The approaches of the work of Taher et al. form the basis for one of the developed alternative wheelchair controls. The second one will based on eye-tracking and voice commands only. Both are described in the next chapter.

SOFTWARE DESCRIPTION The goal of the software is to provide users, medical staff like occupational therapist, of different experience levels with an intuitive and logically structured user interface. Calibration of the BCI hardware occurs within the application, eliminating the need for external software. This calibration is carried out step by step and with illustrated instructions. Users have the option to assign available BCI commands and eye movements to individual control commands. The BCI technology used is based on facial tracking, including eyebrow movements, eye blinks, smiles and frowns. Calibration data and control command assignments can be stored in user profiles along with voice commands.

The primary control method combines BCI, eye tracking and voice control. Using voice commands, you can switch between driving and steering controls and adjust the speed. The software recognizes the user’s line of sight in order to control the wheelchair accordingly. To avoid unintentional steering, the user signals the intention to steer with a selected BCI command.

The secondary control focuses also on eye tracking and voice commands, but this control uses eye tracking for a different purpose. Due to the assumption that BCI might be prone to interference in everyday use, an second alternative control method was developed. This consists of a virtual joystick that is displayed as a two-dimensional circle. When the user fixates on this circle, it turns red and follows eye movements within a limited area, simulating the operation of a physical joystick. The voice commands were

292 Bulk and Tannert

used for setting the maximum speed, toggle the joystick and a rearview camera on and off. It is also possible to steer, start and stop the wheelchair with voice commands.

There are three virtual environments for testing the control. The first environment allows you to learn controls without obstacles on a straight, endless surface. The second environment depicts part of a small town, including houses, streets and sidewalks where obstacles, pedestrians and vehicles can be added. Some homes are walk-in, allowing testing of controls in confined spaces. In the third environment, users can playfully improve their control skills by completing a narrow route with variable obstacles.

Figure 1: Training environments of the software. From left: infinite world environment, neighbourhood, game environment (Own pictures, 2024).
Original PDF Figure 1:

HARDWARE The selection of hardware was based on the criteria of user-friendliness, robustness and cost-effectiveness. As a result, the brain-computer interface (BCI) device Emotiv EpocX and the VIVE Focus 3 VR headset with integrated eye sensors were selected. In case of users who suffer from motion sickness, the VR headset would be replaced with a conventional 27 inch-monitor in combination with a Tobii Eye Tracker 5. The software was designed to automatically switch depending on the connected hardware.

Figure 2: From left: Emotive EpocX, VIVE Focus 3, Tobii EyeTracker 5 (adapted from Emotive, VIVE and Tobii, 2024).
Original PDF Figure 2:

The BCI headset used was characterized by a stable fit and ease of use that could even be used by technical laypeople. Also, the Emotive software and the calibration process of the headset is pretty straight forward. The headset proved to be reliable under controlled laboratory conditions. However, movements and vibrations led to a significant deterioration in the quality of the measurement data, which questioned its applicability in real conditions, such as on the road.

Verification of an Alternative Wheelchair Control in a Virtual Environment 293

TESTS PROCEDURE The first assessment of the control systems for wheelchair users took place at the Center for Paraplegics in Hamburg, Germany. Three wheelchair users, two occupational therapists and the senior doctor took part.

The concept behind using a VR headset aimed to deeply involve the user in the simulation and achieve a high level of immersion and realism. Although virtual reality (VR) has now found widespread consumer acceptance, it was surprisingly found that the majority of the test subjects had no prior experience with VR. This led to initial difficulties in operating the controls, which is why a gradual introduction from simpler to more complex control methods were chosen, starting without a VR headset.

To test the developed control systems, the test subjects positioned themselves in their wheelchairs in front the monitor about one meter away. The eye tracker was attached to the bottom of the height-adjustable monitor, which was aimed at the seated person. This test arrangement was used for both types of control.

During the first test phase, the test subjects were introduced to the operation in a virtual infinite world (Fig. 1, left). Initially, control was carried out using line of sight without the use of BCI technology. This method proved to be quick to learn for all participants; Within one to two minutes, users were able to navigate intuitively. Facial recognition using BCI, on the other hand, proved to be more complex to acquire. Individual facial expressions varied in their ability to initiate control intentions. After a maximum practice time of five minutes in open space, the users switched to the ‘neighbourhood’ environment (Fig. 1, centre).

The second control method, a digital joystick, also underwent a testing phase. In contrast to the first control, the joystick can not only be used to steer. The eye control allows you to move the wheelchair back and forth and control the speed, just like a physical joystick. It took the test subjects about five minutes to feel safe in the infinite world. After that, they also went to the neighbourhood environment. It went out, that controlling the joystick and simultaneously keeping track of the environment and obstacles is a hard task. The test persons trained about ten minutes before they felt comfortable with the joystick.

Figure 3: Using virtual joystick in neighbourhood environment (screenshot from Simulator software, 2024).
Original PDF Figure 3:

294 Bulk and Tannert

RESULTS AND DISCUSSION

The test persons rated the BCI-supported eye tracking control as not suitable for everyday use. Many found controlling the wheelchair challenging as the facial expressions required often led to confusion. In everyday life, facial expressions are often used unconsciously, which could lead to unintentional movements with BCI-supported control. Additionally, the unreliability of BCI technology made precise navigation in the virtual environment difficult.

This highlights the need to develop such systems to be robust to unintentional signals and to enable clear distinction between conscious control commands and random facial movements. It is crucial that the technology is both precise and fault-tolerant to ensure safe and effective use in everyday life. A key argument against this technology was the reluctance of all test subjects to use the BCI headset in public. They expressed that they would feel uncomfortable wearing the headset visibly because it would visually emphasize their disability. Consequently, further development of this alternative control technology will be discontinued.

The second control method, the digital joystick, also underwent a testing phase. Having the immediate control above the speed and driving direction, gave the test persons a safer feeling. The resemblance to a real joystick made the virtual one easy to use. Although the control proved workable, it was shown to require more intensive training, especially in an environment closer to reality. It became clear that a more comprehensive evaluation and fine- tuning of the control only makes sense when the simulation of the virtual wheelchair more precisely replicates the behaviour of a real wheelchair.

All test subjects still agreed that the virtual joystick as a different way of controlling is a viable method that should be further developed.

OUTLOOK In order to increase the effectiveness of the control, a detailed analysis of user interactions with the digital joystick must be carried out. This would make it possible to evaluate the reaction times, the accuracy of the movements and the intuitive aspects of the controls. In addition, adjustments are being made to the user interface to provide better visual feedback, helping the user learn the controls faster and use them more efficiently.

In a next step, the physical properties of a real wheelchair, such as inertia and acceleration behaviour, are integrated into the simulation to create a more authentic driving experience. These further steps will continue to be carried out iteratively together with wheelchair users.

In the next phase of research, the digital joystick will be developed on the Magic Leap 2 augmented reality (AR) glasses, which will later also be used in real wheelchairs. The joystick initially serves as a control element for the virtual wheelchair in the simulated test environment.

As part of the further development, it is planned to display the wheelchair’s user-specific user interface (UI) on the AR display. In addition, the possibility should be created to operate this UI using eye control and voice commands.

After both the test subjects and the medical professionals have rated the control as user-friendly and safe, it is integrated into a real wheelchair. In

Verification of an Alternative Wheelchair Control in a Virtual Environment 295

collaboration with the test subjects, the control is further optimized in an iterative process.

CONCLUSION

The alternative wheelchair control methods presented in this study, particularly the digital joystick, show promising approaches to offering more autonomy to people with severe physical limitations. Iterative development and testing in virtual environments enable user-centred optimization that addresses the specific needs of the target group. Although BCI-based control is not being pursued due to accuracy issues and public perception of the technology, the digital joystick offers a viable alternative that will be further developed and implemented in real-world wheelchairs. Future research will focus on fine-tuning the controls and integrating them into physical wheelchairs to ensure safe and effective use in everyday life.

REFERENCES

Abdulghani, M. M. et al. (2020) ‘Wheelchair neuro fuzzy control and tracking system based on voice recognition,’ Sensors, 20(10), p. 2872. https://doi.org/10.3390/ s20102872 Bastos, T. et al. (2011) ‘Robotic wheelchair commanded by SSVEP, motor imagery

and word generation’ Annu Int Conf IEEE Eng Med Biol Soc. https://doi.org/10. 1109/iembs.2011.6091177. Bai, D. et al. (2016) ‘Design of an eye movement-controlled wheelchair using Kalman filter algorithm’ 2016 IEEE International Conference on Information and Automation (ICIA). https://doi.org/10.1109/icinfa.2016.7832085 Cao, W. et al. (2021) ‘Voice controlled wheelchair integration rehabilitation training

and posture transformation for people with lower limb motor dysfunction,’ Technology and Health Care, 29(3), pp. 609–614. https://doi.org/10.3233/thc- 202386 Chen, W. et al. (2022) ‘An electric wheelchair manipulating system using SSVEP- Based BCI system,’ Biosensors, 12(10), p. 772. https://doi.org/10.3390/bios 12100772 Cojocaru, D. et al. (2019) ‘Using an Eye Gaze New Combined Approach to Control a

Wheelchair Movement, 23rd International Conference on System Theory’ Control and Computing. https://doi.org/10.1109/icstcc.2019.8886158 Dearsley, J. (2024) ‘Let’s Talk: Voicebots, Latency, and Artificially Intelligent Conversation’. https://blog.vapi.ai/lets-talk/ Dua, M., Akanksha and Dua, S. (2023) ‘Noise robust automatic speech recognition:

review and analysis,’ International Journal of Speech Technology, 26(2), pp. 475–519. https://doi.org/10.1007/s10772-023-10033-0 Jayakody, A. et al. (2019) ‘Smart Wheelchair to Facilitate Disabled Individuals,’ 2019 International Conference on Advancements in Computing (ICAC). https://doi.or g/10.1109/icac49085.2019.9103409 Karim, A. et al. (2022) ‘Raspberry Pi Based Voice Controlled Smart Wheelchair’ 2022

International Conference on Emerging Trends in Smart Technologies (ICETST). https://doi.org/10.1109/icetst55735.2022.9922929 Lin, J. and Jiang, Z.-Y. (2016) ‘An EEG-Based BCI system to facial action recognition,’ Wireless Personal Communications, 94(3), pp. 1579–1593. https: //doi.org/10.1007/s11277-016-3700-3

296 Bulk and Tannert

Luo, W. et al. (2021) ‘A Human-Computer control system based on intelligent recognition of eye movements and its application in wheelchair driving,’ Multimodal Technologies and Interaction, 5(9), p. 50. https://doi.org/10.3390/ mti5090050 Palumbo, A., et al. (2021). ‘Motor-Imagery EEG-Based BCIs in Wheelchair

Movement and Control: A Systematic Literature Review,’ in: Sensors 21(18), p. 6285. https://doi.org/10.3390/s21186285 Puanhvuan, D. et al. (2017), ‘Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with disabilities’, in: Cognitive Neurodynamics, Springer, Wang (Ed.), Volume 11. pp. 117–134, https://doi.org/10.1007/s11571-017-9424-6 Rabhi, Y., Mrabet, M. and Fnaiech, F. (2018) ‘A facial expression controlled wheelchair for people with disabilities,’ Computer Methods and Programs in Biomedicine, 165, pp. 89–105. https://doi.org/10.1016/j.cmpb.2018.08.013 Shinde, S. (2023) ‘Smart voice controlled wheelchair for physically disabled

people,’ International Journal for Research in Applied Science and Engineering Technology, 11(11), pp. 536–541. https://doi.org/10.22214/ijraset.2023.56476 Taher, F. B., Amor, N. B. and Jallouli, M. (2016) ‘A self configured and hybrid fusion approach for an electric wheelchair control,’ 2016 IEEE 8th International Conference on Intelligent Systems (IS). https://doi.org/10.1109/is.2016.7737392 Wanluk, N. et al. (2016) ‘Smart wheelchair based on eye tracking’ 2016 9th

Biomedical Engineering International Conference (BMEiCON) [Preprint]. https: //doi.org/10.1109/bmeicon.2016.7859594 Wästlund, E. et al. (2015) ‘Evaluating gaze-driven power wheelchair with navigation support for persons with disabilities,’ Journal of Rehabilitation Research and Development, 52(7), pp. 815–826. https://doi.org/10.1682/jrrd.2014.10.0228

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET 2024)

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET 2024)

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-33-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →