Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Multimodal Visual, Auditory, Thermal, and Tactile Feedback During Brain-Machine Interface Use by a Spinal Cord Injury Patient

1 Citar – Centro de Investigação em Tecnologias da – Porto, Universidade, Católica Portuguesa, Rua de Diogo Botelho, 1327, 4169-005 Porto, Portugal ↗
2 ID+ Research Institute for Design, Media and Culture, School of Design, Polytechnic Institute of Cavado and Ave.- Campus do IPCA - Lugar do, Aldão, 4750-810 Vila Frescainha, Portugal ↗
3 Serviço de Neurologia, Hospital Senhora da Oliveira, R. dos Cutileiros, 114, Creixomil, Guimarães, Portugal ↗
Publication year
Digital Object Identifier10.54941/ahfe100912

Abstract

Background: Brain-machine interfaces (BMIs) have the potential to replace and expand body functions, but also to induce neuroplasticity. In BMIs that include virtual reality and tactile feedback, it is thought that the underlying mechanism may be partially dependent on immersiveness (i.e., how “realistic” the environment is). It is not known however, if continuously increasing the number of simulation modalities with the goal of creating a more immersive environment may eventually lead to an overload of information and prevent BMI performance. Objective: The aim of this study was to investigate the performance of an spinal cord injury (SCI) patient in a BMI setup based on lower limb motor imagery (i.e., requiring the participant to modulate neural activity to control an avatar in complex virtual reality scenarios, while receiving coherent visual, auditory, tactile, and thermal feedback. Setting: Experiments took place in Physical Medicine and Rehabilitation Department at the Hospital Senhora da Oliveira in Guimarães, Portugal.Methods: The participant was a 52-year-old male with an ASIA complete T4 SCI stabilized. The embodiment experiences were generated through a set up where the subject was required to generate lower limb motor imagery commands, while receiving multimodal feedback, delivered through a virtual reality headset (including goggles and headphones) combined with thermo-tactile feedback sleeves. Sessions were characterized by three different phases: habituation (hand control of avatar movement), acquisition (neural data acquired to train the neural network), and real-time decoding (neural signals decoded in real-time to control the avatar movements). Visual stimuli indicated whether the patient should think about “Walking” or “Not walking”. Results: In the sessions presented here, the participant consistently presented performances above chance levels. In a small number of sessions, where performances were closer to chance levels, the patient reported increased levels of stress due to variables not directly related to the experimental setup or design (work stress, personal problems, etc.). Unexpectedly, in a small number of sessions where the virtual scenario included water, the patient reported feeling his “legs cold”. Conclusions: This study demonstrates that a spinal cord injury patient can control a brain-machine interface combining virtual reality (visual and auditory), tactile, and thermal feedback; supporting the notion that the increased number of feedback modalities did not generate an overload of information.

INTRODUCTION

Brain-machine interfaces (BMIs) have the potential to replace and expand body functions (Lebedev and Nicolelis, 2017; Pais-Vieira et al., 2013; Pais-Vieira et al., 2015), but also to induce neuroplasticity (Donati et al., 2016). In BMIs combining Virtual Reality (VR) and tactile feedback, it is thought that the underlying mechanism may be partially dependent on the degree of immersion (i.e., how “realistic” the environment is), which produces a virtual sense of embodiment. It is not known however, if continuously increasing the number of simulation modalities with the goal of creating a more immersive environment may eventually lead to an overload of information and prevent BMI performance. The aim of this study was to investigate the performance of a Spinal Cord Injury (SCI) patient using a lower limb Motor Imagery (MI) based BMI setup (i.e., requiring the participant to modulate neural activity) to control an avatar. The VR scenarios presented complex and realistic patterns and the avatar movements reproduced those of a previously described exoskeleton (Pais-Vieira et al., 2020). Meanwhile, the user received coherent visual,

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 auditory, tactile, and thermal feedback. Our hypothesis was that the large number of modalities used to generate the immersive environment would not constitute an overload of information for the user and therefore would not prevent the user from modulating neuronal activity to control the avatar. To test this hypothesis, a SCI patient was trained to control a BMI setup that included visual and auditory feedback (i.e., virtual reality goggles and headphones), as well as tactile and thermal feedback.

METHODS

The present study took place in Physical Medicine and Rehabilitation Department at the Hospital Senhora da Oliveira in Guimarães, Portugal; and was approved by the local ethics committee (15/2020). The participant was a 52-year-old male with an ASIA complete and stabilized T4 SCI and a history of chronic pain. This patient had previously tested this setup in a context without Brain Control (i.e., the avatar was moving independently of the neural activity of the user). Embodiment experiences were induced through a set up where the subject was required to generate lower limb motor imagery commands, while receiving multimodal feedback, delivered through a virtual reality headset (including goggles and headphones) (HCT Vive Pro Eye, New Taipei City, Taiwan), combined with thermal and tactile feedback sleeves. Neural data was acquired through a 16 channel EEG (V-Amp, actiCAP; Brain Products GmbH, Gilching, Germany) and analyzed in real time in Open Vibe [6]. Processed data allowed control of an avatar in multiple virtual reality scenarios run in Unity (Unity Technologies, San Francisco, USA) and controlled trough Max (Cycling '74, San Francisco, USA) software. Seven different scenarios that could include grass, rock, sand/water, or a mix of these, were available. The patient was allowed to choose the scenario, but the scenario used during the acquisition and the decoding phases was the same throughout the course of a single session. A schematic of the setup is presented in Figure 1. Sessions were characterized by three different phases: habituation, acquisition, and real-time decoding. During habituation, the user controlled the avatar movements through the hand control. During acquisition, the participant received Visual cues (see Figure1 a, green and red targets during Data acquisition and Decoding phases) that indicated whether the patient should think about “Walking” or “Not walking”. This data was then used to train a spatial filter and the neural network (using the original OpenVibe algorithms) (Renard et al., 2010). During the third phase, neural signals were recorded and decoded in real-time to control the avatar movements. To complement the description of the setup, additional questionnaires were used to evaluate the embodiment experiences (Peck and Gonzalez-Franco, 2021), VR side effects (Kennedy, 1993), and pain levels (Collins et al., 1997; Gallagher et al., 2001; Bijur et al., 2003). Due to the small number of sessions studied here, cross correlations between performance, embodiment, and pain levels were not performed. Arbitrary units (a.u.) were used to quantify the results from the questionnaires.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Figure 1. Experimental design and setup. a) During (Habituation) the subject controlled the avatar with the hand commands. In the second phase (Data Acquisition) EEG data was recorded while the subject was instructed to think about “Walk” (green cue) or “Don’t walk” (red cue). In the last phase (Decoding) neural activity was recorded and decoded in real-time. trials. b) Example of a complex scenario with multiple types of feedback. Visual, auditory, tactile, and thermal feedback were coherent throughout different modalities and scenarios.
Original PDF Figure 1.

RESULTS

BMI performances for individual sessions are presented in Figure 2 a. Performances were generally above chance and in one case (session 6) the performance was perfect (78.73±12.92% correct; min: 63.6%; max:100%). The subject reported high levels of embodiment (Figure 2 b) during the motor imagery task (6.71±0.345 a.u.; in a 7-point scale; min:3 a.u.; max:7 a.u.). These levels were also high for the three different domains of this scale, namely: Body: 6.38±1.02 a.u.; Tactile: 6.95±0.38 a.u.; and

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Motor: 6.95±0.38 a.u.. In a small number of sessions (sessions 3 and 7), performances were below 70%. In these sessions, the patient reported the occurrence of external stressful events (e.g., patient arrived late, personal problems, etc.).

Evaluation of pain levels for three different pain scales did not reveal a clear change throughout the small number of sessions tested here (panel c). VAS values (Figure 2 b, black circles) presented an average of 6.29±0.49 a.u. (min: 6 a.u., max 7 a.u.), Faces (Figure 2 b, blue circles) presented an average of 5.21±1.15 a.u. (min: 4 a.u., max 6.5 a.u.); lastly, verbal pain (Figure 2 b, red circles) was reported as moderate in 6/7=85.7% of the sessions; and light in 1/7=14.3% of the sessions.

Additionally, in one session (session 3) where the virtual reality scenario re-quired the avatar to walk in water, the patient reported “I am feeling my legs cold, but this is not uncomfortable”. Such event had previously occurred in another session where we were testing the setup without the use of the BMI (i.e., no brain control of the avatar). As this type of experience could lead to an eventual preference or aversion for a given scenario, we further asked the patient about his preference regarding locations (such as, urban versus natural), as well as to the type of stimulation (i.e., associated with grass, sand, stone, water, or mixed). The patient revealed no preference for any type of scenario or stimulation associated.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Figure 2 Results from sessions. a) (Performance) in the BMI sessions was generally above chance. b) Results from (Embodiment questionnaire). c) Self-reported pain levels in (VAS – black circles), (Faces- blue circles), and (Verbal – red circles) pain scales throughout the sessions. d) Simulator sickness results (VR effects).

DISCUSSION

A BMI setup for neurorehabilitation combining EEG activity, virtual reality (visual and auditory), tactile, and thermal stimulation was tested in a SCI patient to determine if this combination of multimodal feedback would prevent brain control of an avatar. The patient was able to modulate neural activity in order to generate the commands to “Walk” and “Not walk” according to the cues presented, therefore supporting our hypothesis that this multimodal feedback did not prevent brain control of the avatar. Our present results are in line with previous results supporting the notion that the degree of immersion can significantly contribute to performance and possibly to improved neurorehabilitation (Donati et al., 2016; Lenggenhager et al., 2013). The setup presented here, takes these previous studies one step further, through the inclusion of complex scenarios that allow maximization of the feedback experiences. For example, transitions between the different parts of the scenario, which result in different types of multimodal stimulation, were reported as being of particular interest for the user, as previously reported for other virtual reality contexts (Stepanova et al., 2019). More, even though this patient had previous contact with these scenarios and setups (in sessions without BMI control), no reduction in the overall interest to perform the sessions was reported by the patient. These findings suggest that some of the additional features used here may contribute to improve engagement. We propose that these features may be maximized, for example, through the use of detailed “storyboarding”, where a clear match between the subjects’ preferences and the choices performed in the task could be maximized (Vieira et al., 2021). Although one of our scenarios included the possibility of the user to choose between different outcomes, we have not used this scenario for the acquisition and real-time decoding phases. No clear changes in pain were observed throughout the sessions presented here. Previous studies have demonstrated improvements in pain due to the use of virtual reality protocols (Villiger et al., 2013; Chi et al., 2019; Austin et al., 2021). It is not clear from the present results if such decreases in pain were not observed here due to the small number of sessions collected, or otherwise, if the setup does not lead to any significant changes in self-reported pain. An unexpected finding from the present study was the patient reporting feeling the lower limbs cold when the avatar was placed in a water scenario. In the present moment, we do not have a clear explanation regarding the mechanism underlying this observation. For example, it is not clear if this is due to some effect of the setup or

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 otherwise from the SCI. We speculate that this may be related to some form of modulation not related to the SCI (Hoffman et al., 2004; Tieri et al., 2017), because we have meanwhile observed a similar reaction in a control subject being tested in the same scenario during regular setup maintenance. However, the details of this mechanism and its differential activation by specific scenarios remains to be elucidated. Lastly, the present results were obtained from a single patient, and therefore it cannot be excluded that, for other users, this multimodal setup may result in a detrimental effect in BMI performance. In future studies, an increased number of sessions and patients will be studied. Our results support the notion that visual, auditory, tactile and thermal feedback can be used by a SCI patient to control an avatar.

CONCLUSION

Our results support the notion that visual, auditory, tactile and thermal feedback can be used by a SCI patient to control an avatar.

ACKNOWLEDGMENTS

The authors would like to acknowledge the following entities: Santa Casa Neurociências, Prémio MELO e CASTRO (SCM-12/18). FCT - iBiMED – Institute of Biomedicine (UIDB/04501/2020); FCT/MCTES Center for Interdisciplinary Research in Health (UIDB/04279/2020); FCT – Fundação para a Ciência e a Tecnologia, I.P., within the project CISUC - UID/CEC/00326/2020 and by the Fundo Social Europeu through the Programa Operacional Regional Centro 2020.

REFERENCES

Austin, P.D. and Siddall, P.J., (2021) “Virtual reality for the treatment of neuropathic pain in people with spinal cord injuries: A scoping review”. The journal of spinal cord medicine, 44(1), 8-18. Bijur, P.E., Latimer, C.T. and Gallagher, E.J., (2003) “Validation of a verbally administered

numerical rating scale of acute pain for use in the emergency department”. Academic emergency medicine, 10(4), 390-392. Chi, B., Chau, B., Yeo, E. and Ta, P., (2019), “Virtual reality for spinal cord injury-associated neuropathic pain: systematic review”. Annals of physical and rehabilitation medicine, 62(1), 49-57. Collins, S.L., Moore, R.A. and McQuay, H.J.,(1997) “The visual analogue pain intensity scale:

what is moderate pain in millimetres?”. Pain, 72(1-2), 95-97.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Donati, A, Solaiman S. Edgard Morya, Debora SF Campos, Renan C. Moioli, Claudia M. Gitti, Patricia B. Augusto et al. (2016), “Long-term training with a brain-machine interface-based gait protocol induces partial neurological recovery in paraplegic patients”. Scientific reports 6, no. 1, 1-16. Gallagher, E.J., Liebman, M. and Bijur, P.E., (2001), “Prospective validation of clinically

important changes in pain severity measured on a visual analog scale”. Annals of emergency medicine, 38(6), 633-38. Kennedy, R.S., Lane, N.E., Berbaum, K.S. and Lilienthal, M.G., (1993), “Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness”. The international journal of aviation psychology, 3(3), 203-20. Lebedev, M.A. and Nicolelis, M.A., (2017), “Brain-machine interfaces: From basic science to

neuroprostheses and neurorehabilitation”. Physiological reviews, 97(2), 767-837. Lenggenhager, B., Scivoletto, G., Molinari, M. and Pazzaglia, M., (2013), “Restoring tactile awareness through the rubber hand illusion in cervical spinal cord injury”. Neurorehabilitation and Neural Repair, 27(8), 704-08. Pais-Vieira, M., Lebedev, M., Kunicki, C., Wang, J. and Nicolelis, M.A., (2013) “A brain-to-

brain interface for real-time sharing of sensorimotor information”. Scientific reports, 3(1), 1-10. Pais-Vieira, M., Chiuffa, G., Lebedev, M., Yadav, A. and Nicolelis, M.A., (2015) “Building an organic computing device with multiple interconnected brains”. Scientific reports, 5(1), 1-5. Pais-Vieira, C., Allahdad, M., Neves-Amado, J., Perrotta, A., Morya, E., Moioli, R., Shapkova,

E. and Pais-Vieira, M., (2020) “Method for positioning and rehabilitation training with the ExoAtlet® powered exoskeleton”. MethodsX, 7, 100849. Peck, T. C., & Gonzalez-Franco, M., (2021), “Avatar embodiment. a standardized questionnaire”. Frontiers in Virtual Reality, 1, 44. Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O. and

Lécuyer, A., (2010), “Openvibe: An open-source software platform to design, test, and use brain–computer interfaces in real and virtual environments”. Presence, 19(1), 35-3. Stepanova, E. R., D. Quesnel, and B. E. Riecke., (2019) “Understanding AWE: Can a Virtual Journey, Inspired by the Overview Effect, Lead to an Increased Sense of Interconnectedness?”. Frontiers in Digital Humanities, 6. Vieira, C., da Silva Pais-Vieira, C.F., Novais, J. and Perrotta, A. (2021), “Serious Game Design

and Clinical Improvement in Physical Rehabilitation: Systematic Review”. JMIR Serious Games, 9(3), p.e20066. Villiger, M., Bohli, D., Kiper, D., Pyk, P., Spillmann, J., Meilick, B., Curt, A., Hepp-Reymond, M.C., Hotz-Boendermaker, S. and Eng, K., (2013), “Virtual reality–augmented neurorehabilitation improves motor function and reduces neuropathic pain in patients with incomplete spinal cord injury”. Neurorehabilitation and neural repair, 27(8), 675-83.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →