Katherine Hernandez, Veronica Salinas, Hugo Torres, Fernando Estevez, Diego Almeida- Galárraga, Omar Alvarado Cando (2022). Assistive Technology: Design and Implementation of an Eye Tracking Based Electric Wheelchair Control System for Children with Cerebral Palsy. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe1001027
Cerebral palsy (CP) is a set of non-progressive alterations which affect muscle tone, posture, and movement. CP is produced by a brain injury or malformation in one or more brain areas that control movement. It is the most common motor disability in childhood. It affects the autonomy and socialization of children and generates great pressure on the family; therefore, children with CP require the aid of a wheelchair, which is usually driven by the user or pushed by another person. Even using a wheelchair controlled with a joystick becomes complicated because it depends on the control of gross and fine motor skills. Then, there is necessary a more amicable and accessible way to control a wheelchair. Since the ocular muscles are one of the few that still fulfill their function in people with CP, in this paper, we present the design and implementation of an eye-tracking-based electric wheelchair control system for children with spastic and mixed CP. This system includes a low-cost electric wheelchair built with PCV tubes, a Microsoft Surface Pro Tablet where the software is running, and a PCEye Mini device used to monitor the gaze. The PCEeye Mini sends the signs to the mainboard that controls the electric motors, allowing the children to steer the wheelchair by themselves using their gaze. The software has three components: the first manages the application data, the second represents the model graphically displayed on the screen, and the third receives and manages user inputs. The application has three interfaces for the start of the system that children can access since the home screen, and each one gives the user different control system parameters options. Eight children with CP participated in three experiments to validate the efficiency and safety of the system. Each experiment was developed in an internal and external environment with three different floor soils: cement, ceramic, and wood. The results showed a 100% efficiency and safety of the system for children use. The calibration process was fast and simple, the activation of directional arrows on the screen with the gaze gave 100% performance, the power consumption did not show a difference between the type of floor, and the electric wheelchair did not slip on any floor. The software maintained stable communication with the hardware through a serial communication protocol, allowing a correct execution of the commands. This work shows that the implementation of the eye-tracking technology in the electric wheelchair works successfully. This system operates effectively and is easy to control for children with CP, improving the children’s autonomy in terms of mobility. This software can be implemented on any PC, and more functions can be added to the same software without affecting its performance. This developed system offers an intelligent and different option to help children with motor deficits, which is a great contribution to society.
Human Cerebral Palsy (CP) is a set of non-progressive alterations which affect muscle tone, posture, and movement (Ricard and Loza, 2005). CP is produced by a congenital or acquired brain injury or malformation in one or more brain areas that control movement (Hercberg, 2021; Ricard and Loza, 2005). There are various types of CP. Children with CP may have two or more types of impairment. The CP types depend on: i) tone and posture: spastic, athetoid, ataxic, and mixed; ii) affected body part, and iii) degree of mobility: mild, moderate, and severe (Bangash et al. 2014; Hercberg, 2021).
Spastic is the most common type of CP, in which the muscles appear stiff, and the movements look stiff and jerky. Depending on the affected body part, this is classified as diplegic, hemiplegic, and quadriplegic. Diplegic CP affects the legs and sometimes the arms. Hemiplegic CP affects the right or left side of the body. And quadriplegic CP affects the legs, arms, and trunk. This last type is the most severe (Bangash et al. 2014; Sankar and Mundkur, 2005).
The motor deficit is the most common characteristic among children with CP, which can be mild to very severe (Bangash et al. 2014; Hercberg, 2021; Sankar and Mundkur, 2005). Motor disabilities affect the autonomy, quality of life, socialization, and emotional sphere of children, and generate great pressure on the family. In some cases, the cerebral injury may generate alterations in other superior functions and interfere with the development of the Central Nervous System (CNS) generating language, communication, cognitive, and emotional problems (Hercberg, 2021; Sankar and Mundkur, 2005).
The CP is a physical disability that may be accompanied to a greater or lesser degree
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 by a sensory or intellectual disability (Ricard and Loza, 2005). According to the National Disability Council of Ecuador (CONADIS, 2021), there are registered 27,651 children (0 to 12 years old) with disabilities. Of them, 38.43% suffer a physical disability, 39.98% an intellectual disability, 7.01% a hearing disability, 4.12% a visual disability, and 10.46% a psychosocial disability. Therefore, children with CP require the aid of a wheelchair (Boninger, 2012). Manual wheelchairs are driven by the user or pushed by another person, meaning a barrier for the autonomy of children and their families (Boninger, 2012). For children with CP, even the use of a wheelchair controlled with a joystick or other manual aid becomes complicated because these depend on the control of gross and fine motor skills (Boninger, 2012; Valle, 2013). Then, it is necessary a more amicable and accessible way to control a wheelchair.
The eye-tracking technology is of great interest in this proposal since the ocular muscles are one of the few that still fulfill their function (Ghasia et al. 2008). Children with the most severe CP level are at the highest risk for presenting strabismus (a motor deficit condition marked by turning the eyes in different directions) (Ghasia et al. 2008). However, strabismus is rare or absent in children with the mildest CP level (Ghasia et al. 2008). Eye-tracking technology allows to measure and record the behavior of the gaze, knowing the gaze localization on a surface at all times (Dickson, 2017). Some studies used eye-tracking systems for the application of an intelligence test (Andrade-Castro et al. 2020), the evaluation of mathematics performance (Alvarado-Cando et al. 2019), and reading vowels in inclusive education of children CP (Jara et al. 2018). Similarly, eye tracking systems have been used as assistive technology to improve patients with CP communication through the creation of daily use phrases (Galante and Menezes, 2012). However, these studies have not used eye- tracking technology as a tool to help with motor disability, which is the major problem in CP.
Some studies propose the use of different systems to control wheelchairs such as a trackball (Sreejith et al. 2018), a voice recognition system (Yunanto and Setyaki, 2017), electromyography (Hardiansyah et al. 2016), electroencephalography (Maksud et al. 2017), brain-computer interaction (Shinde and George, 2016) and electrooculography (Pingali et al. 2014) techniques. However, none of these solutions use an eye-tracking system; plus, they are costly to implement for the users or caretakers of the CP child.
Since the ocular muscles are one of the few that still fulfill their function in people with CP and these patients already use eye-tracking systems as evaluation and communication tools, using this same system as a mobility aid is of great interest since no more electronic devices will be needed. In this paper, we present the design and implementation of a system to control an electric wheelchair through gaze tracking, using an eye-tracking device for children with spastic and mixed CP. This article is structured as follows: the design and implementation of the solution, including hardware and software implementation, testing and results, and finally, conclusions and future work.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
DESIGN AND IMPLEMENTATION OF THE SOLUTION Design
The proposed eye-tracking system to control an electric wheelchair uses the PCEye Mini by Tobii Technology, which is the smallest device offered by this company (Tobbi Dynavox, n.d.). The user cases diagram presented in Figure 1 was used to design the system. This diagram presents each of the functional requisites that were considered for the design
Figure 1. User cases diagram Hardware Implementation
The hardware includes the construction of a personalized and low-cost electric wheelchair built with PCV tubes (Open Wheelchair, n.d. a+b), a Microsoft Surface Pro Tablet where the software is running and a PCEye Mini device used to monitor the gaze, which sends the signs to the mainboard that controls the electric motors, allowing the children to steer the wheelchair by themselves only with their gaze. Once the structure of the wheelchair is assembled, an EyeMobile Bracket is installed for the Microsoft Surface Pro Tablet and for the PCEye Mini. They are mounted on an aluminum arm, which allows adjusting the position of the tablet and PCEye Mini; therefore, the child feels more comfortable and can calibrate the gaze, thus operate the system according to their needs.
The system has two DC motors with a planetary gearbox. Its operating voltage is 12 VDC, which generates a speed of 135 revolutions per minute (rpm) without load and 100 rpm with load; the power consumption is 1.4 and 3.5 A, respectively. This type of motor was implemented because it offers high efficiency, low noise, and constant
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 high torque for better handling of heavy loads. To control the direction of rotation and supply the required energy, a Pololu dual motor controller (Pololu, 2014) has been chosen, which has two integrated VNH5019 that can operate from 5.5 V to 24 V.
To control the motors, an Atmega328 microcontroller is used, which is designed to operate in a range of 9 to 12 V. It provides analog and digital input/output ports, among other features, 6 channels of PWM (Pulse width modulation) of which 4 channels were used to control the speed of the motors (Microchip, 2019).
Software Implementation The Portable Optical Control System (SCOP) interface was developed using C Sharp (C #) programming language through Microsoft Visual Studio. Specifically, this was developed through a Windows Forms graphical user interface using different controls (e.g., buttons, dialog boxes, list boxes, and image boxes).
Software Architecture. The architecture of the application has 3 components, according to the Model View Controller (MVC) (Deacon, 2009): the model component manages the application data; the view component represents the model in a graphical way that is displayed on the screen, and the controller component receives and manages the users’ inputs.
When the user interacts with the graphical interface, a series of steps is executed internally to respond to the request. i) Notify the controller about the request through the graphical interface. ii) The controller chooses which task to perform. At this point, the model component intervenes. iii) The model performs the assigned tasks using the prior information, and the result is transmitted to the controller and view components. iv) The view with the obtained data, converts the data into visual information that can be understood by the user. Finally, v) the view forwards the graphic information to the controller, which transmits it to the user.
Interfaces Types. The application presents 3 interfaces with different control options. The child can access either one at the home screen after the profile configuration.
Type 1. In this interface, the children have to keep their gaze on the desired directional arrow, allowing the movement of the electrical wheelchair for 1 to 5 seconds. The movement time is regulated during the calibration phase and depends on the training and abilities of each child. Moreover, this interface has the option of changing the colors of the directional arrows to personalize the interface as shown in Figure 2.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Figure 2. Interface type 1 is at the left, and interface type 2 is at the right Type 2. In this interface, the children keep their gaze on the directional arrows, and the electric wheelchair will keep moving until the users create a fixation point outside of the arrows to stop the movement. The activation and deactivation of the directional arrows can be defined by a gaze time of 400 milliseconds to 1 second, these times are changed in the configuration of the profile before the calibration phase. When activating the movement, the speed of the motors starts at 0 and gradually increases. As well, the turning speed is lower than the speed of a straight path. These two conditions give greater movement security to the users. As the interface Type 1, this interface has the option of changing the colors of the directional arrows to personalize the interface as shown in Figure 2.
Type 3. This interface is designed only for children with previous training who achieved adequate control with the other 2 interfaces. In this interface, children control the electric wheelchair by looking directly through the camera of the tablet as shown in Figure 3. The wheelchair will go to the right if the child looks to the right side of the image, and it will go to the left if the child looks to the left. The wheelchair will move forward if the child looks at the top of the image, and it will move backward if the child looks at the bottom of the image. The wheelchair will stop when the child looks at any point outside the image on the screen. On the right side of Figure 3, it appears the image that the camera projected on the screen when it was active.
Figure 3. Interface type 3 Moreover, all three interface types have the following options:
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
First, a camera activation option for a better view of what is present in front of the children. However, only interface type 3 lets to control the electric wheelchair by looking directly through the camera, without the directional arrows. Second, an option to pause the interphase when the child has arrived at the destination. So, the child can look anywhere on the screen, and the wheelchair will not move. Third, in emergency cases, there are two buttons to stop movement. The first one is designed for the family and is located at the back of the wheelchair. The second one can be controlled by the child himself and is located to the side of the wheelchair, where the child through a gross motor movement can activate the button. This second button can be replaced by a capacitive touch button.
TESTING AND RESULTS Eight children with CP participated in three experiments to validate the efficiency and safety of the system. Each experiment was developed in an internal and external environment with three different soils: cement, ceramic, and wood.
Testing Calibration. Through the use of the Gaze Point application and the help of a family member, each child creates a user and performs the simple calibration of the PCEye Mini device. Test 1. Using the Type 2 interface, each child must activate the directional arrows (forward, backward, turn right and turn left) 6 times in the training and 6 times after the training phase. At the training phase, the activation and deactivation time of the directional arrows was of 1 second to take care of the child's safety. After training, the activation and deactivation time was of 400 milliseconds to create a smooth movement trajectory. Test 2. Using the Type 1 interface, the child must move through a pre-defined route, using the directional arrows as shown in Figure 4. This experiment is performed 2 times, each one at the following speeds depending on the phase. Because in the training phase children are learning how to use the PCEye Mini eye tracker, the time of activation was 1 and 2 seconds in indoor environments to avoid hitting the obstacles, and a time of 3 and 4 seconds in outdoor environments, where there is no danger of colliding with objects or people. After the learning and training phase, the movement was calibrated to 3 and 4 seconds in indoor and outdoor environments.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Figure 4. Route of the test 2
Test 3. After the child uses Type 1 and Type 2 interfaces correctly, he must move in indoors and outdoors environments using Type 3 interface. And he must activate an industrial-type button that serves as an emergency brake when they are moving. This button was implemented because braking through the software presents movement due to inertia. This test is performed 4 times.
Results
The calibration process of the "PCEye Mini" device was simple, with a time of 1.4 to 3 minutes, with an average time of 1.86 minutes. In all the experiments, the electric wheelchair did not slip on any floor, showing its safety.
For the first test, each child performed the activation of the directional arrows: forward, backward, right, and left, obtaining 48 movements of the chair in each of the directions giving a 100% performance. In the second test, an evaluation of the power consumption on different kinds of soils, applying various speeds, was developed, resulting in average maximum power consumption of 41.76 W, showing no significant difference in consumption among the type of soils. Finally, the third test presented that every time that a child activated the emergency button, the wheelchair stopped.
Thanks to the characteristics provided by the motors, the chair managed to operate with a weight of 27.45 kilograms without presenting problems in the system. And the gel battery had the functionality of approximately 9 hours without power and speed variations.
In usability tests, the braking time was not immediate, it had 94.8 cs of reaction because of the inertia of the movement. It was decided to implement a value of 50% PWM to move forward and backward and 39% PWM for turns because these changes provide adequate speed for the child to feel safe.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
CONCLUSIONS AND FUTURE WORK This work shows that the implementation of the eye-tracking technology in the electric wheelchair works successfully. This system operates effectively and is easy to control for children with CP, improving the children’s autonomy in terms of mobility. This developed system offers an intelligent and different option to help children with motor deficits, which is a great contribution to society.
This proposal uses a tablet and eye-tracker since these tools are already part of the lives of children with CP as assistive technology for evaluation and communication. The developed software can be implemented on any personal computer, and more functions such as evaluation and communication can be added to the same software without affecting its performance. In this way, no more costs will be added.
For child and chair safety, a proximity sensor can be implemented to avoid possible collisions with an obstacle. Further research is recommended, so the proposed system can potentially work on sloping soils. It is also recommended to implement a PID control to improve the braking system of the electric wheelchair.
REFERENCES
Alvarado-Cando, O., Jara, B., Barzallo, P., and Torres-Salamea, H. (2019), “A Software Based on Eye Gaze to Evaluate Mathematics in Children with Cerebral Palsy in Inclusive Education.” Advances in Intelligent Systems and Computing, 909-915. doi:10.1007/978-3-319-94947-5_89. Andrade-Castro, K., Alvarado-Cando, O., Ortega-Chasi, P., and Cobos-Cali, M.
(2020), “An Eye Tracking System to Perform the Wechsler Intelligence Children-Fifth Edition (WISC-V) Test: A Case Study.” Advances in Intelligent Systems and Computing, 861-868. Bangash, A., Hanafi, M., Idrees, R., and Zehra, N. (2014), “Risk factors and types of cerebral palsy.” The Journal of the Pakistan Medical Association, 64(1), 103– 107. Boninger, M. (2012). La silla de ruedas eléctrica: Lo que el consumidor con LME
necesita saber. Model Systems Knowledge Translation Center. Available at: https://msktc.org/sci/factsheets/la-silla-de-ruedas-el%C3%A9ctrica-lo-que-el- consumidor-con-lme-necesita-saber/ (Accessed 30 September 2021). CONADIS. (2021). Estadísticas de Discapacidad. Consejo Nacional para la Igualdad de Discapacidades-CONADIS. Available at: https://www.consejodiscapacidades.gob.ec/estadisticas-de-discapacidad/ (Accessed 30 September 2021). Deacon, J. (2009), “Model-View-Controller (MVC) Architecture.” JOHN DEACON
Computer Systems Development, Consulting & Training, 1-6. Dickson, B. (2017). What is eye tracking technology?. TechTalks. Available at: https://bdtechtalks.com/2017/01/05/what-is-eye-tracking-technology/ (Accessed 30 September 2021).
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Galante, A., and Menezes, P. (2012), “A Gaze-Based Interaction System for People with Cerebral Palsy.” Procedia Technology, 5, 895-902. Ghasia, F., Brunstrom, J., Gordon, M., and Tychsen, L. (2008), “Frequency and
severity of visual sensory and motor deficits in children with cerebral palsy: gross motor function classification scale.” Investigative Ophthalmology & Visual Science, 49(2), 572–580. doi:10.1167/iovs.07-0525. Hardiansyah, R., Ainurrohmah, A., Aniroh, Y., and Tyas, F.H. (2016), “The electric wheelchair control using electromyography sensor of arm muscle.” 2016 International Conference on Information & Communication Technology and Systems (ICTS), 129-134. doi: 10.1109/ICTS.2016.7910286. Hercberg, P. (2021), “¿Qué es la parálisis cerebral?” In Confederación ASPACE
(eds.). Descubriendo la Parálisis Cerebral. Madrid, 7-11. Available at: https://aspace.org/assets/uploads/publicaciones/726a6-descubriendo-p.- cerebral.pdf Jara, B., Barzallo, P., and Alvarado-Cando, O. (2018), "A Software Based on Eye Gazed to Evaluate Vowels in Children with Cerebral Palsy in Inclusive Education," 2018 IEEE ANDESCON, 1-4. doi: 10.1109/ANDESCON.2018.8564694. Maksud, A., Chowdhury, R., Chowdhury, T., Fattah, S., Shahanaz, C., and
Chowdhury, S. (2017), “Low-cost EEG based electric wheelchair with advanced control features.” TENCON 2017 - 2017 IEEE Region 10 Conference, 2648-2653. doi: 10.1109/TENCON.2017.8228309. Microchip. (2019). ATmega48A/PA/88A/PA/168A/PA/328/P. Available at: https://ww1.microchip.com/downloads/en/DeviceDoc/ATmega48A-PA-88A- PA-168A-PA-328-P-DS-DS40002061A.pdf (Accessed 30 September 2021). Openwheelchair. (n.d. a). Open Wheelchair - Official Site. Available at: https://www.openwheelchair.org/ (Accessed 30 September 2021). Openwheelchair. (n.d. b). Open Wheelchair: Plans. Available at: https://openwheelchair.org/Plans/ (Accessed 30 September 2021). Pingali, T., Dubey, S., Shivaprasad, A., Varshney, A., Ravishankar, S., Pingali, G.R.,
Polisetty, N.K., Manjunath, N., and Padmaja, K.V. (2014), “Eye-gesture controlled intelligent wheelchair using Electro-Oculography.” 2014 IEEE International Symposium on Circuits and Systems (ISCAS), 2065-2068. doi: 10.1109/ISCAS.2014.6865572. Pololu. (2014). Pololu Dual MC33926 Motor Driver Shield User’s Guide. Available at: https://www.pololu.com/docs/0J55 (Accessed 30 September 2021). Ricard, F. and Loza, E. (2005). Osteopatía y Pediatría. España: Editorial Médica
Panamericana S.A. Available at: https://dialnet.unirioja.es/servlet/libro?codigo=308849 Sankar, C., and Mundkur, N. (2005), “Cerebral palsy-definition, classification, etiology and early diagnosis.” Indian Journal of Pediatrics, 72(10), 865–868. doi:10.1007/BF02731117. Shinde, N., and George, K. (2016), “Brain-controlled driving aid for electric
wheelchairs.” 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN), 115-118. doi: 10.1109/BSN.2016.7516243. Sreejith, T., Vishnu, J., and Vijayan, G. (2018), “Trackball Controlled Novel, Cost Effective Electric Wheelchair.” 2018 International Conference on Control,
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Power, Communication and Computing Technologies (ICCPCCT), 205-209. doi: 10.1109/ICCPCCT.2018.8574266. Tobii Dynavox. (n.d). PCEye Mini Featuring Gaze Point & Gaze Viewer. Available at: https://www.tobiidynavox.com/devices/eye-gaze-devices/pceye-mini-track- learn-gaze-point- viewer/?MarketPopupClicked=true&utm_medium=organic&utm_source=ww w.google.com. (Accessed 30 September 2021). Valle, I. (2013). Sillas de ruedas eléctricas. Opciones de mandos de control y cambios
de postura. 4th edn. Madrid: Centro de Referencia Estatal de Autonomía Personal y Ayudas Técnicas-CEAPAT, 1-55. Available at: http://riberdis.cedd.net/bitstream/handle/11181/4995/Sillas_de_ruedas_electric as.pdf?sequence=1&rd=0031825621214361 Yunanto, D.C., and Setyati, E. (2017), “Speech recognition with Indonesian language for controlling electric wheelchair.” 2017 International Seminar on Application for Technology of Information and Communication (iSemantic), 90-94. doi: 10.1109/ISEMANTIC.2017.8251849.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional