Youngmi Christina Choi, Kelly Fischer (2024). Mixed Reality in Physical Rehabilitation, Opportunities and Challenges. In: Interdisciplinary Practice in Industrial Design. AHFE Open Access, vol. 144, pp. 1–9. https://doi.org/10.54941/ahfe1005112
Mixed reality has begun to find applications in new areas as hardware, from smart phones to head mounted displays, have become more widespread, powerful and affordable. Research on effectiveness, acceptability and other issues related to the use of these technologies have been carried out in different contexts. While more work remains to be done, it is possible to envision new cross disciplinary applications with the potential to be highly effective. which would not have been possible without results from these different research streams. This paper will examine the use of augmented reality in the context of upper limb rehabilitation. Related research on the perceived accuracy and validity of of augmented (and tangible augmented) reality, integration with external sensors, supporting product-service-systems and customization will be examined. Potential applications to upper limb rehabilitation will be discussed and open issues highlighted.
The goal of this paper is to investigate the application of mixed reality in physical rehabilitation for upper limb conditions. In particular, we look at the combination of Augmented Reality (AR) and Tangible Augmented Reality (TAR) combined with the use of electromyography to allow users to perform specific rehabilitation tasks with the ability to monitor and guide the user in real time to perform the tasks accurately and effectively.
We want to investigate this by exploring a couple of questions. First is whether the use of an augmented reality based tool is capable of providing a perceived experience of performing a task utilizing virtual elements that is equivalent to performing that same task with fully physical elements. If this is possible, then the second question is whether it is possible to accurately mea- sure physical actions/movement in order to detect how a task is performed. For a rehabilitation based activity, it will be important to confirm that a user not only feels like an action is being performed accurately but that movements are actually executed in a particular way.
MIXED REALITY AR (Augmented Reality) and TAR (Tangible Augmented Reality) allow com- pute generated virtual elements to be displayed and overlaid within a real
environment. These elements might replace objects within the real environ- ment or may be entirely new. AR and TAR differ from Virtual Reality where the everything within the field of view is computer generated and contains no real elements from the current environment.
There are several ways of displaying AR to a user. The main ones can ge generally classified as Handheld Displays (HHD), Head Mounted Displays (HMD), and Spatial Augmented Reality (SAR) (vanKrevlan and Poleman, 2010). HHD type devices are the most widely accessible and familiar devices. Most smartphones available today are capable of acting as a display for AR. Displays of this type typically work through the use of a marker, which might be a QR code or other unique symbol. The marker is tracked by the front fac- ing camera. Software installed on the device detects the marker and puts in its place a digital 3D model or object. This can be imagined as taking a video of a real space with a virtual element added to the view in real time. As long as the user views the environment through the HHD, then the virtual ele- ment will always appear where the marker has been placed. A disadvantage to these types of displays is that it must be held to maintain the view. HMDs avoid the hand held issue by integrating a screen into a headset. Most often these devices utilize a transparent display allowing the environment to be seen normally, as if through glasses, while digital elements are overlaid. The Microsoft HoloLens and Apple Vision Pro are examples of these kinds of devices. The devices include cameras for tracking as well as onboard proces- sors. While freeing the hands, a couple of immediate drawbacks are that they tend to be bulky, must be worn and fitted properly, and have generally low resolution and narrow field of view. Another drawback is that they are expen- sive, specialized equipment and not something that a person would generally take along everywhere with them (like with a smartphone). SAR displays are even more specialized. These work by projecting a digital image onto a semi-transparent medium in a real environment. These kinds of displays have been used to allow virtual performances of musicians (both living and dead) on stage with other live performers (Peddie, 2017). The technology is more expensive, requiring specialist customized equipment. The display also works only at static locations where it has been setup.
AR has been effectively employed in a number of industries and environ- ments to achieve various tasks. This has been particularly true for training scenarios where it has been used to help reduce errors while performing tasks, improving the memorability of tasks, and reducing costs and time related to training (Helin et al., 2018). AR has been applied to create alternate com- puter interfaces with increased customizability, new intuitive interactions and enablement of three-dimensional interfaces (Cometti et al., 2018). The most successful application of AR has likely been in support of industrial mainte- nance and repair tasks (Re & Bordegoni, 2014), industrial assembly guidance (Ong et al., 2008) and as job aids (Anastossova et al., 2005).
PERCIEVED USABILITY OF AR AND TAR So can performing an action through a fully or partially virtual interface give a similar experience to doing it non-virtually? It is important to the topic
Mixed Reality in Physical Rehabilitation, Opportunities and Challenges 3
here because a person performing a task with a virtual product/object should ideally feel no different doing it fully physically. This has been explored in sev- eral scenarios in the context of product design and usability assessment. This area of research is particularly relevant since the goal of a product designer when testing a product or seeking feedback is to ensure that the user is get- ting the same experience in using the product as if it were real. The highest standard for this is to perform testing using highly detailed or fully func- tional prototypes. This way the test of the product is as close as possible because the prototype is a literal embodiment of a proposed design concept. This allows testing of both objective performance attributes as well as impor- tant subjective elements that contribute to the perceived execution of a task (such as aesthetics, ergonomics, product integrity, craftsmanship) (Srinivasan, Lovejoy, & Beach, 1997). These prototypes are of course very time consum- ing and expensive to produce. A mixed reality representation of a product allow the possibility of more easily testing a concept to get accurate feed- back and assessments without the time and cost involved with producing a physical prototype.
The accuracy of the user experience has been explored under a number of circumstances. Barbieri et al. (2013) employed a TAR setup to study the interface layout of home appliances. The setup allowed physical elements to be rearranged and repositioned so that tags for different virtual elements could be applied. This allowed different arrangements and designed elements to be tested. The advantage to this approach was that it allowed different arrangements to be easily assessed compared to each other.
Performance and ergonomics of a product of a TAR representation of a projector were explored by Faust et al. (2018). This study included the com- parison of the mixed reality TAR representation alongside an actual projector prototype. Both were evaluated for usability as well as task performance and errors. It found performance and errors were positively correlated with task difficulty and showed the utility of mixed reality for evaluating a product’s operation, performance, and ergonomics.
A direct comparison of the usability of different product representations with AR and TAR were made by Choi and Mittal (2015). This study com- pared an mp3 player with a touchscreen interface with both AR and TAR representations of it. The product function and interface was mocked up virtually, allowing an AR and TAR version of the product interface to be repli- cated in both look and function. 60 participants were recruited to evaluate the representations. They were split into three independent groups of 20, with each group evaluating one of the representations. All users performed the same set of tasks to select, play, pause songs and navigate the interface. The results showed that there were no significant differences between the usabil- ity evaluations given for the AR an TAR versions of the product compared to the real one.
In another study (Choi, 2019), a similar setup was used only this time the product being studied was a space heater. 70 participants took part and were randomly assigned to evaluate one of the representations: AR, TAR (Figure 1) and the actual space heater. The results in this case showed that there were no differences between the TAR representation and the real space
4 Choi and Fischer
heater. There were significant differences between the AR and real version. The main difference of the product in this study is that the user interface of the space heater included physical elements (knobs). The results of the AR evaluations were different because the way of interacting with the physical knobs in the virtual environment was very different from reality. Instead of being able to grab and turn something in AR, users had to tap on the virtual knob and drag to turn it. With the TAR version, users were able to grab and turn a physical element in the same way as the real product in order to change the settings of the virtual heater.
These studies highlight a couple of important results. It is possible for AR or TAR to provide an accurate experience of performing a task to a user. It is important however to ensure that the type of AR matches the requirements of the task such that the way of performing/executing the virtual task does not have to be altered from the way it would actually be performed.
Figure 1: Left: a space heater represented in AR with interaction performed on a touch- screen. Right: a space heater represented in TAR with interaction performed on the physical model and results shown in the virtual view.
ELECTROMYOGRAPHY With mixed reality in the form of AR and TAR demonstrated to be able to provide users with the same experience of performing a task virtually as in reality, we want to move on to the issue of being able to confirm that a task is performed in a particular way. With the usability of a product, the actual method of performing a task is not as important as if it can be successfully accomplished. In other words, to turn a knob it really doesn’t matter much whether one user grabs and turns it in a different way from another, so long as it is successful. In the context of rehabilitation, this is not the case. A task in the scenario will have a requirement to be executed in a particular way each time.
If one thinks of the case of a ubiquitous smart phone as an AR display device, an obvious extension is the additional integration of external sensors. There are a number of potential ways of measuring movement, one of which is Electromyography. This is a technique that records the electrical activity of the muscle fibers when activated for movement (Brown, 2013). There are a couple of ways of detecting these signals. One is by using electrodes that are
Mixed Reality in Physical Rehabilitation, Opportunities and Challenges 5
attached at various points to the surface of the skin. This is referred to as sur- face electromyography (sEMG). Another way of recording electrical signals is by using needles to embed electrodes directly into the muscles. This is referred to as intramuscular electromyography (imEMG). EMG is already utilized in clinical applications, prosthetics, rehabilitation, and human-machine interac- tions. In the rehabilitation context, EMG and other devices are employed in physical therapy with a professional trainer or in some home settings (Klein et al., 2018).
EMG sensors are also increasingly incorporated into common consumer devices, such as fitness trackers. While they can provide a overview of general activity, the detailed accuracy of the measurements taken by these sensors is often questioned. The accuracy can be impacted by changes in position, temperature, the type of tissue they are in contact with, blood flow and many other variables. Additionally, quality signals can be affected by the activation of adjacent muscles, obviously also impacted by the positioning of the sensor (Esposito et al., 2018). Large surface electrodes can provide clear signals; however, noise increases as they are miniaturized (Chowdhury et al., 2013).
EMG signals can be detected, processed, and classified, though it can be difficult to do this where there is noise in the signal (Chowdhury et al., 2013). While it often cannot be removed completely, newer electronics and ampli- fication along with signal filtering techniques can generate cleaner EMG signals (Gerdle et al., 1999). The resulting signals can be further analyzed in software for pattern recognition and classification algorithms in order to distinguish unique signals associated with different movements from one another (Parker & Scott, 1986). Applying these techniques along with col- lecting multiple channels of data has been shown to provide 97% accuracy in signal classification from sEMG sensors (Hargrove, 2007).
TESTING SEMG WITH MIXED REALITY With non-invasive sEMG providing a way to accurately measure muscle acti- vation in order to detect how a task was performed, a study was conducted to compare the actual differences in the way of performing a task while inter- acting with a normal object or with one represented in TAR. 18 participants were recruited with the objective of measuring both the muscle activation while performing each task as well as the perceived usability for each case.
Each participant was fitted with a Myo gesture control armband. This is an surface EMG sensor equipped with eight medical grade electrodes, an onboard unit for measurement and a module for transmitting those mea- surements. The band was fitted identically on the dominant arm of each user. Participants were then instructed to perform the same set of three tasks: removing the cap from a pen, lifting a teacup from the table, and remov- ing the lid from a disposable coffee cup. Participants performed each task in real life first and then transitioned to performing them using TAR. For the TAR tasks, participants observed their interactions through a mounted phone screen. While performing the task, users were able to view their action through the screen and view feedback on how it should be performed. Once a task was completed, participants completed a System Usability Survey.
6 Choi and Fischer
RESULTS
The EMG arm band captured the waveform of the activation of muscles dur- ing the performance of each task. These were analyzed by converting all of the measured waveforms to a rectified value, which converts negative signal val- ues to positive ones. This is done so that the mean value of the measurement is not zero (Figure 2).
The recorded signals was modeled by using an integrated EMG (iEMG) measurement, which is the area under the curve of the rectified EMG signal (Christopher, 2018). This measurement equates to the energy information of the signal. Waveform Length (WL), which is the cumulative length of the waveform, was also used. This measurement equates to the complexity of the signal information.
Figure 2: Recorded EMG signal before and after rectification.
Data for each participant for each task was analyzed and compared. The most important results were found when comparing the differences of the iEMG (energy) measurements and the WL (complexity) measurements between the real and TAR versions of the tasks (Figure 3). What we find is that the measured usability (x axis) increases as the difference between the EMG measurements (y axis) decreases. The regression slope between the percent difference in iEMG measurement and SUS score was: r2 = 0.695. Sim- ilarly the percent change between the WL and the SUS score, while weaker was r2 = 0.412. The closer the muscle activation of the virtual task the closer the resulting usability score compared to the real task.
Figure 3: iEMG and WL % change vs SUS scores.
Mixed Reality in Physical Rehabilitation, Opportunities and Challenges 7
DISCUSSION AND CONCLUSION From the studies reviewed, there is clear potential in the use of both AR/TAR and EMG sensors applied to rehabilitation tasks. From a task perspective, it is possible to provide an equivalent experience of perform- ing a task through AR/TAR and from the measurement standpoint, it is possible to measure and classify an EMG signal to track how the task is performed.
Even with drawbacks inherent in an HHD type device like a smart phone, the ever-increasing power and capability of these devices make the use of mixed reality possible for large groups of people. The ubiq- uity and familiarity of these devices in the form of smartphones is a great advantage. Alone they are not as useful, however it is already common to integrate a phone with external devices to enable additional functionality, from pairing with a car infotainment system to smartwatches and health monitors.
The increasingly common use of additional devices with smart phones, par- ticularly health monitors, provides an opportunity. It is doubtful that an EMG sensor integrated into a smartwatch will be able to provide the signal detail and accuracy presented in the study here. Much like an HMD, the Myo arm- band is a highly specialized device and not something that will be commonly available. However, given improvements in sensor hardware and or signal processing approaches, paring the computing power present in most phones (to serve as AR displays as well as signal processors) and increased capabili- ties of common sensors, many applications for health and rehabilitation can be envisioned.
Of course, more specific study would be needed to confirm the types of tasks for which a TAR approach would be appropriate and valid. More guide- lines on building and validating AR models to assure that they are accurate enough to replace physical tasks would also be required. Finally, design and implementation of systems using the new guidelines and understanding are needed to validate their ability to lead to effective solutions.
Applying AR in rehabilitation has similar characteristics to product devel- opment. The representation of the task performed must be designed so that it accurately mimics real life actions for a user. Combined with accurate sen- sors and signal processing, commonly available hardware that many people already have can be utilized to improve health and well-being. These may be integrated with product-service system like designs that allow improved com- munication between patients and healthcare providers. Instead of requiring in person visits for physical therapy for example, a provider might pre- scribe a set of rehabilitation tasks for a patient to perform, transmit virtual instructions and details that will walk a patient through the proper/accurate method to do the tasks, measure the accuracy of the task to provide real time feedback, and update the provider both on accuracy as well as out- comes over time. There is potential not only for similarly positive outcomes, but also the ability to provide the same level of care more widely and cost effectively.
8 Choi and Fischer
REFERENCES
Anastassova, M., Burkhardt, J., Mégard, C., & Ehanno, P. (2005). Results from a user-centered critical incidents study for guiding future implementation of aug- mented reality in automotive maintenance. International Journal of Industrial Ergonomics. 35, 67–77. Barbieri, L., Angilica, A., Bruno, F., & Muzzupappa, M. (2013). Mixed prototyping
with configurable physical archetype for usability evaluation of product interfaces. Computers in Industry, 64(3), 310–323. Brown, W. F. (2013). The physiological and technical basis of electromyography.
Butterworth-Heinemann. Choi, Y. M., & Mittal, S. (2015). Exploring benefits of using augmented reality for usability testing. In DS 80–4 Proceedings of the 20th International Conference on Engineering Design (ICED 15) Vol. 4: Design for X, Design to X, Milan, Italy, 27–30.07. 15, (pp. 101–110). Choi, Y. M. (2019). Applying Tangible Augmented Reality for Product Usability
Assessment. Journal of Usability Studies, 14(4). Chowdhury, R. H., Reaz, M. B. I., Ali, M. A. B. M., Bakar, A. A. A., Chellappan, K., & Chang, T. G. (2013). Surface electromyography signal processing and classifi- cation techniques. Sensors (Basel, Switzerland). 13(9), 12431–12466. Cometti, C., Païzis, C., Casteleira, A., Pons, G., & Babault, N. (2018). Effects of
mixed reality head-mounted glasses during 90 minutes of mental and manual tasks on cognitive and physiological functions. PeerJ. 2018(11), 1–18. Esposito, D., Andreozzi, E., Fratini, A., Gargiulo, G. D., Savino, S., Niola, V., & Bifulco, P. (2018). A Piezoresistive Sensor to Measure Muscle Contraction and Mechanomyography. Sensors (Basel, Switzerland). 18(8), 2553. Faust, F. G., Catecati, T., de Souza Sierra, I., Araujo, F. S., Ramírez, A. R. G., Nickel, E.
M., & Ferreira, M. G. G. (2018). Mixed prototypes for the evaluation of usability and user experience: Simulating an interactive electronic device. Virtual Reality, 23(2), 1–15. Gerdle, B., Karlsson, S., Day, S., & Djupsjöbacka, M. (1999). Acquisition, Processing and Analysis of the Surface Electromyogram. Modern Techniques in Neuroscience Research. (Springer, Berlin, Heidelberg). Hargrove, L., Englehart, K., & Hudgins, B. (2007). A Comparison of Surface and
Intramuscular Myoelectric Signal Classification. IEEE Transactions on Biomedi- cal Engineering. 54, 847–53. Helin, K., Kuula, T., Vizzi, C., Karjalainen, J., & Vovk, A. (2018). User experience of augmented reality system for astronaut’s manual work support. Frontiers Robotics AI. 5(SEP), 1–10. Klein, C. S., Li, S., Hu, X., & Li, X. (2018). Editorial: Electromyography
(EMG) Techniques for the Assessment and Rehabilitation of Motor Impairment Following Stroke. Front Neurol, 9, 1122. McMahan, R. P., Bowman, D. A., Zielinski, D. J., & Brady, R. B. (2012). Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE Transactions on Visualization and Computer Graphics. 18(4), 626–633. Ong, S K & Yuan, Miaolong & Nee, Andrew. (2008). Augmented reality applications
in manufacturing: A survey. International Journal of Production Research. 46(10) 2707–2742. Park, Y. H., An, C. M., & Moon, S. J. (2017). Effects of visual fatigue caused by smartphones on balance function in healthy adults. Journal of Physical Therapy science. 29(2), 221–223.
Mixed Reality in Physical Rehabilitation, Opportunities and Challenges 9
Parker, P. & Scott, R. (1986) “Myoelectric control of prostheses,” CRC Critical Reviews in Biomedical Engineering. 13(4), 283–310. Pontonnier, C., Samani, A., Badawi, M., Madeleine, P., & Dumont, G. (2014).
Assessing the Ability of a VR-Based Assembly Task Simulation to Evaluate Phys- ical Risk Factors. IEEE Transactions on Visualization and Computer Graphics. 20(5), 664–674. Re, G. M., Bordegoni, M. (2014). An Augmented Reality Framework for Supporting and Monitoring Operators during Maintenance Tasks. HCI International: Virtual, Augmented, and Mixed Reality. Part II (LNCS 8526), 443–454. Srinivasan, V., W. S. Lovejoy and D. Beach (1997). “Sharing user experiences in the
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-20-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional