Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Movement Recognition to Analyze Disease-Related Changes in Motor Skills of Dementia Patients

RheinMain University of Applied Sciences, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1001024

Abstract

Currently, about 46.8 million people worldwide have dementia. More than 7.7 million new cases occur every year. Causes and triggers of the disease are currently unknown and a cure is not available. This makes dementia, along with cancer, one of the most dangerous diseases in the world. In the field of dementia care, this work attempts to use machine learning to classify the activities of individuals with dementia in order to track and analyze disease progression and detect disease-related changes as early as possible.In collaboration with several care communities, exercise data is measured using the Apple Watch Series 6. Consultation with several care teams that work with dementia patients on a daily basis revealed that many dementia patients wear watches. Thus, smartwatches provide an unobtrusive way to measure data.These devices have the following functions: global positioning system, accelerometer, inclinometer, gyroscope, magnetometer, heart rate monitor, oximetry sensor, skin conductivity sensor, and skin temperature sensor.The goal of this project is to gain knowledge about locating, providing, and documenting motor skills during the course of dementia.Caregivers will document patient activities while wearing the watch. Data from the aforementioned sensors are sent to the database at 20 data packets per second via a socket.DecisionTreeClassifier, KNeighborsClassifier, logistic regression, GaussianNB, RandomForestClassifier, Support Vector Machine, and Multilayer Perceptron classification algorithms are used. The test and training data are generated from different subjects to eliminate possible overfitting.The system transmits the labeled data on a six-second frequency (beams), in which 120 data sets are compressed from the previously mentioned sensors. Thus, it is possible to detect minute changes in arm positions. This methodology, after a six-month series of training and testing runs, shows results of over 90% probability for arm positions and over 80% probability for very fine granular activities (reading, playing games, eating), depending on the classification algorithm.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

INTRODUCTION

As a result of demographic changes, there are far more new cases of illness than deaths among those already ill. If there is no breakthrough in prevention and therapy, the number of dementia patients will increase to 74.7 million by 2030 and to around 131.5 million by 2050 according to population projections. In Germany alone, this corresponds to an average increase of around 40.000 dementia patients per year or around more than 100 per day, according to the German Federal Ministry for Health (Bundesministerium für Gesundheit, 2020). The shortage of junior staff due to the lower birth rate is leading to a decrease in population figures and a massive increase in people in need of care. This work deals with the machine tracking of activities during the course of dementia by means of sensor technology. Main part of this work is a system that is able to collect data from smartwatches in real time and send it to a server for further processing via a Web-Socket. With the collected data the following research questions could be addressed: To what extent can smartwatches be used to analyze people with dementia? What activities can be derived from the movement and acceleration data? In cooperation with several nursing communities, with this work the measurement of training data using Smartwatch starts. A consultation with various nursing teams that work with people suffering from dementia on a daily basis has shown that many patients wear watches. Smartwatches allow for an unobtrusive way of measuring data. These devices usually integrate the following: global posi- tioning system (GPS), accelerometer, light sensor, gyroscope, magnetometer, ambi- ent temperature sensor, heart rate monitor, oxymetry sensor, skin conductance sensor, and skin temperature sensor. This work answers the following question: how can fine- grained activities of dementia patients be classified using watch sensor technology and machine learning?

After this introduction, chapter 2 compares similar work with the present work. This is followed by chapter 3, which describes the data tracking setup. Chapter 4 explains the machine classification algorithm. Chapter 5 includes a presentation of results and chapter 6 concludes this work with a conclusion and a look into the future.

RELATED WORK

Health information technologies have been revolutionizing healthcare for years. The variety and range of software and hardware technologies as well as the number of applications has increased considerably. There is an increasing global demand for the implementation of health information technologies in hospitals, clinics, and homes according (Lau et al. 2019). In their work, they investigate the current status of mobile devices and software in relation to health information. In contrast to traditional health interventions originating from clinical researchers, mobile health applications are of- ten developed commercially with little input from clinical researchers or consumers. Portable devices such as smartwatches and fitness tapes are becoming increasingly popular in all demographic groups, from children to older adults. Reasons for this increase are fitness tracking and health monitoring. Detecting mental and physical

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 disorders and supporting people with difficulties can significantly improve the health of users, according to (Malu and Findlater 2016). Several of these applications are based on data collected by sensors on smartwatches, including heart rate monitor, GPS, accelerometer and gyroscope. Various interaction techniques make smart- watches unique and ubiquitous as a data tracking device. The literature supports this statement in various works of the past years. (Ravi et al. 2007) have successfully measured various human activities using an accelerometer. (Shoaib et al. 2015) have used both smartphones and smartwatches together to identify various daily human activities. (Dong et al. 2013) as well as (Ramos-Garcia and Hoover 2013) have meas- ured eating cycles of smartphones users. In these studies, accelerometers and gyro- scope sensor data from smartphones were used. (Da Silva and Galeazzo 2013) ob- tained various data on eight daily actions using accelerometer data, using an ez-430 Chronos smartwatch. It should be noted that the detection of general activities is pos- sible using accelerometers and gyroscope sensors. However, it is necessary to realize these activities for the health-related data in a much more fine-grained way. A crucial point is the arm movement detection. In their project in which only inertial sensors of the smartwatch were used, (Jose Manjarres et al. 2019) present the challenge of rec- ognizing human behavior by means of arm motion detection and its possibilities. They were able to calculate the workload according to the Frimat method using trained random forest with an accuracy of 97.5% in validation and 92% accuracy in real-time tests with 20 sub-jects. (Xu et al. 2015) classified hand and finger gestures as well as characters from smart-watch motion sensor data. Similarly, (Riaz et al. 2015) and (Tautges et al. 2011) attempted to reconstruct body movements using sev- eral portable devices by comparing accelerometer data with the data generated from motion detection. The present work is most similar to the project of (Serkan Balli et al. 2018). In their work, using an accelerometer and gyroscope in a smartwatch, the authors extracted 14 features from the obtained sensor data, condensed them through a dimensionality reduction algorithm filter and tested several methods (C4.5, SVM, random forest and kNN methods) for classifying human actions on five subjects to identify the following activities: brushing teeth, walking, writing on paper, writing with the keyboard, and vacuuming. The study shows how well machine activity clas- sifications can be real-ized using the sensor technology of smartwatches. For exam- ple, writing using the kNN method was rated with a success rate of over 98%. Ran- dom forest and C4.5 methods classifies the action walking with 100% accuracy. The present project extends the applied motion sensors (accelerometer and gyroscope) by the pedometer and heart rate sensors. The authors expect that this will represent an improvement beyond the state of the art.

This work demonstrates the potential that smartwatches offer for the healthcare sector. In the following, the prototype is described.

EXPERIMENT SETUP – DATA TRACKING For this work, a standalone watchOS application for the Apple Watch Series 6 was implemented using state-of-the-art technology. The application communicates with a WebSocket that both outputs the watch data packets to a user interface and stores

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 them in a MySQL database. The backup of the data is used for further machine pro- cessing. Figure 1 provides an overview of the work.

Figure 1: Overview The application provides methods for querying motion and health data, temporarily saving data in the smartwatch memory, labeling data, and an interface for exchanging sensor data with a web server via WebSocket. In case of complications, backup meth- ods can trigger a resend of sensor data generated in a session. Instead of caching sen- sor data in a CSV file on an iPhone, this work enables direct reuse of sensor data on the server side in real time.
Original PDF Figure 1:

The sensor technology in the focus of this work controls through the application in detail accelerometer, gyroscope, magnetometer, altimeter, GPS sensor, pedometer and heart rate sensor as well as the electrical heart sensor (ECG) and blood oxygen sensor.

In this work, processed data is queried instead of acquiring raw data from the sensor system. Raw data is data provided without prior correction. Accordingly, dis- turbance factors and external influences such as gravitational acceleration are in- cluded in the acceleration values of the three axes. Processed data, by contrast, is acceleration data that has been freed from distortions according to the (Apple Devel- oper Documentation 2021). The values are thus already adjusted for distortions such as acceleration caused by gravity. With respect to the accelerometer, Processed Data represent the user-initiated acceleration, but at the same time also provide the ex- tracted gravity values in three dimensions. In contrast to the query of individual raw data of the individual sensors, a query of processed data results in a complete CMDeviceMotion object which contains all motion data and hence data of the accel- erometer, the gyroscope, combined data such as the device orientation as well as the magnetic field vectors. Accelerometer, gyroscope and magnetometer contribute to the

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 mathematical calculation of the device orientation. The gravitational acceleration can be used to determine where south is, and the magnetic field vectors can be used to determine where north is from the device's point of view. Rotation values of the gy- roscope are integrated to estimate the deviation from the previous position. Thus, the combination of several sensors is used to calculate the attitude and to use the strengths of each sensor to compensate or minimize the weaknesses of each sensor. Since an acceleration sensor picks up any forces such as vibrations, this can cause unwanted noise. All movement and health data tracked by the realized application is first stored as arrays in a respective predefined structure and then encoded as JSON and then prepared for data exchange. If required, the user interface enables, among other things, the starting and stopping of tracking and will be examined in more detail be- low. Figure 2 shows the structure of the application's interface.

Figure 2: Structure of the application's interface Starting from the home page (see Figure 2 - 1), the app shows two functionalities that allow various actions. The first functionality establishes the connection with the server. The second one is responsible for locking the screen. This functionality for the dementia patients was introduced. The purpose behind it is to give the dementia patients the image that they are not wearing a smartwatch at all, so that the patients do not interact with the process and thus cannot disturb or stop it. For this purpose, an analog watch is displayed (see Fig. 2 - 2) and interaction with the app is prevented. This prevents the ill person from feeling stressed by tracking, fulfilling (Ali Darejeh's 2013) requirement. Rotating the Digital Crown can unlock the lock screen. Thus, pro- bands have a chance to access more features of the application. Among other things, labels can be created (see Fig. 2 - 4) and the list of labels can be displayed with re- spective buttons, such as "food". Great attention was paid to the consistency of the application, which according to (S. Sridevi 2014) is one of the golden rules in user interface design. All buttons are displayed in the same way and include an icon and a description of the action for a better overview. Only three buttons are given an addi- tional color to emphasize the main functions. These include starting and disconnect- ing the server connection as well as the stop button shown in screen 5. In addition, the data can be labeled with millisecond precision as well as monitored via a web interface as illustrated in Figure 3.
Original PDF Figure 2:

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 3: Monitored via a web interface
Original PDF Figure 3:

The label process can be started by the probands via the clock or the web interface as described and is used for subsequent classification by training the different classifi- cation models described below.

MACHINE LEARNING The data per second does not represent a temporal progression, but only a snapshot of the sensors. The data we tracked is at the temporal frequency of 20 hertz (Hz). Each of the data packets tracked at intervals of 50 milliseconds contains the following parameters: accelUserX, accelUserY,accelUserZ, attitudePitch, attitudeRoll, atti- tudeYaw, gravityX, gravityY, gravityZ, gyroX, gyroY, gyroZ, magneticFieldX, mag- neticFieldY, magneticFieldZ and heartrate.

When creating the test and training data, it was agreed that the movement would be approximately six seconds long. With one data record every 50 ms, this corre- sponds to 20 data records per second and thus 120 data records in total (20 data rec- ords/second x 6 seconds).The data was divided into blocks of 0.5 seconds, with the mean value over the values being calculated for each block. Thus, twelve values must be determined for each feature (12 x 0.5 seconds = 6 seconds). In the example of 120 lines (= 6 seconds), the average of a block is thus formed over 10 lines. With a label duration of 140 lines (= 7 seconds), this results in 11.6 (rounded down to 11) and thus eight lines remain (140 - [11x12]). In order to include these in the calculation of the average, they are distributed to the individual data sets (in this case to the first eight data sets). This ensures that there are always exactly twelve features for a value (e.g. AttitudePitch1, AttitudePitch2, ..., AttitudePitch12). The actual duration of the labels is thus variable, while the number of features remains constant. In summary, the length of the label determines the number of features, and the number of data per feature is determined by the hertz (Hz) of the sensor.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

EVALUATION The previously described methodology was applied to different classification algo- rithms in two series of experiments. The first series of experiments is the test for classification of three arm position sequences, the second is the test for classification of three activities. Five probands were involved. 60 labels per arm position sequence and activity were created. This results in 3 * 5 * 60 * 20 = 18,000 data sets per test series. The subjects wore the watch on their dominant hand at all times. The test and training data were created separately, i.e. the data of one proband were placed against the data of the other four probands. Figure 4 shows the respective arm position se- quences. Each sequence (A to C) consists of two arm positions, each held for 3 sec- onds. Arm position sequences were detected up to 99% accuracy.

Figure 4: Arm position sequences
Original PDF Figure 4:

The most frequently documented activities of the dementia living communities in- clude eating, drinking, and writing. The activities were detected to an accuracy of 95%.

CONCLUSIONS AND OUTLOOK The best results were obtained with the accelerometer, gyroscope, attitude and gravity features. After further tests with more than three arm position sequences, it can be concluded that the arm position sequences of a person can be detected with an accu- racy of 90% to 99% based on the data of accelerometer, gyroscope, attitude and grav- ity of the Apple Watch 6, depending on the duration of the arm position. After further testing with more than three activities, the authors conclude that a person's activities can be detected significantly more accurately or inaccurately with the Fast Forest, Logistic Regression, Support Vector Machine algorithms based on accelerometer, gy- roscope, attitude and gravity data from the Apple Watch 6, depending on the motion sequences of an activity. The more simultaneous the motion sequences, the more data (length of labels) is needed to get predicted accuracies above 90%. The authors will

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 attempt to identify the activities of individuals with dementia in real time using long short-term memory. The matching will then follow based on the documentation of the dementia living communities.

REFERENCES

Apple Inc. (2021) Understanding Reference Frames and Device Attitude | Apple Developer Documentation. Balli, S., Arif Sağbaş, E., Peker, M. (2018), “Human activity recognition from smart

watch sensor data using a hybrid of principal component analysis and random forest algorithm”, SAGE journals Bundesministerium für Gesundheit (2020), „Online-Ratgeber Demenz“ Da Silva, FG., Galeazzo, E. (2013), “Accelerometer based intelligent system for human movement recognition”, IEEE, DOI:10.1109/IWASI.2013.6576063 Darejeh, A. (2013), “A review on user interface design principles to increase soft-

ware usability for user with less computer literacy, University Kebangsaan Malaysia”, DOI: 10.3844/jcssp.2013.1443.1450 Dong, Y., Scisco, J., Wilson, M. (2013), “Detecting Periods of Eating During Free- Living by Tracking Wrist Motion”, IEEE, DOI:10.1109/JBHI.2013.2282471 Lau, F., Bartle-Clar, J., Bliss, G., Borycki, E., Courtney, K., Mu-Hsing Kuo, A.,

Kushniruk, A., Monkman, H., Vahabpour Roudsari, A.(2019), “Improving Us- ability, Safety and Patient Outcomes with Health Information Technology, IOS Press”, ISBN: 978-1-61499-950-8 Malu, M., Findlater, L. (2016), “Toward Accessible Health and Fitness Tracking for People with Mobility Impairments”, Proc. 10th EAI Int. Conf. Pervasive Com- put. Technol. Manjarres, J., Narvaez, P., Gasser, K., Percybrooks, W., Pardo, M. (2019), “Physi-

cal Workload Tracking Using Human Activity Recognition with Wearable De- vices, MDPI”, DOI: https://doi.org/10.3390/s20010039 Ramos-Garcia, RI., Hoover, AW. (2013), “A Study of Temporal Action Sequencing During Consumption of a Meal”, ACM Research2Guidance, mHealth app economics (2017), “Research Partners”, Ger-

many Ravi, N., Dandekar, N., Mysore, P. (2007), “Activity Recognition from Accelerometer Data on a Mobile Phone”, Springer, Berlin, Heidelberg, DOI: https://doi.org/10.1007/978-3-642-02481-8_120 Riaz, Q., Tao, G., Krüger, B., Weber, A. (2015), “Motion reconstruction using very

few accelerometers and ground contacts”, AMC Shoaib, M., Bosch, S., Scholten, H. (2015), “Towards detection of bad habits by fusing smartphone and smartwatch sensors”, IEEE, DOI:10.1109/PERCOMW.2015.7134104 Sridevi, S. (2014), “International Journal of Computer Science and Information

Technology Research”, Saveetha University Tautges, J., Zinke, A., Krüger, B., Baumann, J., Weber, A., Helten, T., Müller, M., Seidel, H., Eberhardt, B. (2011), „Motion reconstruction using sparse accelerometer data“, ACM, DOI: https://doi.org/10.1145/1966394.1966397 Xu, C., Pathak, P.H., Mohapatra, P., Finger-writing with Smartwatch (2015), “A

Case for Finger and Hand Gesture Recognition using Smartwatch”, AMC

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8988-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →