Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Behavior-based Understanding of Elderly People with Dementia: A Hierarchical Classification of Daily Object Use

1 Tokyo Institute of Technology, Meguro-ku Tokyo 1528552, Japan ↗
2 National Institute of Advanced Industrial Science and Technology, Koto-ku, Tokyo, 1350064, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1004385

Abstract

Providing individualized daily living care is quintessentially important to ensure the quality of life for elderly individuals, especially in nursing homes. Such care involves facilitating independent living, supporting social participation within nursing home settings, and preventing unintentional injuries such as falls. To effectively implement this, caregivers need to thoroughly understand the daily living activities of elderly people and to improve their living environments. The purpose of this study is to develop a system that can assist in planning residents' daily living care through automatically summarizing the daily activities in their rooms using depth cameras that respect privacy. The developed system consists of a function for extracting how elderly individuals use daily objects and another function for classifying behaviors based on the object use activities through hierarchical clustering. This system allows caregivers to understand the daily routines of the residents without predefining behaviors to be identified. To evaluate the effectiveness of the proposed method, the authors applied the method to analyzing 9 days’ worth of activities of an 87-year-old female resident in a nursing home. The experimental results demonstrate that the system was able to detect abnormal behaviors such as the repeated, unnatural use of drawers, without any predefined criteria for abnormal behaviors. The caregivers confirmed the utility of the system in summarizing daily behavior patterns and automatically detecting abnormal behaviors typically seen in elderly individuals with dementia.

INTRODUCTION

The population of nursing home residents is increasing every year. In the US, more than 1.3 million people reside in nursing homes, and in Japan, the number exceeds 950,000 (Center for Disease Control and Prevention, 2018; Ministry of Health, Labour and Welfare, 2023). Within nursing homes, understanding the daily activities of residents is essential to provide individ- ualized daily living care. In this paper, daily living care includes the following

© 2023. Published by AHFE Open Access. All rights reserved. 317

tasks: sensor selection to prevent hazards, environmental design to ensure the safety of elderly people, updating of the daily living care plan if abnormal behaviors are detected or daily activities change, and outcome evaluation of various interventions.

Concurrently, the fields of data science and behavior measurement tech- nologies have been advancing rapidly in recent years, making it possible to develop new technologies that enable a continuous understanding of daily living activities (Hamada et al., 2021).

While various sensors have been used in research for understanding activ- ities of daily living, this study focuses especially on RGB-D cameras. Recent innovations, such as Microsoft’s Azure Kinect, can simultaneously capture skeletal data, RGB images, and depth images (Microsoft, 2020). These RGB-D cameras do not need to be worn like wearable sensors, placing less burden on elderly people. They can also capture more detailed human motion and posture data compared to other sensors such as pyroelectric sensors, microphones, and conventional RGB cameras. This detailed information opens the way for a “behavior-based activity understanding,” allowing for the detection of behaviors indicative of conditions like dementia (Nishida et al., 2022). Hence, the authors have developed a system using depth cameras to intricately monitor elderly people’s movements without requir- ing any manual analysis by humans. To prioritize privacy protection, only depth images and skeletal data are recorded by the system developed by the authors.

Previous studies on using camera sensors to understand daily activities have covered topics such as fall detection (Abobakr et al., 2017) and activ- ity recognition (Burgess et al., 2013, Leightley et al., 2013). In these studies, to create models to recognize falls or gestures, the specific behaviors to be identified needed to be defined in advance. However, in the case of under- standing the behavior of elderly people, it is difficult to predefine every action due to unpredictable changes in activities caused by conditions like demen- tia, as well as the varying arrangements of furniture and welfare equipment across different facilities. Therefore, there is a strong need for a system that can summarize behaviors without requiring them to be predefined (Cicirelli et al., 2021).

In the present study, the authors propose a system that records daily activities using a privacy-conscious depth camera and automatically sum- marizes behaviors in living spaces without the need for predefining them. This is primarily aimed at assisting with the planning of daily living care. The system can elucidate behaviors based on human-object interactions. In this paper, the authors describe the details of this newly-developed sys- tem and present an evaluation of the proposed method by applying it to the behavioral data of an 87-year-old female in an actual nursing home environment.

Behavior-Based Understanding of Elderly People With Dementia 319

SYSTEM PROPOSAL FOR UNDERSTANDING ACTIVITIES OF DAILY LIVING FOCUSING ON HUMAN-OBJECT INTERACTIONS Necessary Functions for Understanding Activities of Daily Living Based on On-Site Needs Survey

The authors interviewed caregivers working in nursing homes to clarify on- site needs. The results indicated that caregivers need the following functions to understand the activities of daily living.

The first is the capability to understand how elderly people use everyday items such as a bed, a desk, a chair, etc. in their rooms. Caregivers place a high priority on providing an appropriate living environment for elderly people that minimizes injuries, such as falls. For example, some elderly individuals, accustomed to using futons before moving into the nursing home, encounter unfamiliar nursing beds upon admission. Unfamiliarity with these new beds might lead them to inappropriately attempt to climb over the bed rails to get off the bed, resulting in falls.

The second is the ability to summarize the daily behavioral patterns of elderly people to promote independent living. The layout and selection of furniture, as well as other welfare equipment, influences the ease with which one can stand up, move around, sit down, and so forth. Continuous improve- ment of living environments is important for facilitating social participation and independence.

In response to these requirements, this study develops a system that encap- sulates the above functions. An overview of the system for understanding the activities of daily living is shown in Figure 1. The proposed system accumu- lates skeletal data and depth images acquired from camera sensors installed in nursing home individual residents’ rooms. The system uses the acquired data to extract instances of human-object interactions. Then, the system classi- fies and visualizes the behaviors in order to summarize what behaviors were taken and when they occurred.

Figure 1: Schematic diagram of the proposed system.
Original PDF Figure 1:

Method for Summarizing Behaviors Based on Human-Object Interaction Activities Elderly people take on a variety of postures and perform various motions in living environments. To extract important behaviors and summarize them, it is necessary to segment meaningful behaviors by trimming the time-series behavior data. Many studies have been conducted to extract basic behaviors from behavioral data (Xu et al., 2023; Adama et al., 2021, etc.). A previous study segmented behaviors by focusing on the change in speed of the behav- ior, where the speed increases from a stationary state, decreases at a certain point, and then comes to a standstill (Das et al., 1998). However, one issue in this approach lies in the fact that the movement speed of elderly people is very slow, which makes it difficult to clearly observe speed changes. In this study, the authors assumed that interactions with everyday items occurs fre- quently in the individual residents’ rooms, and that the system can segment these recurrent behaviors by focusing on human-object interactions. Based on this assumption, our method for segmenting behaviors consists of a function for detecting contact with the object and a function for extracting behaviors during contact with the object.

Method to Detect Contact With Objects First, only the point cloud data of an object are extracted from the environ- mental point cloud data captured by the camera. The shortest distance lmin between the skeletal coordinates of the left or right hand and the object can be calculated as follows:

lmin = min

pi −phand

(1) i

where phand are 3D skeletal coordinates of the left or right hand, N is the number of point clouds of the object, and pi (i = 1, 2, · · · , N) are 3D coordinates of the i-th point in these point clouds.

Finally, the system uses a threshold for the distance between the skeletal coordinates and the object, and determines object contact to be when the shortest distance was less than this threshold.

Method to Segment Behavior Using Object Contact The system assumes that an elderly individual is engaging in some behavior using an object when in contact with it, and that this behavior concludes when contact with the object is interrupted for a period of time. Figure 2 shows the times of contact with an object derived using formula (1), where high piriods of the graph represent the recorded frames in which object con- tact was identified. The authors set a threshold for the time interval between contact with one object and the next instance of contact to segment the behav- ior. The system then segments the time-series skeletal data obtained during each interval, categorizing it as a single behavior.

Behavior-Based Understanding of Elderly People With Dementia 321

Figure 2: Method for segmenting behaviors from time-series data.
Original PDF Figure 2:

Methods for Summarizing Similar Behaviors by Classifying Them It is important for the system to be able to properly classify the activities of daily living. The proposed method of classifying behaviors involves calculat- ing the similarity of body movements and then grouping actions based on this similarity.

The acquired skeletal data is in a coordinate system centered on the cam- era. Classifying the skeletal data as-is would be affected by differences in posture position and orientation. Therefore, in this study, posture posi- tion and orientation were standardized to classify behaviors based on body movements.

Next, we classify the standardized behavior data using the dynamic time warping (DTW) distance method (Senin et al., 2008). As an example, we calculate the similarity of two time series datasets, S = s1, s2, · · · , sm, T = t1, t2, · · · , tn. When representing these series on the horizontal and vertical axes of a grid, a point

i, j

on the grid corresponds to an alignment of the elements si and tj.

The distance between two elements can be calculated as follows:

=

δ

i, j

si −tj

(2)

The DTW distance can be calculated as follows: K X

(3)

Mathematical equation reproduced from the original PDF
Equation reproduced from the original PDF.

wk

1 δ

where W = w1, w2, · · · , wK is the alignment of each element of the two time series datasets S and T so that the distance is minimized.

By performing the above operation for all combinations of M behavioral data, a DTW distance matrix of size M × M is created. The system classifies the behaviors using hierarchical clustering (Ward method) using the obtained DTW distance matrix.

Simulated Experiments on Classification of Behaviors First, the authors conducted a simulated experiment to evaluate the devel- oped functions for processing human-object interaction activities in a lab- oratory setting. An RGB-D camera (Azure Kinect, Microsoft) was used for accuracy validation. In the experiment, a participant performed five types of behaviors (labeled A through E) using a dresser commonly found in everyday settings as showin in Figure 3. The participant, a healthy 24-year-old female, performed each behavior five times. Between each action, there was a non- contact time of at least 10 seconds. The experiment was conducted with the approval of the Tokyo Institute of Technology’s Ethics Review Committee for Human Subjects Research and with the consent of the participant.

Figure 3: Photographs of the five behavior patterns relating to dresser use recorded for the validation experiment.
Original PDF Figure 3:

The system trimmed each obtained behavior dataset and performed hier- archical clustering to classify the behaviors by type. Figure 4(a) shows the clustering results for the 25 dresser-use behaviors. Figure 4(b) shows a two- dimensional visualization of the behavioral data using multidimensional scaling (MDS). In Figure 4, letters A through E indicate the clusters, and individual behaviors in the same cluster are denoted by A1, A2, . . . , A5. The experimental results show that the system was able to successfully clas- sify the dresser use behaviors into 5 clusters when adopting the red dotted line in Figure 4(a) as the threshold. In particular, even though behaviors C and D both involve opening and closing the lower drawers, they could be differentially classified by considering the differences in body movements.

Behavior-Based Understanding of Elderly People With Dementia 323

Figure 4: (a) Clustering of 25 behavioral data points, and (b) Visualization of 25 behavioral data points in 2D using multidimensional scaling (MDS).
Original PDF Figure 4:

VALIDATION OF THE PROPOSED SYSTEM IN AN ACTUAL NURSING HOME ENVIRONMENT This section describes the verification results of the proposed system in an actual nursing home room. The participant was an 87-year-old female with severe dementia. Data were collected for nine days. The everyday items included in the analysis were a dresser, a nursing bed, and a basin. This section specifically describes the results pertaining to classifying behaviors observed during dresser usage. Figure 5 shows an image of an individual residents’ room and examples of depth images of observed behaviors during everyday items usage.

Figure 5: Example of everyday items usage behaviors recorded by the Kinect camera.
Original PDF Figure 5:

From the acquired data, 58 distinct dresser usage behaviors were detected. Figure 6 shows the hierarchical clustering results of these 58 behaviors. The extracted behaviors were arranged in chronological order and numbered, and are shown below in the figure. When the classification threshold was applied

(where the dendrogram is cut by the red dotted line), the behaviors were classified into three clusters (labeled Clusters A, B, and C), as indicated by the colored frames in Figure 6. The authors manually checked the video data of the five behaviors included in Clusters B and C. These videos were cre- ated from depth images. The authors confirmed that the five behaviors were repetitive drawer opening and closing behaviors, and that the behaviors in Cluster B included more body bending movements when opening and clos- ing the lower drawers. These results demonstrate that the system is capable of classifying repetitive behaviors, which is characteristic of patients with dementia.

Figure 6: Clustering of 58 behaviors recorded in an actual nursing home setting.
Original PDF Figure 6:

The system also features a function to display videos of primary behaviors. The user manually selects the everyday items and number of clusters, which allows caregivers to set the level of the detail of the behaviors they want to check. For example, for a broad, daily overview of an elderly individual’s behavior patterns, the caregiver can set the number of clusters to be low. In contrast, if the caregiver wants to view the elderly individual’s behavior patterns in detail, they can set the number of clusters to be higher, which provides a more in-depth understanding of daily behaviors.

Next, we visualized when and how often the 58 behaviors in the three clusters occurred. Figure 7 shows an example visualization of 37 behaviors observed across February 14, 15, and 16. The horizontal axis shows the time of day, and the vertical axis shows the number of behaviors within each cluster, tabulated every 30 minutes. We confirmed that repetitive behaviors occurred more frequently on February 15 when we compared the number of behaviors in each cluster over a 3-day period. By tracking the type and fre- quency of daily behaviors, it will be possible to understand daily variations

Behavior-Based Understanding of Elderly People With Dementia 325

and lifestyle patterns. Ultimately, this aids in understanding an individual’s baseline and identifying changes over the long term, in a non-ergodic manner.

Figure 7: Visualizing the timing and frequency of 37 behaviors that occurred between the 14th and 16th days.
Original PDF Figure 7:

The authors also confirmed the utility of the system to caregivers by show- ing them videos of behaviors classified using the hierarchical clustering and visualizations of the number of behaviors. The caregivers indicated that the repetitive behaviors extracted by the system, which are behaviors unique to patients with dementia, were crucial information for effective caregiving. They also commented that the system was effective for caregivers with lim- ited time, as it summarized activities of daily living and automatically filtered for only the necessary information. Thus, the proposed system could enhance caregivers’ understanding of the behavior of elderly people.

CONCLUSION

In this study, the authors proposed a system to help caregivers develop appro- priate daily living care plans for nursing home residents. This system can facilitate the understanding of daily behavioral patterns without the need for predefining the behaviors to be recognized. The system consists of two func- tions: the first is to extract human-object interaction activities using skeletal data and depth images acquired from camera, and the second is to classify behaviors based on body movements. Deploying the system in the living quar- ters of a resident with dementia at an actual nursing home, 58 behaviors tied to dresser usage were detected. Among them, 5 behaviors were identified as repetitive behaviors, which are characteristic of patients with demen- tia. These results demonstrate that the system was able to automatically detect abnormalities among various behaviors occurring over several days. In addition, visualizations of when and how often the classified behaviors occurred enabled day-by-day comparisons, facilitating a better understand- ing of the behavior patterns. An evaluation of the system’s outputs by the caregivers also suggested that the proposed system is useful for understanding the activities of daily living of elderly people.

In the future, the authors plan to improve the system to automatically extract daily variations in human-object interaction activities. The daily activities of elderly people undergo constant changes due to decline in life functions and differences in living environment. The authors expect

that understanding changes in behavior in a non-ergodic manner can assist caregivers in planning appropriate daily living care for their residents.

ACKNOWLEDGMENT

The authors would like to express their gratitude to Ms. Kazuyo Maruyama and Mr. Shogo Ito for their cooperation in the nursing home needs sur- vey. This work was supported in part by JST Grant Number JPMJPF- 2101 and was commissioned by the New Energy and Industrial Technology Development Organization (JPNP20006).

REFERENCES

Abobakr, A., Hossny, M. and Nahavandi, Sa. (2017) “A skeleton-free fall detection system from depth images using random decision forest”, IEEE Systems Journal 12.3: 2994–3005. Adama, D. A., Lotfi, A. and Ranson, R. (2021) “Adaptive segmentation and

sequence learning of human activities from skeleton data”, Expert Systems with Applications 164: 113836. Burgess, R. A., Hartley, T., Mehdi, Q. and Mehdi, R. (2013) “Monitoring of patient fluid intake using the Xbox Kinect”, In Proceedings of CGAMES’2013 USA, pp. 60–64, IEEE. Centers for Disease Control and Prevention, (2018) “Nursing Home Care”, Website:

https://www.cdc.gov/nchs/fastats/nursing-home-care.htm Cicirelli, G., Marani, R., Petitti, A., Milella A. and D’Orazio, T. (2021) “Ambient Assisted Living: A Review of Technologies, Methodologies and Future Perspec- tives for Healthy Aging of Population;” Sensors, Vol. 21, No. 10, Article 3549. Das, G., Lin, K. I., Mannila, H., Renganathan, G., & Smyth, P. (1998) “Rule

Discovery from Time Series”, In KDD Vol. 98, No. 1, pp. 16–22. Hamada, M., Kitamura, K.and Nishida, Y. (2021) “Individual and longitudinal trend analysis of stairway gait via ambient measurement using handrail-shaped force sensor”, IEEE International Conference on Sensors. Leightley, D., et al. (2013) “Human activity recognition for physical rehabilitation”,

2013 IEEE International Conference on Systems, Man, and Cybernetics. IEEE. Ministry of Health, Labour and Welfare, (2023) “Report on the status of long-term care insurance business (preliminary)”, Website: https://www.mhlw.go.jp/topics/ kaigo/osirase/jigyo/m23/2301.html Misrosoft, (2020) “Azure Kinect”, Website: https://azure.microsoft.com/ja-jp/prod

ucts/kinect-dk/ Nishida, Y., Kitamura, K. and Hagiwara, H. (2022) “Elderly behavior video database for innovative and safe universal design of environment”, Injury Prevention, Vol. 28, supple 2. Senin, P. (2008) “Dynamic time warping algorithm review”, Information and

Computer Science Department University of Hawaii at Manoa Honolulu, USA 855.1–23: 40. Xu, L., et al. (2023) “Skeleton-based Tai Chi action segmentation using trajectory primitives and content”, Neural Computing and Applications 35.13: 9549–9566.

Cover of the proceedings volume Emerging Technologies in Healthcare and Medicine

Explore the complete digital volume

Emerging Technologies in Healthcare and Medicine

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-92-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →