Yongho Lee, Nahyun Lee, Vinh Pham, Jiwoo Lee, Tai-myoung Chung (2023). Privacy Preserving Stress Detection System Using Physiological Data from Wearable Device. In: Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems. AHFE Open Access, vol. 69, pp. 340–347. https://doi.org/10.54941/ahfe1002853
Stress is considered to be an emotional state deserving of special attention, as it brings about harmful effects on human health when exposed to in the long term. Stress may also induce general health risks, including headaches, sleep disorders, and cardiovascular diseases. Continuous monitoring of emotion can help patients suffering from psychiatric disorders better understand themselves and promote the emotional well-being of the public in general. Recent advancements in wearable technologies and biosensors enable a decent level of emotion and stress detection through multimodal machine learning analysis and measurement outside of lab conditions. As machine learning solutions demand a large amount of training data, collecting and combining personal data is a prerequisite for accurate analysis. However, due to the highly sensitive nature of medical data, the additional implementation of measures for the preservation of user privacy is a non-trivial task when developing an AI-based stress detection solution. We propose a novel machine learning stress detection system that facilitates privacy-preserving data exploitation based on FedAvg, a renowned federated learning algorithm. We evaluated our system design on a standard multimodal dataset for the detection of stress. Experiment results demonstrate that our system may achieve a detection accuracy of 75% without jeopardizing the privacy of user data.
Emotion recognition is the act of recognizing and differentiating human affe- ctions. Detecting human emotion has a wide range of potential applications, especially in the domain of human-computer interfaces and healthcare. The implementation of affection detection in autonomous vehicles, human-robot interactions, video games, and recommendation systems are some of the most desired topics of research [Bota et al., 2019]. Continuous emotion monito- ring has the potential to help patients suffering from psychiatric disorders, and to promote the emotional well-being of the public, resulting in improved quality of life for many, along with making the prevention of mental illness possible in the first place.
Privacy Preserving Stress Detection System Using Physiological Data 341
In healthcare applications, stress is considered a noticeable emotional state, as it harms human health when exposed to in the long-term. According to the British Health and Safety Executive (HSE), stress and emotional difficulties accounted for half of the cases of all work-related illnesses in the UK [Owen, 2022]. Stress can also lead to general health problems, such as headaches, sleep disorders, and cardiovascular diseases. It may also decrease the work efficiency of employees. An effective and automated stress detection method may prove to be an excellent way to mitigate these problems.
Stress is a physiological response to a stimulus triggered by the sympath- etic nervous system (SNS). The response can differ depending on the subject and the measuring environment. Recent advancements in wearable tech- nologies and biosensors enable decent stress detection through multimodal machine learning analysis and off-lab measurement. Different sensors, inclu- ding photoplethysmography (PPG) sensors to measure blood volume pulse (BVP), thermometers to measure body temperature, and electrodermal acti- vity (EDA) sensors to measure galvanic skin response may be loaded on wearable devices. Data collected from these devices may in turn be used to train artificial intelligence (AI) models for the detection of stress and emotion.
However, the application of AI in healthcare has certain challenges. Mach- ine learning (ML), especially deep-learning (DL) solutions, demand a large amount of training data which may require a significant amount of time to collect. Also, due to the highly confidential nature of medical data, many privacy related regulations such as the General Data Protection Regula- tion (GDPR) are being actively introduced. While these legal protections are necessary, one cannot deny that they add on to the burden when developing AI solutions for medical applications.
Federated Learning (FL) is a learning paradigm that seeks to address the problem of data management and privacy by training algorithms collabora- tively without exchanging the training data itself. FL involves the training of statistical models between remote devices or siloed data centers, such as mobile phones or hospitals, all without sharing data. Therefore, FL is able to address the problem of training data deficiency and privacy-preservation [Rieke et al., 2020, Li et al., 2020]. Due to its significant advantage, FL is proposed as an appropriate machine learning concept in the context of heal- thcare. In this work, we aim to propose a stress detection system based on the concept of federated learning.
RELATED WORKS Physiological Data
Blood Volume Pulse (BVP) The blood volume pulse (BVP) signal is a mea- sure of changes in blood volume in arteries and capillaries. It is measured by shining an infrared light through human tissues and detecting their refle- ctions using PPG sensors. The BVP amplitude displays moment-by-moment heart rate variability (HRV) and may offer significant insight into individual emotional responses. Electrodermal Activity (EDA) In the last few decades, electrodermal acti- vity (EDA) has been one of the most popular physiological signals when
Table 1. Dataset distribution.
Table available in the original PDF paper.
Class # of samples Amusement 195 Baseline 628 Stress 355
concerning affective computing and emotion recognition. The detection of changes in sweat gland activity for specific stimuli has allowed for the expan- sion of the frontier in affective computing by providing real time autonomous nervous system (ANS) activity monitoring. The EDA signal is mainly divided into two parts: phasic and tonic components.
Skin Temperature (TEMP) Simple, accurate measurements of skin surface temperature, particularly taken by wearable devices, are of growing interest given their physiological significance.
Respiration Rate (RESP) The respiration and heartbeat signals are affected by the emotional status of the subject. The mechanical activities of the heart muscles and lungs induce a vibration inside the chest wall. A piezoelectric cantilever placed on top of the chest surface is capable of collecting such vibrations and converting them into an electrical voltage signal.
Dataset We employ the WESAD dataset [Schmidt et al., 2018] to evaluate the perfor- mance of our proposed system and compare the results. The WESAD dataset consists of raw data collected by the RespiBAN Professional (PLUX Wire- less Biosignals S.A., Portugal) wearable respiration monitoring system and the Empatica E4 (Empatica Inc., USA) wristband. This dataset features the physiological data of 15 subjects collected by the above devices, which are classified into three classes; Amusement, Baseline, and Stress. The labels were assigned using machine learning models, which were trained on sample data from different emotional states [Schmidt et al., 2018]. Tab. 1 describes the class distribution of the data in detail.
RespiBAN Professional The RespiBAN Professional is a chest-worn device capable of recording the electrocardiogram (ECG), EDA, electromyography (EMG), piezoelectric respiration (RESP), and TEMP data at sampling rates of up to 1000Hz [Bio, 2018]. All signals were sampled at a rate of 700Hz for the dataset. In this research, only RESP data was used for analysis.
Empatica E4 The Empatica E4 is a wrist-worn device which is capable of recording acceleration (ACC), BVP, EDA, and TEMP. ACC data was sampled at a rate of 32Hz, BVP at 64Hz, EDA at 4Hz, and TEMP at 4Hz. All types of data measured using the Empatica E4 were used in this research.
System Design We propose a system that analyses physiological data collected from weara- ble devices to determine the emotional state of a given individual, aided by
Privacy Preserving Stress Detection System Using Physiological Data 343
occasional user input via a dedicated mobile application, with an emphasis on privacy.
Data From Wearable Devices As our current work is focused on the privacy-oriented nature of the proposed emotional state classification system, we will simulate the collection of data from wearable devices by utilizing the WESAD dataset; described in Section 2.2. In the future, we will advance the proposed system based on directly performed data collection using a smart watch that is fully integrated into our system.
Custom Mobile Application We make use of a custom mobile application that is designed to periodically collect physiological data of choice from wearable devices. The mobile appli- cation performs the function of visualizing the various kinds of physiological data that was collected by the wearable device in use, and certain extra fea- tures that were derived from the initial set of raw data. Enabling user-aided emotion labelling and user interaction will be integrated to the application in the future. Fig. 1a demonstrates our custom mobile application.
Privacy Oriented Analysis Given the inherently private and sensitive nature of personal health data, it could be said that maintaining the privacy of personal health data while using it for analysis is of great importance. We attempt to tackle this issue by using the FedAvg algorithm, a promising federated learning algorithm. By implementing federated learning in our model, we rule out the possibility of user privacy breach by ensuring that the central server never handles raw data collected from wearable devices, but only uses the weights generated from the user’s device on premise. Fig. 1b illustrates the overall learning process of our system.
EXPERIMENT We conduct experiments to evaluate our proposed system using PyTorch as the machine learning framework with Python as the programming language of choice. To accelerate the training process of the proposed machine learning model, we use one NVIDIA RTX 2080 SUPER GPU with 8GB of memory.
Pre-Processing The raw data (BVP, RESP, TEMP) from sensors are processed and extracted into a form that can be employed in machine learning model training. The time series data is analyzed with a window frame of 30 seconds as a data point. We calculate the mean, standard deviation, minimum value, and maxi- mum value for each window frame. On EDA data, we applied the cvxEDA algorithm [Greco et al., 2015], which decomposes the measurement into the tonic, phasic, and additive noise term using the convex optimization. The components reflect changes in skin such as changes in skin conductivity and
(a) Custom Mobile Application (b) Privacy-oriented learning
Figure 1: Sample human systems integration test parameters (Folds et al. 2008)..Table 2. Complete list of features.
Table available in the original PDF paper.
Type of data Features BVP BVP-mean, BVP-std, BVP-min, BVP-max, BVP-peak-freq EDA phasic EDA-phasic-mean, EDA-phasic-std, EDA-phasic-min, EDA-phasic-max EDA smna EDA-smna-mean, EDA-smna-std, EDA-smna-min, EDA-smna-max EDA tonic EDA-tonic-mean, EDA-tonic-std, EDA-tonic-min, EDA-tonic-max Respiration rate Resp-mean, Resp-std, Resp-min, Resp-max Body temperature TEMP-mean, TEMP-std, TEMP-min, TEMP-max, TEMP-slope Personal measurement Age, Height, Weight
the sweat glands, thus enabling stress detection. We also calculated statistical values for the three types of data, respectively.
Tab. 1 shows the number of samples for each class we can extract from 15 subjects in the WESAD dataset. Also, the complete list of 29 features extracted from sensor data and personal measurement is shown in Tab. 2.
Training Procedure We simulate the training process with 14 clients on our custom designed fede- rated learning framework. The server first broadcasts a neural network model (Fig. 2 with all parameters initialized to zero. Each client owns the data from one subject from the WESAD dataset. The clients train the received model using the local data and send the model parameter values back to the server. The server aggregates the model weights from all the clients using the cano- nical FedAvg algorithm [McMahan et al., 2017]. The aggregated model is
Privacy Preserving Stress Detection System Using Physiological Data 345
Figure 2: Neural Network Architecture.
Figure available in the original PDF paper.
Table 3. Accuracy comparison between centralized training and the proposed federated learning approach.
evaluated at the server on unseen test data and is broadcasted back to the clients for further training. The entire process continues until the aggregate model converges.
The entire process mimics the scenarios in which multiple users separa- tely train the machine learning model on their personal data and contribute to the global model by providing only the trained model weights, thus not exposing the private data. We simulate the training process 14 times; each training includes 200 communication rounds, resulting in a fixed learning rate of 0.005.
EVALUATION The proposed federated learning stress detection system is simulated with 14 clients; each client is assigned to a single subject data from the WESAD dataset. Data of one subject is reserved as the unseen test data to evaluate the aggregated model after each round. This method is similar to the Leave One Out Cross Validation (LOOCV) method, which is also a widely used method for evaluating machine learning models. For comparison, we conduct traditi- onal centralized training; data from 14 subjects are used to train one model, and the remaining subject data is used for evaluation. The training hyper- parameters are kept constant during both federated learning and centralized learning. The accuracy values of the two methods are presented in Tab. 3 for comparison.
(a) Accuracy of each subject for unseen test data
(b) Accuracy over communication round
Figure 3: Evaluation results.
Overall, our proposed system shows an equal level of performance in com- parison to the traditional centralized system, where the users must agree to share their data with the server to train a single model on a central server. The accuracy of each method is evaluated with the unseen test data, which is presented in Fig. 3a. We can observe that the physiological data of the stres- sed state varies vastly according to each person. Thus personalization of the machine learning model is crucial in the stress detection system and will be considered in future research to enhance performance.
Lastly, the accuracy of the aggregated model over each communication round is presented in Fig. 3b. The plot shows that the aggregated model converges after 100 communication rounds, thus achieving the best accuracy possible at that stage. Nevertheless, as the volume of training data increases over time, increasing the number of communication rounds is necessary to guarantee that the aggregated model converges to its global optimum.
CONCLUSION
Physiological data from sensors embedded in wearable devices combined with artificial intelligence facilitates the detection of stress and emotion,
Privacy Preserving Stress Detection System Using Physiological Data 347
which brings benefits not only to patients with psychiatric disorders, but also to the general population. We proposed a stress detection system based on federated learning to help preserve the privacy of user data while engaging in analytic activities concerning user stress and emotional state. We demon- strated that our design could achieve the same performance with traditional centralized system designs. In the future, we plan to improve the strategy for updating the detection model with newly collected data over time as well as enhancing the robustness of the system to cope with heterogeneous data from malfunctioning sensors and user dropouts.
ACKNOWLEDGMENT
This work was supported by the Technology Innovation Program (or Industrial Strategic Technology Development Program-Source Technology Development and Commercialization of Digital Therapeutics) (20014967, Development of Digital Therapeutics for Depression from COVID19) funded By the Ministry of Trade, Industry & Energy (MOTIE, Korea).
REFERENCES
Bio, 2018. (2018). Biosignalsplux — wearable body sensing platform - RespiBAN Professional. https://web.archive.org/web/20181207012906/http://www.biosigna lsplux.com/en/respibanprofessional. Bota et al., 2019. Bota, P. J., Wang, C., Fred, A. L. N., and Pla’cido Da Silva,
H. (2019). A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals. IEEE Access, 7:140990–141020. Greco et al., 2015. Greco, A., Valenza, G., Lanata, A., Scilingo, E. P., and Citi, L. (2015). cvxeda: A convex optimization approach to electrodermal activity processing. IEEE Transactions on Biomedical Engineering, 63(4):797–804. Li et al., 2020. Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. (2020). Federa-
ted learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60. McMahan et al., 2017. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR. Owen, 2022. Owen, C. (2022). Hse publishes annual work-related ill-health and
injury statistics for 2021/22. Rieke et al., 2020. Rieke, N., Hancox, J., Li, W., Milletar‘ı, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., Ourselin, S., Sheller,
M., Summers, R. M., Trask, A., Xu, D., Baust, M., and Cardoso, M. J. (2020). The future of digital health with federated learning. npj Digital Medicine, 3(1). Schmidt et al., 2018. Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., and Van Laerhoven, K. (2018). Introducing wesad, a multimodal dataset for wea- rable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction, pages 400–408.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-45-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional