Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Non-contact physiological monitoring of heart rate, facial temperature, and respiration rate with thermal and RGB cameras

1 Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, France ↗
2 Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, France; IPAL Singapore International Research Laboratory on Artificial Intelligence, 138632 Singapore ↗
Publication year
Digital Object Identifier10.54941/ahfe1005731

Abstract

In this paper, we evaluated a non-contact physiological measurement technique using a thermal camera and an RGB camera aimed at the participant's face. The thermal camera effectively measured the temperature of specific facial regions, such as the tip of the nose, which is related to stress and mental workload. It also accurately measured respiration rate, which is an important indicator of mental state. On the other side, the RGB camera successfully measured heart rate by detecting subtle color changes in the face. However, the thermal camera was not effective in measuring heart rate, possibly due to a lack of thermal sensitivity and image resolution. Overall, our results confirmed that using thermal and RGB cameras can be a practical and discreet method for monitoring an individual's mental state. Additionally, these cameras can monitor movements and detect states of medical incapacitation, such as loss of consciousness.

INTRODUCTION

It is widely recognized that human operators controlling complex systems such as aircraft, trains, or nuclear power plants are sometimes exposed to high levels of stress and mental workload (Causse et al., 2017; Wickens, 2008). Many researchers in human factors, neuroscience, and affective computing have attempted to implement physiological (e.g., heart rate, respiration rate…) and neurological measures to assess the mental state of operators (Kramer, 2020). This would allow to detect any degraded cognitive states such as cognitive incapacitation, resulting from excessive stress or mental workload. Unfortunately, these measures are often complex to use in the workplace. For example, fNIRS provides direct measurements of brain activity but is too cumbersome and tedious for operational use. Similarly, physiological measurements often require the placement of electrodes, which can cause discomfort to the operator.

© 2024. Published by AHFE Open Access. All rights reserved. 1633

One solution that has been explored for several years is remote physiology (Hurter & McDuff, 2017; McDuff, 2023; McDuff et al., 2017). The idea is to perform physiological measurements from a distance in a non-contact manner, for example using cameras aiming at the face. An advantage is that remote physiology can be easily conducted in realistic conditions such as flight simulators (Bonyad et al., 2024; Hassoumi et al., 2022). Detecting heart rate (HR) with RGB cameras filming the human face has already been proven possible (Poh et al., 2010), even though this technique is very sensitive to movements. When the heart beats, blood flows through the face, producing subtle changes in skin color that can be captured by an RGB camera after various signal processing techniques. However, RGB cameras may be unusable in the dark or in the presence of smoke, which can happen in a cockpit, for example.

A complementary technique is to use a thermal camera. Face temperature, in particular nose tip and forehead, can be recorded to collect clues about stress and mental workload levels (Abdelrahman et al., 2017; Bonyad et al., 2024; Hassoumi et al., 2022). Additionally, recording the nostril temperature can provide respiration rate (RR), another metric related to stress and mental workload. The advantage of thermal cameras is that they can still monitor face temperature and movements in dark and smoky conditions. In this paper, we present a proof of concept of the combined use of these two techniques. We synchronously collected data from RGB and thermal cameras to assess their efficiency during a memory task (n-back). The thermal camera was used to record facial temperature, especially at the nose tip, and to monitor breathing rate. Both RGB and thermal cameras were assessed for their ability to collect heart rate.

METHOD

Both cameras were pointed toward the participant’s face. The thermal imprints were recorded using an Optris Xi 400 infrared thermal camera (Optris GmbH, Germany) with an optical resolution of 382 × 288 pixels, a frame rate of 27 Hz, 80 mK thermal sensitivity, and the capability to collect thermal radiation in the 8–14 µm band. The instrument was black-body calibrated for thermometrically accurate measurements. The RGB camera was a standard webcam from a laptop computer. Both thermal and RGB cameras were used to measure HR by analyzing color or temperature value changes in the facial skin. Additionally, the thermal camera was used to measure facial skin temperature (e.g., nose tip, forehead) and RR by detecting temperature changes in the nostrils. For RR, the thermal camera had to be positioned correctly, a bit below the face, to efficiently capture the nostril temperature.

Non-Contact Physiological Monitoring of Heart Rate, Facial Temperature 1635

Figure 1: The different processes followed in this work to measure heart rate, respiration rate, and face temperature (nose tip) with the RGB and thermal cameras. Although heart rate was also assessed with the thermal camera, it did not yield reliable results and thus is not indicated in the figure.
Original PDF Figure 1:

Heart Rate Calculation With Thermal and RGB Signals Regarding HR, we first implemented a standard approach for face recognition and signal processing to monitor physiological parameters using both thermal and RGB cameras. For this, we were inspired by preliminary methods (McDuff, 2023; Poh et al., 2010). After each heartbeat, the skin color changes slightly due to the incoming blood; this is the phenomenon we intended to capture.

Face Recognition: A mesh was created around the participant’s face using the Mediapipe library. Inside the created mesh, the nodes from the forehead were connected to create a mask. This mask was colored in green for easy visualization and represents the ROI (Region of Interest) used for data acquisition (see Figure 1 with the forehead ROI).

Figure 2: Face mesh recognition of a subject with ROI detection in green using the Mediapipe library.
Original PDF Figure 2:

Signal processing. For the RGB camera, the average color values of the ROI were calculated for each frame, providing three signals (red, green, blue). Signal processing involved normalizing the raw signal using z-score,

followed by high- and low-pass filtering to isolate the heartbeat frequency. A power spectral density (PSD) was applied on the final signal, see Figure 2. In order to further improve results, an independent component analysis (ICA) was then applied to further isolate the heartbeat signal from the three-color channels, improving the PSD results. Validation with an oximeter finger pulse confirmed the accuracy of the detected heartbeat frequency. The color yielding the best result was the green.

Figure 3: Heart rate signal processing example for a face ROI with the green color of the RGB camera. From up to bottom. Average raw signal in the ROI; z-score normalization; High pass filtering (HPF); low pass filtering (LPF). A power spectral density (PSD) was applied on the final signal. It shows a peak at 0.96 Hz frequency, representing a heartbeat of 57.6 bpm, the most present frequency in the studied signal, which is consistent with the heart rate of the considered participant.
Original PDF Figure 3:

For exploratory purposes, we also attempted to extract HR from the thermal camera output. While a similar heart rate extraction pipeline was possible with the thermal camera using a single channel (temperature) instead of the three color channels from the RGB camera, our results did not show any conclusive outcomes. We hypothesize that the sensitivity of our thermal camera may not be sufficient to accurately capture temperature

Non-Contact Physiological Monitoring of Heart Rate, Facial Temperature 1637

variations. Additionally, the thermal noise produced by the thermal camera could possibly be higher than the user’s thermal variations.

Breathing Rate Measure With the Thermal Camera We investigated the RR with the thermal camera, which captured the temperature difference as air was breathed in and out through the nostrils. We calculated the average temperature of each pixel in the ROI for every frame.

Signal processing. Signal processing involved applying a z-score to eliminate trends, followed by a bandpass filter at frequency range 6 to 30 Hz to isolate the respiration rate. A PSD calculation was then applied to determine the respiration rate, similar to the method used for heartbeat extraction. In order to carry out experiments, a Python application was developed, see Figure 3. It contains all the required information for the experimenter, in order to verify that the data acquisition is being done as intended.

Figure 4: The developed application performs the recording of the different metrics. In the illustration, the nose tip temperature (32.6◦C), respiration rate (15 respiration per minute), and heart rate (60 beats per minute) are visible. The application also displays a live video feed from the thermal and RGB cameras.
Original PDF Figure 4:

EVALUATION OF THE PERFORMANCE OF THE MEASURES The reliability of the HR and RR measures was assessed on 7 student participants during the performance of a memory task (n-back). For HR, an oximeter was used as a ground truth validation. Facial temperature measurements (e.g., nose tip) were reliable but not evaluated here. The tests were conducted in a well-lit room. The subjects were positioned in front of the laptop camera at a distance of approximately 30 cm from the screen. The thermal camera was positioned on the table at an angle of 40

  • to point directly at the subject’s nostrils. The experimental setup is schematised in
Figure 4. The oximeter was placed in the subject’s index finger, and the test would not begin until a stable heart rate was obtained from the oximeter.

Figure available in the original PDF paper.

Figure 5: Illustration of the validation experiment with RGB (webcam) and thermal cameras.
Original PDF Figure 5:

Table 1 shows performance for HR measurements with the RGB camera, with an average of 58.15 bpm for the considered participant, and a very low minimum average error (MAE) in comparison to the oximeter, showing good accuracy of the measure with the RGB camera. Respiration rate measures with the thermal camera was consistent with expected values in human participants, see also Table 1.

Table 1. Heart rate and respiration rate measures during an n-back task.

Table available in the original PDF paper.

Variable Mean Median Standard Deviation MAE HR 58.15 59 3.96 0.15 RR 14.96 14 2.28 -

Below, we display an illustration of the signal for the respiration rate obtained with the thermal camera, Figure 5.

Figure 6: illustration for one participant of the respiration rate measure in breath per minute with the thermal camera.
Original PDF Figure 6:

Non-Contact Physiological Monitoring of Heart Rate, Facial Temperature 1639

CONCLUSION

In this work, we could achieve the measurement of face temperature and breathing rate with a thermal camera, as well as a reliable heart rate detection with an RGB camera. These measurements could be validated during an n- back task performed by seven participants Only, the measure of the heart rate with the thermal camera yielded poor results, maybe due to lack of resolution and temperature sensitivity. Overall, our preliminary findings suggest that combining RGB and thermal cameras could be a viable solution for real-time assessment of a subject’s mental and physiological state, offering valuable insights into their current condition.

The measurement of physiological parameters using RGB and thermal cameras is a rapidly developing field with potential applications in healthcare, sports, security and more. However, several challenges need to be addressed to make these technologies more effective and accurate (sensor calibration, noise reduction etc.). Future works should aim at increasing the reliability of these measures to participant movements. It is also necessary to carry out thorough studies to compare the physiological measurements obtained with these cameras with traditional methods. The robustness to environmental conditions is also an important point, and measurements should be evaluated under variable lighting conditions and ambient temperature. The integration of AI is also an important aspect to improve facial recognition, the accuracy of measurements, and the ability to identify complex patterns in the data. AI may also help make these measurements more robust to movements. Future studies may also investigate how to make thermal cameras able to reliably measure heart rate.

ACKNOWLEDGMENT

This research was supported by a chair grant from Dassault Aviation (“CASAC”).

REFERENCES

Abdelrahman, Y., Velloso, E., Dingler, T., Schmidt, A., & Vetere, F. (2017). Cognitive heat: Exploring the usage of thermal imaging to unobtrusively estimate cognitive load. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3), 1–20. Bonyad, A., Abdessalem, H. B., & Frasson, C. (2024). The relation between mental

workload and face temperature in flight simulation. Journal of Behavioral and Brain Science, 14(2), 64–92. Causse, M., Chua, Z., Peysakhovich, V., Del Campo, N., & Matton, N. (2017).

Mental workload and neural efficiency quantified in the prefrontal cortex using fNIRS. Scientific Reports, 7(1), 1–15. https://doi.org/10.1038/s41598-017- 05378-x Hassoumi, A., Peysakhovich, V., Le Coz, A., Hurter, C., & Causse, M. (2022).

Thermal Imaging of the Face: Mental Workload Detection in Flight Simulator. Hurter, C., & McDuff, D. (2017). Cardiolens: Remote physiological monitoring in a mixed reality environment. In ACM siggraph 2017 emerging technologies (pp. 1–2).

Kramer, A. F. (2020). Physiological metrics of mental workload: A review of recent progress. Multiple Task Performance, 279–328. McDuff, D. (2023). Camera measurement of physiological vital signs. ACM

Computing Surveys, 55(9), 1–40. McDuff, D., Hurter, C., & Gonzalez-Franco, M. (2017). Pulse and vital sign measurement in mixed reality using a HoloLens. 1–9. Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010). Non-contact, automated cardiac

pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762–10774. Wickens, C. D. (2008). Multiple resources and mental workload. Human Factors, 50(3), 449–455.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →