Philipp Román, Eva Maria Knoch (2025). Wide-Angle Thermal Sensing for Personalized Climate Control: An Infrared Fisheye Camera Approach in Commuter Vehicles. In: Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems. AHFE Open Access, vol. 160, pp. 650–657. https://doi.org/10.54941/ahfe1005871
Thermal comfort is a critical aspect of electric vehicle (EV) design, particularly in shared mobility scenarios where passengers with diverse comfort preferences frequently enter and exit. Current climate control systems are primarily reliant on traditional sensors such as those for temperature, sun load, air quality, etc. However, these systems are often unable to provide personalized thermal comfort in dynamic environments, especially in autonomous vehicles where passenger numbers, seating arrangements, and environmental conditions can change significantly. In this study, a novel solution is proposed, utilizing a single infrared (IR) fisheye camera to monitor and optimize thermal comfort across the entire vehicle cabin. The camera is used to provide a full 360-degree view of the vehicle interior, allowing simultaneous monitoring of postures, body temperature, heat dissipation, and environmental factors such as solar heat gain or cold air drafts. This approach enables the dynamic tracking of multiple passengers, ensuring that changes in occupancy and cabin conditions are accommodated in real time. While the focus of the study is placed on an autonomous electric commuter vehicle (AECV) developed at the Karlsruhe Institute of Technology and its partners, the approach is applicable to a range of other vehicles and transportation systems.To process the thermal data, machine learning and deep learning techniques are employed. Spatial features are extracted from the thermal images using convolutional neural networks, allowing patterns such as body temperature distribution and localized hot or cold zones to be identified. Additionally, temporal changes in passenger movement and cabin conditions are modeled using recurrent neural networks, or more specifically, long short-term memory networks, enabling the prediction of thermal preferences based on historical and real-time data. Training of these models is conducted on datasets that combine thermal imagery with contextual behavioral data, such as posture and gestures, detected by a standard fisheye camera in previous studies.The predictions generated by the system are designed to guide real-time HVAC adjustments to provide personalized comfort. For example, cooling is applied to areas where passengers are exposed to sunlight, while targeted heating is activated for those in cooler zones. Validation of the system is planned through controlled laboratory experiments and real-world trials in shared mobility scenarios. These evaluations will assess the ability of the system to monitor thermal comfort accurately, respond dynamically to changes in occupancy, and optimize energy usage when compared to traditional HVAC systems.This single-camera approach is designed to streamline design, reduce costs, and enable advanced thermal comfort systems in dynamic environments. While its primary focus is on optimizing thermal conditions in electric vehicles, the system also presents opportunities to address motion sickness, a common issue in autonomous vehicle passengers. Additionally, its potential applications extend beyond transportation, offering promising avenues in areas such as indoor heating and cooling.
Thermal comfort of passengers in vehicle cabins is a multifaceted phenomenon influenced by a wide range of factors. Traditional air conditioning systems in vehicles often fail to cater to the individualized thermal preferences and perceptions of passengers (Diga et al., 2021; Yun et al., 2021; He et al., 2022; Donsì et al., 2022). This shortfall arises from a limited or overly simplistic approach to addressing the complex interactions between psychological and physiological determinants of thermal comfort (Chen et al., 2024).
A promising strategy for addressing these challenges lies in the development of predictive models capable of accurately estimating passengers’ thermal perception based on thermal imaging data and relevant environmental variables (Zhou et al., 2024). These models enable air
Wide-Angle Thermal Sensing for Personalized Climate Control 651
conditioning systems to respond in real time, dynamically adjusting the cabin climate to align with individual passenger comfort requirements.
Thermal imaging provides a precise and immediate method for capturing the human body’s thermal response to varying climatic conditions. Existing studies have explored the use of conventional thermal cameras in vehicle settings to monitor localized temperature distributions for improving HVAC control (Yang et al., 2020). However, these approaches are often limited by their restricted field of view (Zheng et al., 2024), making them less effective for monitoring dynamic and shared mobility scenarios with multiple passengers. To address this limitation, this study leverages a custom-designed 360-degree fisheye infrared camera to comprehensively capture thermal data across the entire cabin, providing a holistic view of passenger thermal behavior and environmental interactions.
This research is part of the C2CBridge project led by the Karlsruhe Institute of Technology in collaboration with the Karlsruhe Mobility High Performance Center, which focuses on developing next-generation mobility solutions for shared transportation in autonomous vehicles. One of the project’s goals is to enhance the passenger experience by improving factors such as thermal comfort using innovative HVAC systems. By integrating this approach into the C2CBridge vehicle, we aim to create a more personalized and efficient climate control system.
The aim of this study is to develop a robust and efficient predictive model for assessing passengers’ climatic perception. By incorporating this model into the air conditioning system of the C2CBridge vehicle, we anticipate a significant improvement in passenger thermal comfort. In addition to enhancing individual satisfaction, these advancements could contribute to the broader acceptance and adoption of this mode of public transportation.
METHODS AND MATERIALS This study introduces a thermal comfort prediction system that utilizes an array of MLX90640 long-wave infrared (IR) cameras to monitor and assess the thermal states of passengers within vehicle cabins. All temperature measurements are recorded in degrees Celsius (
C), and all scales in future figures within this study will also be presented in Celsius.
The cameras are temperature calibrated using a blackbody reference source with emissivity ε = 0.98, consistent with human skin emissivity. A flat-field correction technique is employed to remove fixed-pattern noise, ensuring uniform pixel response across the entire array. The sensor outputs are temperature-compensated using Equation (1): Tcorrected = Tmeasured −Tambient + Calibration Offset
Where: Tcorrected: Raw temperature reading from the pixel Tambient: Camera core temperature measured via onboard thermistor
Calibration offset is experimentally determined for each camera
652 Román and Knoch
The cameras have a resolution of 32x24 pixels and a field of view of 110
by 75
. Due to the high cost of commercially available 360-degree thermal cameras, multiple MLX90640 sensors are arranged to simulate a 360-degree thermal view. To fabricate this view, four MLX90640 sensors are positioned 90
from each other to maximize overlap and coverage. Custom software seamlessly stitches the individual thermal images from each sensor into a unified panoramic thermal map of the vehicle interior. Homography transformations and bilinear interpolation serves to align the overlapping regions and to eliminate the spatial gaps between the adjacent cameras. For clearer analysis, the figures presented in this study use the expanded 360
rectilinear view, rather than the wide, distorted fisheye-style 360
view.
To assess the thermal sensation and comfort of passengers, this study uses the “human thermal comfort model” according to H. Zhang (2003). Like Zhang we use a Thermal Sensation Vote (TSV) scale, where −4 corresponds to extremely cold, 0 indicates a neutral sensation and +4 represents a sensation of extremely hot. This is supplemented by a comfort scale, where −4 means very uncomfortable and +4 is very comfortable.
Data Acquisition Thermal data acquisition is conducted using the centrally positioned 360-degree camera array (Figure 1), providing a comprehensive view of the vehicle cabin and its occupants. This configuration eliminates the need for multiple cameras or environmental sensors, simplifying hardware requirements. The multi-camera array captures thermal images using the Ironbow/Inferno palette, where pixel color corresponds to relative surface temperatures. These images are recorded at a frame rate of 5 frames per second.
Figure 1: Camera array mounted on the interior roof of the cabin (self-created).
The experimental environment simulates various cabin conditions. These consist of the Static scenarios where passengers remain seated in pre- determined configurations, allowing baseline calibration of the system for thermal and positional variations. On the other hand, the Dynamic scenarios simulate Passengers who enter, exit, or change positions to test the system’s
Wide-Angle Thermal Sensing for Personalized Climate Control 653
adaptability to real-world mobility patterns. Thermal imaging is performed under different ambient temperature and airflow settings to reflect diverse operating conditions (Figure 2). The data was collected in a Mercedes Vito 9-seater. Participants provide self-reported TSVs on a scale of extremely cold (−4) to extremely hot (+4) respectively very uncomfortable (−4) to very comfortable (+4) regularly during the whole experiment, offering subjective thermal comfort feedback to label the collected thermal data. Demographic details, including age and gender are recorded to ensure dataset diversity.
Figure 2: View from the thermal camera array, showing one passenger seated in the rear left seat (self-created).
Thermal Feature Extraction and Analysis Once thermal data is acquired from the camera array, the subsequent stage focuses on preprocessing the images, extracting relevant features, and analyzing dynamic passenger behavior. Given the low resolution of each individual thermal sensor, each image follows a pre-processing pipeline which incorporates image upscaling and noise reduction techniques to enhance spatial detail while preserving critical features. Gaussian smoothing with a kernel size of 3 × 3 pixels is applied to reduce random thermal noise, while bilateral filtering is used to increase the effective resolution while maintaining fine details. Using bicubic interpolation, the images are upscaled to a target resolution of 128 × 96 pixel. Figure 3 depicts an unprocessed image, while Figure 4 shows an image after preprocessing.
To isolate passenger-specific thermal regions and ensure accurate feature extraction, the object detection algorithm YOLOv5, is utilized. Regions of interest (ROIs) representing passenger-specific thermal zones, such as faces, torsos, and seating areas, are identified in the thermal images. These regions are then dynamically segmented from the stitched panoramic view, ensuring that localized thermal data is captured for each occupant. To improve accuracy, the identified ROI is normalized to compensate for varying thermal image brightness caused by external factors, including sunlight or reflections.
After preprocessing, the cropped and normalized ROIs undergo spatial feature extraction to identify thermal patterns and heat distribution. A convolutional neural network (ResNet-50) with several convolutional and pooling layers (see Table 1) to extract hierarchical thermal features is employed for this step. Key features extracted from the CNN include Mean surface Temperature (of each ROI), Temperature gradients (spatial variations) and thermal hotspot/coolspot Regions: Thermal comfort is dynamic and evolves over time due to changes in passenger movements, posture, and cabin conditions. To capture these
654 Román and Knoch
Figure 3: View from the thermal camera array, showing an unprocessed image with one passenger seated in the rear left seat and another passenger occupying the left seat in the last row (self-created).Figure 4: Image from the thermal camera array, after preprocessing (self-created).Table 1. CNN architecture for thermal feature extraction.
Table available in the original PDF paper.
Layer Type Layer Details Input Size Output Size Purpose Input Layer Raw stitched thermal image 128 × 96 x 1 128 × 96 x 1 Receive the thermal image Conv1 7 × 7 Conv, stride 2, ReLU activation
128 × 96 x 1 64 × 48 x 64 Extract low-level spatial thermal features Pooling1 3 × 3 Max Pooling, stride 2
64 × 48 x 64 32 × 24 x 64 Reduce dimensions and retain key spatial features Residual Block 1
3 × 3 Conv + BatchNorm + ReLU x 2
32 × 24 x 64 32 x 24 × 256 Capture mid-level spatial features Residual Block 2
3 × 3 Conv + BatchNorm + ReLU x 3
32 × 24 x 256 16 × 12 x 512 Extract higher level localized patters Residual Block 3
3 × 3 Conv + BatchNorm + ReLU x 4
16 × 12 x 512 8 x 6 × 1024 Deep feature extraction for thermal regions Residual Block 4
3 × 3 Conv + BatchNorm + ReLU x 3 Spatial average across feature map
8 × 6 x 1024 4 x 3 × 2048 High level feature aggregation Global Average Pooling
4 × 3 x 2048 1 × 1 x 2048 Aggregate spatial features into a feature vector Fully Connected
Dense layer with 512 neurons + ReLU
1 × 1 x 2048 1 x 1 × 512 Reduce dimensionally, maintain key features Output Layer Fully connected, 3 neurons (Softmax)
1 × 1 x 512 1 x 1 × 3 Predict Thermal Sensation Vote
temporal dependencies, a Long Short-Term Memory (LSTM) network is integrated into the analysis pipeline. LSTM layers (see Table 2) analyze sequential thermal data to model how passenger-specific temperature distributions change over time. The sequence length is configured to capture changes in passenger conditions due to posture adjustments, exposure to airflow, or evolving environmental factors.
The input is composed of a 512-dimensional feature vector from the CNN at the time t, representing spatial thermal attributes. The sequential input is denoted as a sequence X = {x1, x2 . . . xt} where xt represents the feature
Wide-Angle Thermal Sensing for Personalized Climate Control 655
vectors extracted from the thermal image at the t-th time step. The order of operations described in Table 3 explains the flow of computational steps inside the LSTM cell at each time step t.
Table 2. LSTM parameter configuration.
Table available in the original PDF paper.
Layer Type Layer Details Input Size Output Size Purpose Input Layer Time sequenced feature vectors T x 512 T x 512 Accept input time series of feature vectors LSTM Layer 1 128 LSTM units, return sequences = True
T x 512 T x 128 Learn short-term temporal dependencies Dropout Layer
Dropout rate = 0.2 T x 128 T x 128 Regularize and reduce overfitting LSTM Layer 2 64 LSTM units, return sequences = False
T x 128 64 Capture long-term temporal dependencies Fully Connected
Dense layer with 32 neurons + ReLU
32 3 Predict (TSV)
64 32 Further reduce temporal feature dimensionality Output Layer Dense layer with Softmax activation
Table 3. Summary table of computational steps.
Table available in the original PDF paper.
Step Computation Purpose
Equation reproduced from the original PDF.
+ bf ) Scale down irrelevant parts of the previous cell state. Input gate it = σ(Wi ·
Equation reproduced from the original PDF.
+ bi) Determine importance of the current input for cell state updates. Candidate cell state
+ bC) Compute proposed updates to the cell state based on new input. Updated cell state
Ct = ft ⊙Ct−1 + it ⊙e Ct Combine retained past information with new updates. Output gate ot = σ(Wo ·
ht−1 , xt
+ bo) Determine the extent of the exposure of the cell state to the new hidden state. Hidden state update
ht = ot ⊙tanh(Ct) Update the hidden state, summarizing the temporal thermal patters relevant for predicting thermal comfort.
The final hidden state update ht is used for the TSV prediction, it firstly is passed through a fully connected layer:
Equation reproduced from the original PDF.
The transformed feature vector z, representing the passengers evolving thermal state is then converted into a probability distribution using Softmax activation:
Equation reproduced from the original PDF.
, i ∈{−4, −3, −2 −1, 0, + 1, + 2, + 3, + 4}
ˆy = argmaxp
yi
656 Román and Knoch
Where i corresponds to the thermal comfort categories. The LSTM outputs a temporal feature representation that complements the CNN-extracted spatial features. This combined representation provides a robust basis for predicting thermal comfort levels, capturing both the current thermal state and its evolution over time.
Thermal Comfort Prediction and HVAC Control Predicted TSV values guide the vehicle’s HVAC system (parameters such as Airflow rate (Q) and Vent Temperature (Tvent)) to ensure personalized comfort. These adjustments are performed dynamically and independently for each occupant.
RESULTS
This study evaluates the effective integration of an infrared camera array with the ResNet50 deep neural network and Long Short-Term Memory networks for optimizing thermal comfort in dynamic environments, such as shared and autonomous mobility. The thermal images, captured using the IR camera array from 5 individuals during winter, were processed by the ResNet50 model to classify thermal conditions into different categories.
The ResNet50 model, trained with 50 epochs, an initial learning rate of 0.0001, and a batch size of 16, achieved a good performance, demonstrating its ability to classify thermal zones effectively. The model’s capability to recognize key spatial features of thermal images without requiring additional sensor data highlights the efficiency of thermal image-based approaches for personalized thermal comfort optimization.
One key finding from initial trials is the system’s capacity to track multiple passengers and adapt to changes in occupancy. The camera’s 360-degree field of view allows for real-time adjustments based on occupancy patterns, even in vehicles with highly variable seating arrangements.
CONCLUSION
Beyond its primary application in vehicles, the camera solution proposed here has potential for other industries. The system could be adapted for use in other forms of public transportation, or even personalized indoor climate control systems in buildings. Furthermore, the system’s potential to address motion sickness, a common issue in autonomous vehicle environments, could further bolster its relevance in future transportation systems.
Although the initial results are promising, future research will be necessary to refine the system and optimize its performance across various environmental conditions and vehicle layouts. Further investigation will focus on expanding the dataset to include different weather conditions and a larger and more diverse number of participants. More extensive testing is required to fully evaluate its performance and to assess long-term energy savings. Newer machine learning models could further enhance the predictive capabilities of the system.
Wide-Angle Thermal Sensing for Personalized Climate Control 657
ACKNOWLEDGEMENTS
“This paper was created in the “Country 2 City - Bridge” project of the “German Center for Future Mobility”, which is funded by the German Federal Ministry for Digital and Transport.”
State-of-the-art thermal comfort models for car cabin Environment. Building and Environment, 262, 2. https://doi.org/10.1016/j.buildenv.2024.111825 Diga, D., Severin, I., & Ignat, N. D. (2021). Quality study on vehicle heat ventilation and air conditioning failure. Sustainability, 13(23), 1–3. https://doi.org/10.3390/ su132313441 Donsì, F., Naddeo, A., Cappetti, N., & Vink, P. (2022, October 18). Future
Vehicles: how to improve comfort and reduce discomfort. A comprehensive study of subjective and objective data. https://elea.unisa.it/handle/10556/7449 He, X., Zhang, X., Zhang, R., Liu, J., Huang, X., Pei, J., Cai, J., Guo, F., & Wang, Y. (2022). More intelligent and efficient thermal environment management: A hybrid model for occupant-centric thermal comfort monitoring in vehicle cabins. Building and Environment, 228, 109866. https://doi.org/10.1016/j.buil denv.2022.109866 Yang, B., Li, X., Hou, Y., Meier, A., Cheng, X., Choi, J., Wang, F., Wang, H.,
Wagner, A., Yan, D., Li, A., Olofsson, T., & Li, H. (2020). Non-invasive (non-contact) measurements of human thermal physiology signals and thermal comfort/discomfort poses -A review. Energy and Buildings, 224, 110261. https://doi.org/10.1016/j.enbuild.2020.110261 Yun, S., Chun, C., Kwak, J., Park, J., Kwon, C., Kim, S., & Seo, S. (2021).
Prediction of thermal comfort of female passengers in a vehicle based on an outdoor experiment. Energy and Buildings, 248, 111161. https://doi.org/10.1016/ j.enbuild.2021.111161 Zhang, H. (2003). Human thermal sensation and comfort in transient and nonuniform thermal environments. UC Berkeley: Center for the Built Environment. Retrieved from https://escholarship.org/uc/item/11m0n1wt Zheng, P., Liu, Y., Wu, H., & Wang, H. (2024). Non-invasive infrared thermography
technology for thermal comfort: A review. Building and Environment, 248, 111079. https://doi.org/10.1016/j.buildenv.2023.111079 Zhou, X., Miao, Z., Yuan, M., Li, K., Guo, H., Lin, X., Zeng, Y., Tu, R., & Zhong, J.
(2024). Dual-phase prediction model of passenger thermal sensation using facial thermal imaging and environmental factors. Case Studies in Thermal Engineering, 58, 104439. https://doi.org/10.1016/j.csite.2024.104439
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-36-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional