Jonas Birkle, Ruben Weber, Knut Möller, Verena Wagner-Hartl (2022). Psychophysiological Parameters for Emotion recognition – Conception and first evaluation of a measurement environment. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe100958
Emotions are of great importance for human interaction. The recognition of emotions is an essential basis for the social interaction of people, relevant at all levels of human interaction. However, the ability varies greatly among people, even to the point of a pathological deficit in autism (e.g., symptoms of "social blindness"). Furthermore, it is also an interesting topic in the current COVID-19 pandemic-situation with its restrictions regarding personal contacts and the resulting increase of digital contacts through online communication. For this reason, the measurement of emotions is of great interest for a wide variety of applications and, in particular, for product design or the development of trainings (e.g., for children with autism, leaders and managers, employees working in virtual teams etc.). The presented paper focuses on a concept and first evaluation of a measurement environment. Based on the objective emotional assessment (OEA) of Boucsein and his colleagues, a multidimensional approach was chosen for the developed concept. The studied psychophysiological measures were: cardiovascular (ECG) and electrodermal activity (EDA) as well as the forearm and neck electromyogram (EMG). In addition, the facial expression was recorded via camera. The results of a first evaluation (different tasks like a short game, a backward counting task, or thinking of different emotional contents) show, that the combination of the different measures was successful and promising for the use in further research. The study was able to show a need for improvement with regard to the derivation of the ECG (chest strap vs. adhesive electrodes). The design of the measuring environment can be seen as successful regarding the combination of the different psychophysiological measures and devices. In the future, an expansion with additional parameters such as pulse volume amplitude (PVA) or eye tracking will be considered.
Emotions are of great importance for human interaction (cf. Scholl, 2013, Ekman and Oster, 1979, Ekman, 1992, Erickson and Schulkin, 2003). The recognition of emotions is an essential basis for the social interaction of people, relevant at all levels of human interaction. However, the ability varies greatly among people, even to the point of a pathological deficit in autism (e.g., symptoms of "social blindness") (Sigman and Capps, 1997, Frith, 2001, Uljarevic and Hamilton, 2013). Furthermore, it is also an interesting topic in the current COVID-19 pandemic-situation with its restrictions regarding personal contacts and the resulting increase of digital contacts through online communication. Recent research results show first associations between social isolation caused by the COVID-19 pandemic and emotional and social cognitive abilities (Bland et al. 2020, Bland et al. 2021). In addition, Schlegel et al. (2021) showed that participants with higher emotion recognition ability reported less feelings of burden and less negative affect during the first two weeks of public life restrictions caused by the COVID-19 pandemic (March/April 2020).
The measurement of emotions is of great interest for a wide variety of applications. One example is product design and product development (Lee et al. 2009, Shaw, 2007, Hirschman & Holbrook, 1982), there a product must be a sensual and emotional experience to be perceived as special and unique. In addition to that, emotions can also have an influence on purchase decisions. Another example can be the development of trainings e.g., for children with autism (Kandalaft et al. 2013, Yuan & Ip, 2018), leaders and managers (Köppe et al. 2019, Riggio and Reichard, 2008) or for employees working in virtual teams (Holtz et al. 2020).
The presented paper focuses on a concept and first evaluation of a measurement environment. Therefore, the research questions which had to be answered were: Is it possible to combine the different psychophysiological measures to investigate emotions that occur during different tasks in one measurement environment? Which technical aspects have to be considered? Is it possible to synchronize the different psychophysiological measures in an adequate way? Can a setup be found that can be used for future studies?
Based on the objective emotional assessment (OEA) of Boucsein and his colleagues (Boucsein and Schaefer, 2008, Boucsein et al. 2002), a multidimensional approach was chosen for the developed concept. A multidimensional approach is useful
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 because a single measure or even a single psychophysiological system cannot account for all the changes that may be triggered by different emotions (Boucsein, 1989, Boucsein and Backs, 2000). Psychophysiological measures represent a possibility to make emotional experience objectively measurable, for example during product usage or when performing different tasks (Boucsein, 2006, Wagner, 2014, Wagner and Kallus, 2014; 2015, Mandryk and Atkins, 2007, Yagi, 2000). Another advantage of this multidimensional approach is that psychophysiological responses are, under normal circumstances, not volitional controllable by individuals (Boucsein and Backs, 2009) and can be measured using noninvasive techniques (Boucsein, 2006). The studied psychophysiological measures were: cardiovascular (ECG) and electrodermal activity (EDA) as well as the forearm and neck electromyography (EMG). In addition, the facial expression was recorded via camera. The integration of the facial expression is a further development of the previously described approach. During former research of the research group (Project EmTra, grant from the Institute of Applied Research of Furtwangen University), a system for emotional recognition via facial expression was developed (Arabian et al. 2021a, 2021b; Arabian et al. 2021a, 2021b), which should be integrated in the measurement environment. In the future it should be enhanced and used in further studies.
PREPARATION OF THE MEASUREMENT ENVIRONMENT Psychophysiological Measures
As described in the introduction section, cardiovascular activity (ECG), electrodermal activity (EDA), and electromyography (EMG) should be combined in the measurement environment. In this case, ECG and EDA data were collected using the EcgMove 3 and EdaMove 3 of movisens (2022), the EMG data using the TEA CAPTIV T-Sens EMG of TEA Ergo Inc (2022).
The ECG was recorded using disposable adhesive electrodes which were attached directly at each side of the sensor. The standard sensor position (chest) recommended by movisens was used (movisens, 2022). Electrodermal activity (EDA) was measured using reusable, non-polarizing sintered Ag/AgCl electrodes with a diameter of 10 mm. The electrodes were attached at the participants' non-dominant hand (thenar and hypothenar). To obtain muscle activity, adhesive electrodes were used and attached at the upper trapezius (musculus trapezius pars descendens) and forearm muscle (musculus flexor carpi radialis).
Facial Expression To recognize the facial expression of the participants, a webcam (Logitech C925e) to
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 record participants’ faces was included in the developed measurement environment. The webcam was placed on the top of a computer screen (23-inch HP-screen; distance to the participant: 50 cm).
Synchronization To combine all psychophysiological measures in one measurement environment, different computers are needed for data recording because the different manufacturers use varying recording methods. Running different recording programs on one device was found to increase the probability of interference between the programs and can cause worse latencies than running every recording program on a dedicated computer. The synchronization-difficulties that can occur using multiple computers, are a well- known problem. The approach to cope with this within the developed measurement environment was to include an additional trigger program as synchronization system.
The trigger program used in this setting was developed in the context of a bachelor thesis project to synchronize measurements between functional near infrared spectroscopy and motion tracking (Birkle, 2021, Claassen, 2021, Picka, 2021). It was updated on a regular basis to gain better latencies and to simplify the usage for the developed measurement environment. The tool utilizes the internet protocol (IP) and sends packages from one transmitter to all the recording devices via the transmission control protocol (TCP). This communication procedure was chosen to guarantee a high compatibility between the different devices. The graphical user interface (GUI) of the trigger tool for both, the transmitter and the receiver view, is presented in Figure 1. It is recommended to use a separate wired local network disconnected from the frequently used network infrastructure to reduce the present network traffic. On the receiver side the trigger program performs the trigger by connecting to the interface of the recording program. If the recording program doesn’t have an interface, it saves the necessary trigger information including the corresponding system times into a text file. The round-trip delay never exceeded 50 milliseconds. This value consists of the transmission time from the transmitter to the receiver, the required time to perform the trigger on the receiver device, and the backward transmission time from the receiver back to the transmitter.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Figure 1. Graphical user interface (GUI) of the trigger tool (left side: transmitter view; right side: receiver view).
It is crucial that the trigger arrives earlier at the recording device than the psychophysiological reactions on the stimulus do. Otherwise, the correct recognition of the physiological reaction to the stimulus during the data analysis is not given. Latencies in this range don’t interfere with the reaction times of the human body to a presented stimulus for the used physiological measurements. E.g., the reaction time of the skin conductance varies from one to two seconds, when a stimulus is presented (Boucsein, 2012).
There are two ways to perform a trigger with the help of the developed trigger tool. On one hand, one can select a trigger button with the mouse (c.f. Figure 1, left side) which sends a manual trigger signal to all devices. This procedure was used in the presented measurement environment. On the other hand, there is the possibility to call the trigger function via the python interface from any other program or directly from the command line. Therefore, it is very easy to include the trigger functionality into nearly every digital stimulus presentation tool.
FIRST EVALUATION OF THE MEASUREMENT ENVIRONMENT
Method
Sample. Due to the restrictions regarding personal contacts during the time when the measurement environment was developed (spring 2021, COVID-19 pandemic), only one participant took part in the first evaluation of the measurement environment. The participant was a 22 year old male student.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Study Design, Materials and Procedure. A repeated measurement design was chosen for the evaluation of the measurement environment. After a baseline measurement (overall 5 minutes – mean two minutes were used for the analyses), the participant had to perform five different tasks (each with a duration of two minutes). Each task was followed by a rest measurement of two minutes. The different performed tasks (measurement repetition factor) were: A short game (knife hit from Ketchapp played via app on an iPhone 8), a backward counting task (count backwards from 100 in steps of six and seven), and thinking of different emotional contents (e.g. a quiet place – forest glade; a situation the participant faced in the last days, that was emotionally arousing). The measured psychophysiological responses were cardiovascular (ECG) and electrodermal activity (EDA) as well as the forearm and neck electromyography (EMG). In addition, the facial expressions were recorded via camera (see section “preparation of the measurement environment” for more details). The experimental setup is shown in Figure 2.
Figure 2. Experimental setup
Data Analyses. Cardiovascular and electrodermal activity were analyzed using the software DataAnalyzer from movisens (2022), electromyography was analyzed using the software package CAPTIV from TEA Ergo Inc (2022). The analyzed parameters for cardiovascular activity were heart rate (HR) in beats per minute and heart rate variability (HRV RMSSD). Regarding the electrodermal activity, skin conductance level (SCL), amplitude of non-specific electrodermal responses (NS.SCR amp), frequency of non-specific electrodermal responses (NS.SCR freq) and mean sum amplitude (NS.SCR amp/NS.SCR freq) were used. For EMG the root mean square amplitude (RMS) was included in the analyses.
Results
For the cardiovascular activity (ECG) the results showed that due to a large number of artifacts, data loss occurred for both parameters HR and HRV. The analysis of the protocol and post-tests with the sensor revealed, that one explanation for this can be that the adhesive electrodes attached directly at the sensor and positioned at the chest
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 together with the sensor, were very sensitive to artifacts caused by motion. Regarding the electrodermal activity (EDA) the results suggest no data loss and a good quality of the measurement (Figure 3). In addition, electromyography was measured without data loss in the developed experimental setting. The results suggest that within the used measurement environment electromyography (EMG) of the musculus trapezius seems to be more sensitive regarding motion-related artifacts than the EMG of the forearm muscle. Furthermore, no inherence caused by the different sensors and devices and/or the measurement environment were visible in the different analyzed psychophysiological measures. Additionally, the mimic was recorded without any data loss or motion blur.
Figure 3. Example for the measured signals of the electrodermal activity
DISCUSSION
The results of a first evaluation (different tasks like a short game, a backward counting task, or thinking of different emotional contents) show, that the synchronization of the different measures was successful and promising for the use in further research. The study was able to show a need for improvement with regard to the derivation of the ECG (chest strap vs. adhesive electrodes). The design of the measuring environment can be seen as successful regarding the combination of the different psychophysiological measures and the different used devices. In the future, an expansion with additional parameters such as pulse volume amplitude (PVA) or eye tracking will be considered.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Furthermore, in future study settings the trigger tool should always be used in an automatic way by implementing it into the stimulus presentation system as described in chapter “Synchronization” (second alternative). With this automatic implementation the impact of human response times can be reduced and human errors can be avoided.
However, the first evaluation of the measurement environment has some limitations. Due to the restrictions regarding personal contacts during the time when the measurement environment was developed caused by the COVID-19 pandemic. So, only one participant was allowed to participate in the first evaluation of the measurement environment. We are aware of the fact that this can be only a first step. Unfortunately, this was the only possibility to receive first data to assess the measurement environment at that time (spring 2021). Therefore, another validation study was conducted when the restrictions were eased and the development of a study with more participants was possible. These results will be presented in the paper of Schmid, Braunmiller, Hansen, Schonert, Möller and Wagner-Hartl (2022; also within this volume).
ACKNOWLEDGMENTS AND AUTHOR’S STATEMENT Partial support by a grant from the Institute of Applied Research (EmTra-Tandem) is gratefully acknowledged. The authors state no conflict of interest. Informed consent has been provided from the participant.
REFERENCES
Arabian, H., Wagner-Hartl, V., and Möller, K. (2021a), “Facial emotion recognition based on localized region segmentation.” In AUTOMED – Automation in Medical Engineering 2021. 2pp. Arabian, H., Wagner-Hartl, V., and Möller. K. (2021b), “Traditional versus Neural
Network Classification Methods for Facial Emotion Recognition.” Biomed. Eng.-Biomed. Tech., 66(s1), 168-175. Arabian, H., Wagner-Hartl, V., Geoffrey Chase, J., and Möller. K. (2021a), “Image Pre-processing Significance on Regions of Impact in a Trained Network for Facial Emotion Recognition.” IFAC PapersOnLine, 54-15, 299-303. Arabian, H., Wagner-Hartl, V., Geoffrey Chase, J., and Möller. K. (2021b), “Facial
Emotion Recognition Focused on Descriptive Region Segmentation.” IEEE EMBC /IEEE Xplore 2021. , 3415–3418. Birkle, J. (2021), “Analyse eines Doppelaufgabenparadigma mit fNIRS: Kognitive Belastung und Autofahren [Analysis of a dual task with fNIRS: Cognitive workload and car driving].” Unpublished bachelor thesis, Furtwangen University.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Bland, A.R., Roiser, J.P., Mehta, M.A., Sahakian, B.J., Robbins, T.W., and Elliott, R. (2020), “COVID-19 induced social isolation; implications for understanding social cognition in mental health.” Psychological Medicine, First View, 1-2. Bland, A.R., Roiser, J.P., Mehta, M.A., Sahakian, B.J., Robbins, T.W., and Elliott,
R. (2021), “The impact of COVID-19 social isolation on aspects of emotional and social cognition.” Cognition and Emotion, Ahead-of-print, 1-10. Boucsein, W. (1989), „Die elektrodermale Aktivität als Beanspruchungsindikator in der Arbeitspsychologie.“ In W. Rohmert, and H.G. Wenzel (eds.). Aspekte der Leistungsfähigkeit. Different aspects of performance. Frankfurt am Main: Verlag Peter Lang, 171-179. Boucsein, W. (2006), „Psychophysiologische Methoden in der
Ingenieurspsychologie.“ In B. Zimolong, and U. Konradt (eds.). Sonderdruck aus Enzyklopädie der Psychologie: Themenbereich D Praxisgebiete: Serie III Wirtschafts-, Organisations- und Arbeitspsychologie. Band 2: Ingenieurs- psychologie. Göttingen: Hogrefe, 317-358. Boucsein, W. (2012), “Methods of Electrodermal Recording.” In W. Boucsein (ed.).
Electrodermal Activity, Boston: Springer, 87-258. Boucsein, W., and Backs, R.W. (2000), “Engineering Psychophysiology as a Discipline: Historical and Theoretical Aspects.” In R.W. Backs, and W. Boucsein (eds.). Engineering Psychophysiology. Issues and Applications, Mahwah, New Jersey: Lawrence Erlbaum Associates, 3-30. Boucsein, W., and Backs, R.W. (2009), “The Psychophysiology of Emotion,
Arousal, and Personality: Methods and Models.” In V.G. Duffy (ed.). Handbook of Digital Human Modeling. Research for Applied Ergonomics and Human Factors Engineering. Boca Raton: CRC Press, 35-1 – 35-18. Boucsein, W., Schaefer, F., Kefel, M., Busch, P., and Eisfeld, W. (2002), “Objective emotional assessment of tactile hair properties and their modulation by different product worlds.” International Journal of Cosmetic Science, 24(3), 135-150. Boucsein, W., and Schaefer, F. (2008), “Objective Emotional Assessment of
Industrial Products.” In J.H.D.M. Westerink et al. (eds.). Probing Experience. From Assessment of User Emotions and Behaviour to Development of Products. Dordrecht: Springer, 69-76. Claassen, O. (2021), “Analyse eines Doppelaufgabenparadigma mit fNIRS: Kognitive Belastung und Gehen [Analysis of a dual task with fNIRS: Cognitive workload and effects on gait]” Unpublished bachelor thesis, Furtwangen University. Ekman, P., and Oster, H. (1979), “Facial Expressions of Emotion.” Annual Review
of Psychology, 30, 527-554. Ekman, P. (1992) “An Argument for Basic Emotions.” Cognition and Emotion, 6(3/4), 169-200. Erickson, K., and Schulkin, J. (2003), “Facial expressions of emotion: A cognitive
neuroscience perspective.” Brain and Cognition. 52, 52-60. Frith, U. (2001), “Mind Blindness and the Brain in Autism.” Neuron, 32(6), 969-
979. Hirschman, E.C., and Holbrook, M.B. (1982), “Hedonic Consumption: Emerging Concepts, Methods and Propositions.” Journal of Marketing, Summer, 92-101.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Holtz, K., Castella, V.O., Abad, A.Z., and González-Anta, B. (2020), “Virtual Team Functioning: Modeling the Affective and Cognitive Effects of an Emotional Management Intervention.” Group Dynamics: Theory, Research, and Practice, 24(3), 153-167. Kandalaft, M.R., Didehbani, N., Krawczyk, D.C., Allen, T.T., and Chapman, S.B.
(2013), “Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism.” Journal of Autism and Developmental Disorders, 43, 34-44. Köppe, C., Held, M.J., and Schütz, A. (2019), “Improving Emotion Perception and Emotion Regulation Through a Web-Based Emotional Intelligence Training (WEIT) Program for Future Leaders.” International Journal of Emotional Education, 11(2), 17-32. Lee, L., Amir, O., and Ariely, D. (2009), “In search of Homo economicus: cognitive
noise and the role of emotion in preference consistency.” Journal of Consumer Research, 36, 173-187. Mandryk, R.L., and Atkins, M.S. (2007), “A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies.” International Journal of Human-Computer Studies, 65, 329-347. Movisens (2022), Movisens. Available at: https://www.movisens.com/de/ (Accessed
21 January 2022). Picka, E.F. (2021), “Analyse eines Doppelaufgabenparadigma mit fNIRS: Auditivkognitive Belastung und Fahrradfahren [Analysis of a dual task using fNIRS: auditory-cognitive load while riding a bike].” Unpublished bachelor thesis, Furtwangen University. Riggio, R.E., and Reichard, R.J. (2008), “The emotional and social intelligences of
effective leadership.” Journal of Managerial Psychology, 23(2), 169-185. Schlegel, K., von Gugelberg, H.M., Makowski, L.M., Gubler, D.A., and Troche, S.J. (2021), “Emotion Recognition Ability as a Predictor of Well-Being During the COVID-19 Pandemic.” Social Psychological and Personality Science, 12(7), 1380-1391. Schmid, R., Braunmiller, L., Hansen, L., Schonert, C., Möller, K., and Wagner-
Hartl, V. (under review), “Emotion recognition – Validation of a measurement environment based on psychophysiological parameters.” Proceedings of the 5th International Conference on Intelligent Human Systems Integration: Integrating People and Intelligent Systems (IHSI 2022), Taylor & Francis. Scholl, W. (2013), “The socio-emotional basis of human interaction and communication: How we construct our social world.” Social Science Information, 52(1), 3-33. Shaw, C. (2007), The DNA of Customer Experience. How Emotions Drive Value.
New York: palgrave macmillan. Sigman, M., and Capps, L. (1997), Children with Autism. A Developmental Perspective. Cambridge, MA: Harvard University Press. TEA Ergo Inc (2022), TEA. Available at: https://www.teaergo.com/ (Accessed 21
January 2022). Uljarevic, M., and Hamilton, A. (2013), “Recognition of Emotions in Autism: A Formal Meta-Analysis.” Journal of Autism and Developmental Disorders, 43, 1517-1526.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Wagner, V. (2014), “Hochwertigkeit von Geräuschen im Fahrzeuginnenraum.“ In P.
Sachse, and E. Ulich (eds.). Beiträge zur Arbeitspsychologie. Lengerich: Pabst Science Publishers. Wagner, V., and Kallus, K.W. (2014), “Use of a multidimensional approach in the automotive product development: Quality of turn indicator sounds.” In D. de Waard et al. (eds.). Proceedings of the HFES Europe Chapter 2013 Annual Conference. 169-181. Wagner, V., and Kallus, K.W. (2015), “Sound Quality of Turn Indicator Sounds –
Use of a Multidimensional Approach in the Automotive Product Development.” Journal of Traffic and Transportation Engineering, 3(3), 158- 165. Yagi, A. (2000), “Engineering Psychophysiology in Japan.” In R.W. Backs, and W.
Boucsein (eds.). Engineering Psychophysiology. Issues and Applications. Mahwah, New Jersey: Lawrence Erlbaum Associates, 361-368. Yuan, S.N.V., and Ip, H.H.S. (2018), “Using virtual reality to train emotional and social skills in children with autism spectrum disorder.” London Journal of Primary Care, 10(4), 110-112.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional