Minjung Lee, Sungwon Byon (2023). Uncertainty-Aware AI-Facilitated Decision Support System for Emergency Call Takers. In: Emerging Technologies and Future of Work. AHFE Open Access, vol. 117, pp. 49–56. https://doi.org/10.54941/ahfe1004404
This paper presents an artificial intelligence (AI) based decision support system to assist call takers sitting behind emergency numbers, such as 1-1-2 or 9-1-1. Accurate situational awareness of emergency incidents during emergency report intake is crucial for the initial response. However, the caller’s utterances in an emergency situation can be uncertain, therefore the AI model can often assist call recipients with false predictions, delaying the appropriate initial response to emergencies. To address this issue, we propose using an uncertainty-aware AI model in the decision support system. The proposed system displays a set of emergency situational candidates predicted based on the text transcribed in real-time as the voice report intake, supporting the call takers in effectively receiving the emergency calls. The type and number of emergency situational candidates are determined considering the uncertainty inferred from an uncertainty-aware AI model. We provide a detailed explanation of the proposed system and evaluate its performance using actual domestic 1-1-2 police emergency report data.
In recent times, there has been a growing trend in both the private and public sectors to leverage various technologies through the integrated appli- cation of Information and Communication Technology (ICT) and Artificial Intelligence (AI). The deployment of these technologies across public sys- tems, including safety, defense, education, law enforcement, and culture, can potentially lead to numerous innovations. These innovations include the enhancement of public service efficiency and cost minimization. In this context, the present study introduces an AI-facilitated system specifically designed to assist call takers operating behind emergency numbers, such as 1-1-2 or 9-1-1(Baek et al., 2021 & Blomberg et al., 2021). Emergency call centers play a pivotal role in the overall emergency response process. The initial interaction between the caller and the call taker often determines the effectiveness of the subsequent response. During this interaction, the call taker must quickly and accurately identify the type of
emergency based on the information provided by the caller. Accurate situa- tional awareness during the emergency report intake is crucial for an effective initial response. However, recognizing the type of emergency, while simulta- neously asking a series of questions to the caller and recording information to be relayed to first responders (e.g., police officers, firefighters, and ambu- lance crews), is a challenging task. Consequently, there has been a growing demand for AI models to alleviate the workload of call takers in emergency report intake systems.
A representative example of this technology is the speech recognition AI model that automatically transcribes voice reports into text. In various countries, including the United States and South Korea, speech recognition technology for emergency calls has reached considerable maturity and has been implemented in actual intake systems. This technology help to prevent the waste of crucial initial response time, often lost to repeated questions when call takers struggle to understand the caller’s voice. Additionally, it enables the collection of a large volume of spoken text data in emergency situations, disasters, and crime-related fields. The text data, alongside with images, is an area where recent AI models have made significant advance- ments, paving the way for the development of various AI-facilitated systems. This study also explores a text classification AI model that predicts the type of emergency based on the real-time transcribed voice of the caller and the call taker. Such a model could be instrumental in developing a system that supports the real-time decision-making process of call takers.
Despite the convenience of the AI-facilitated emergency call-taking system mentioned above, there are concerns about the direct integration of AI mod- els into systems that are crucial to public safety. Callers are non-experts in emergency reporting, may not know what information is vital for the call taker. Furthermore, in emergency situations, callers can be flustered and agitated, driven by urgency and fear, which may result in unclear and inaccu- rate speech. This raises a critical question: Can we trust the results obtained from the AI model when it processes such uncertain and potentially unreli- able utterance data? If the AI model’s unreliable predictions lead to incorrect emergency responses, it could pose serious threats to citizen safety. To mit- igate these risks, we have incorporated an uncertainty-aware AI model into the emergency call-taking system. The proposed system is designed to display a broader range of emergency situational candidates in cases of high uncer- tainty, and a more refined set when the uncertainty is low, thereby minimizing the risk of unreliable predictions.
CRIME SITUATIONAL RECOGNITION IN EMERGENCY CALL INTAKE In Korea, there are two primary emergency call numbers: 1-1-2 and 1-1-9. The number 1-1-2 is designated for crime-related emergencies. According to the 1-1-2 manual of the Korean National Police Agency, Korean police officers utilize the 1-1-2 system, which incorporates advanced IT technol- ogy and the police communication network, for handling cases reported through this emergency number. When receiving calls, the police officer in charge of reception is responsible for leading the conversation with the caller.
Uncertainty-Aware AI-Facilitated Decision Support System for Emergency Call Takers 51
This involves asking a series of questions to ascertain critical information such as the caller’s location, perpetrator details, the urgency of the situation, and the type of crime. This information is then recorded to the 1-1-2 system by typing, enabling police officers dispatched to the scene to access and verify it through the system.
To enhance crime situational awareness, the receptionist should categorize each report into one of the 58 crime categories, such as murder, robbery, violence, fraud, and traffic accidents, and enter this information into the 1-1-2 system. However, relying solely on the receptionist’s subjective judg- ment for situational awareness can be problematic, as it may be influenced by their competence and condition. Incorrectly perceived crime situations at the reception stage can adversely affect subsequent stages (such as dispatch, com- mand, and response), leading to wasted time and police resources. To address this issue, an AI model has been developed to automatically recognize crime situations based on the report summaries entered by the receptionist into the 1-1-2 system, as illustrated in Table 1. However, this existing model, which determines the crime situation based on the receptionist’s summarized report, has limitations in early detection and may be biased due to the subjectivity of the receptionist.
This study develops an AI model that recognizes crime situational candi- dates based on the dialogue between the receptionist and the caller, which is automatically transcribed by a voice recognition AI model. By employing an AI model capable of quantifying uncertainty, this system can display a range of possible crime situations to the receptionist, depending on the level of uncertainty. This prompts the receptionist to ask more detailed questions to the caller about the situation, thereby enhancing the accuracy and efficiency of the emergency response process.
Table 1. Differences between the existing and the proposed AI model.
Table available in the original PDF paper.
Feature Existing AI model Proposed AI model Recognition Timing After entering report During the emergency call Input Report summary Transcribed dialogue Uncertainty Awareness X O
DISSONANCE-AWARE NEURAL NETWORKS Deep neural networks have significantly advanced the predictive accuracy in text classification tasks (Balkus & Yan., 2022, Sun et al., 2019). How- ever, traditional deep neural networks often face issues of high-confidence misclassifications without considering the inherent uncertainty of specific task (Gall & Ghahramani. 2016). To address this, recent studies have conducted to provide approaches in deep neural networks that not only measure predicted values but also quantify the associated uncertainty, aim- ing to circumvent the issue of misclassification (Kendall & Gal 2017, Chen et al., 2020).
In our decision support system, we focus on ‘dissonance (Josang et al., 2018)’, a type of uncertainty obtainable from evidential neural networks
52 Lee and Byon
(ENNs) (Sensory et al., 2018, Hu & Khan., 2021). We hypothesize that the information made by callers describing an event may contain words and sen- tences that could be confused with various emergency situations. Therefore, our proposed model is designed to measure dissonance, relates to topic mis- match (conflicting evidence). Consequently, we have named our proposed model Dissonance-aware neural networks (DANNs).
The output of a standard neural network classifier is a probability assign- ment over possible classes, but DANNs assume a Dirichlet distribution for the probabilities of categories (opinions), treating the model’s predictions as subjective opinions and learning a function to gather evidence that elicits such opinions form the data. A multinomial opinion in given xdialogue and xextra is represented by ωY =
bY, uY, aY
where a domain is Y ≡{1, . . . , K} , a random variable Y takes value in Y, K = |Y| ≥2 and ωY is given as P
y y aY
y∈Y bY(y) + uY = 1. bY denotes belief mass function over Y. uY denotes uncertainty mass representing vacuity of evidence. aY represents base rate distribution over Y, with P
= 1. Then the projected probability distribution of a multinomial opinion is given by:
Equations and mathematical formulas are available in the original paper PDF.
Multinomial probability density over a domain of cardinality K is repre- sented by the K-dimensional Dirichlet PDF where the special case with K = 2 is the Beta PDF as a binomial opinion. It denotes a domain of K mutually disjoint elements in Y and αY the strength vector over y ∈Y and PY the probability distribution over Y.
Y
Dir
pY; αY
pY(y)(αY(y)−1), (2)
where B (αY) is a multivariate beta function as the normalizing constant, αY ≥0, and pY(y)̸ = 0 if αY < 1.
Given a sample with the input xdialoguei and xextrai and the ground-truth label yi, let f(xdialoguei, xextrai|θ) represents the predicted evidence vector pre- dicted by the classifier with parameters θ. Then the corresponding Dirichlet distribution has parameters αi = f(xdialoguei, xextrai |θ) + 1. The Dirichlet density Dir
is the prior on the Multinomial distribution Multi
pi; α
yi; pi
. Then the following sum of squared loss is optimized for training:
Z yi −pi
L
p(αij−1)
, yi
ij dpi
xdialoguei, xextrai
θ
2 B (αi)
+E
y2
p2
pij
Uncertainty-Aware AI-Facilitated Decision Support System for Emergency Call Takers 53
Dissonance is determined to be high when the Dirichlet distribution param- eter values assigned to multiple categories are large, indicating data with conflicting evidence. In the emergency call reception support system, the can- didate set to be presented to the call taker is determined using the predicted probability vector and topic dissonance obtained from the trained model. Given a multinomial opinion with non-zero belief masses, the measure of dissonance is: !
bY(yi) P
X
yj∈Y\yi bY(yj)Bal(yj, yi) P
yj∈Y\yi bY(yj) yi∈Y
where the relative mass balance between a pair of belief masses bY(yj) and bY(yi) is expressed by:
Figure 1 illustrates the structure of DANNs. To effectively extract fea- tures from the multi-turn dialogues between the call taker and the caller, we employed a pre-trained model specifically designed for the Korean lan- guage, named ‘roberta-ko-small’ (https://huggingface.co/lassl/roberta-ko-sm all), guided by the LASSL (Language framework for Self-Supervised Learn- ing) (https://github.com/lassl/lassl). For training, multi-turn dialogues were augmented by sampling dialogue turns of arbitrary lengths. Additionally, we incorporated incident-specific information, such as the reporting time (rang- ing from 1 to 24 hours). This was used for the model’s training as a one-hot encoded vector.
EXPERIMENTS AND RESULTS The dataset used for training the model consists of 17,218 reports from real cases recorded in the Korean police’s 112 system. This dataset encom- passes a wide range of crime categories, including 10 major crimes (such as homicide, robbery, sexual assault), 15 miscellaneous crimes (such as violence, fraud, threats, gambling), 8 public order-related crimes (such as quarrels, disturbances), 6 traffic-related crimes (such as traffic accidents, traf- fic inconveniences), 9 other police-related tasks (such as inquiries, suicide) and 10 other agency-related crimes (fire, noise). Consequently, the proposed deep neural network model is designed to classifies 58 types of emergency situations.
To evaluate out models, we consider Accuracy and macro F1 score. Accu- racy is a straightforward metric that quantifies the proportion of correctly predicted instances out of the total instances. The F1 score is the harmonic mean of precision and recall, where precision is the ratio of correctly pre- dicted positive observations to the total predicted positives, and recall (or sensitivity) is the ratio of correctly predicted positive observations to all the observations in the actual class.
Table 2 presents the experimental results, which compare different combi- nations related to the type of input text data, the inclusion of uncertainty estimation, the use of additional information, and the presence of multi- turn conversation augmentation. The proposed model demonstrated the best performance with a classification accuracy of 83.04% and an F1 score of 77.96%.
Table 2. Experimental results.
Table available in the original PDF paper.
(Proposed)
Input text Report summary
Transcribed dialogue
Transcribed dialogue
Transcribed dialogue
Transcribed dialogue Uncertainty X X O O O Extra information X X X O O Data augmentation X X X X O Accuracy 63.13 65.40 71.92 80.42 83.04 F1 score 60.77 63.70 67.98 73.97 77.96
Uncertainty-Aware AI-Facilitated Decision Support System for Emergency Call Takers 55
CONCLUSION
This paper introduces an artificial intelligence-based system designed to assist emergency call recipients. The system displays a set of potential emergency situation candidates, which are determined based on the dialogue transcribed in real-time between the caller and the receiver through a speech recognition model during an emergency call. A key feature of the proposed classifica- tion model is its ability to estimate uncertainty about its predictions. When the dialogue contains a mix of various emergency situation contents, the model estimates high uncertainty. In response to this, the system presents a broader set of candidate to the receiver. This approach is expected to prompt the receiver to ask more specific questions to the caller, thereby obtaining information that can help narrow down the set of candidates to the correct emergency situation. This method aims to address the issue of incorrect initial responses caused by the AI model’s misjudgement of the emergency type. The proposed system is anticipated to enhance the work efficiency of reception- ists and enable faster, more accurate emergency responses. In the future, this system has the potential to be utilized as a tool in various fields that require multi-turn dialogue-based situation (or topic) recognition.
ACKNOWLEDGMENT
This research was supported and funded by the Korean National Police Agency. [Project Name: 112 Emergency Dispatch Decision Support System / Project Number: PR08-03-000-21].
REFERENCES
Baek, M. S., Park, W., Park, J., Jang, K. H., & Lee, Y. T. (2021). Smart policing technique with crime type and risk score prediction based on machine learning for early awareness of risk situation. IEEE Access, 9, 131906–131915. Balkus, S. V., & Yan, D. (2022). Improving short text classification with augmented
data using GPT-3. Natural Language Engineering, 1–30. Blomberg, S. N., Christensen, H. C., Lippert, F., Ersbøll, A. K., Torp-Petersen, C., Sayre, M. R.,... & Folke, F. (2021). Effect of machine learning on dispatcher recog- nition of out-of-hospital cardiac arrest during calls to emergency medical services: a randomized clinical trial. JAMA network open, 4(1), e2032320–e2032320. Chen, W., Zhang, B., & Lu, M. (2020). Uncertainty quantification for multilabel
text classification. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(6), e1384. Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050–1059). PMLR. Hu, Y., & Khan, L. (2021, August). Uncertainty-aware reliable text classification.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 628–636). Josang, A., Cho, J. H., & Chen, F. (2018, July). Uncertainty characteristics of subjective opinions. In 2018 21st International Conference on Information Fusion (FUSION) (pp. 1998–2005). IEEE. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in bayesian deep learn-
ing for computer vision?. Advances in neural information processing systems, 30.
56 Lee and Byon
Sensoy, M., Kaplan, L., & Kandemir, M. (2018). Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31. Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to fine-tune bert for text
classification?. In Chinese Computational Linguistics: 18th China National Con- ference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18 (pp. 194–206). Springer International Publishing.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Emerging Technologies and Future of Work
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-93-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional