Christian Knecht, Ravi Kanth Kosuru (2024). Protecting the First Responders: Improving FR situational awareness through multi-modal interfaces leveraging the ubiquitous personal smartphone. In: Safety Management and Human Factors. AHFE Open Access, vol. 151, pp. 78–87. https://doi.org/10.54941/ahfe1005304
First responders (FRs) in disaster scenarios often lack access to real-time, context-specific information, which is crucial for maintaining situational awareness. To address this issue, we introduce PathoVIEW, an innovative interface ecosystem built around a smartphone app, that leverages the ubiquitous availability of such devices for everyone and especially the localization feature for delivering context-aware alerts. PathoVIEW provides vital and relevant information local to the First Responder in a vast disaster scenario. It also includes the use of audible feedback and other technologies that could be coupled with the phone over Bluetooth such as smartwatches or a vest with vibration motors for warning the FR about situations in their proximity. The system's feedback mechanisms are designed to be adaptive, responding to the FRs' roles, their distance from potential hazards, and the nature and severity of the threats. Thus, providing a novel way to deliver information specific to the situation and location of the FR. PathoVIEW relies on a centralized common information exchange platform that addresses the challenges of sharing data across different agencies, which can be restricted by various regulations and the diversity of tools in use. By leveraging FIWARE Smart Data Models, PathoVIEW is able to function independently of the Incident Management Systems (IMS) that organizations may employ, promoting compatibility and ease of integration. In addition to active stakeholder involvement in the design and development process to ensure a relevant solution, the system was evaluated in large-scale simulated exercises in four European countries. The FR used and interacted with PathoVIEW in close conjunction with their current systems and processes. The feedback from the FR indicated that PathoVIEW was beneficial for enhancing communication, coordination, and situational awareness. The ease of use and the seamless integration in their existing context-of-use validated the user acceptance and the FR expressed a high interest in utilising the PathoVIEW system in actual disaster response situations.
Protecting the First Responders: Improving FR Situational Awareness 79
INTRODUCTION
First Responders (FR) are the first line of assistance for the public in a disas- ter scenario. Increasing situational awareness through in-situ information is required for potential dangers that are not perceptible to FR visually (Chan et al., 2016). This is increasingly seen in disaster situations that lead to water related emergencies, such as contamination of water sources after floods or infrastructure damage after earthquakes (Bakratsas et al., 2022). This could also be compounded due to poor visibility in bad lighting conditions such as nighttime activities. Emergency Incident Management Systems (IMS) are extensively being used for task delegation and mission control with limited communication feasibility for each FR on the field (Sarshar et al., 2015). Mul- tiple agencies are involved in the various aspects of disaster response who employ specific tools and there is a challenge in cross-agency information exchange (Duro, 2016). In addition, depending on the command and policy structures, FR teams in various countries include volunteers who might not have access to all equipment, especially if it’s not crucial to the mission at hand (Fuchs-Kittowski et al., 2017).
PathoVIEW introduces applications for mobile and wearable user inter- faces that can be used by FRs on the field to enhance FR’s sensing and situ- ational awareness capabilities, operations, and information sharing between all personnel. This is to be accomplished by providing FR with any necessary information relevant to the situation through a smartphone or smartwatch using multi-modal feedback such as vibrations or auditive alerts. It was developed as part of the EU Horizon 2020 funded project Pathogen Con- tamination Emergency Response Technologies (PathoCERT). The goal was to create technologies and processes to reliably detect pathogens in water and initiate actions to mitigate any threats by interconnecting all relevant information.
Related Work
Emergency Incident Management Systems are a critical tool for coordinat- ing and handling disasters and have seen constant development and progress to keep pace with technology and requirements (Molino, 2006). Computer aided dispatch is an important component that deals with the delegation and management of First Responders and teams in such situations (Shaghaghi et al., 2019).
Cloud based technologies have vastly contributed to scalability and large- scale deployment of software solutions that can aid different FR needs and agencies. A cloud-based Mobile Computing Applications Platform MCAP (Chung et al., 2013) outlines the use of such a platform and lays the ground- work for creating the technical architecture. It also highlights the strengths of establishing an ecosystem to ensure close connections and involvement among First Responders, suppliers, and application developers.
ENSURE (Fuchs-Kittowski et al., 2017) takes advantage of mobile tech- nologies to effectively engage and coordinate volunteers in disaster situa- tions. The system ensures that volunteers can spontaneously register and get
80 Knecht and Kosuru
involved in various emergency activities. The advantage of using such inter- connected systems along with the widespread availability of smartphones has led to the development of many alerting and coordination apps. Quite a few of these apps are developed to ensure they are tightly integrated into the ICT systems of the different FR organisations. There are however con- cerns relating to the need to buy such apps separately or usability in general (Von Der Linde et al., 2022).
The various IMS and alerting apps provide an efficient way to ensure infor- mation flow and communication not just within the agency but also across agencies thus aiding the situational awareness of the responders on the field (Steen-Tveit and Radianti, 2019).
Worker safety is an active area of research where multimodal feedback systems are being explored to improve situational awareness. This is espe- cially relevant in hazardous working environments where the attention of the users might be occupied by different tasks. Vibration or tactile feedback through smart vests or other wearable devices can be used to warn users of collisions with heavy construction equipment (Sakhakarmi et al., 2021) or drawing attention to relevant or unanticipated events (Sklar and Sarter, 1999). The smart e-vest (Hines et al., 2015) goes on to combine motors for haptic feedback with other modalities, LEDs for visual feedback and speakers for auditory feedback. The main driver of such systems is the gaming indus- try for increasing immersion in games1,2 or VR simulations for trainings for police, firefighters3, or the military4. These systems provide an ideal platform to use them for prototyping and testing across other applications.
The increased adoption of mobile devices and wearables brings with it significant cybersecurity, privacy and resilience concerns that must be consid- ered during device selection and system development (Franklin et al., 2020). For instance, there could be availability issues associated with heavy reliance on services such as Single Sign-On (SSO), as well as risks related to the Bring Your Own Device (BYOD) practice, where devices with unpatched operating systems or application vulnerabilities may compromise sensitive information. Additionally, the potential for device failure due to insufficient ruggedization cannot be overlooked.
DEVELOPMENT AND EVALUATION METHODOLOGY In spite of the extensive related work, there is still a major gap in integrating IMS with technologies that focus on FR safety. This along with the need to expedite the step from research and development to active deployment shows a need for a methodology that integrates relevant users from the onset as well as possible on-field evaluations for accessibility in conjunction with current and other technologies in use. We followed the user-centred design process
Protecting the First Responders: Improving FR Situational Awareness 81
described in ISO 9241–210 (International Organization for Standardiza- tion, 2019) to iteratively develop and improve PathoVIEW. FR stakeholders from various European countries were integrated into all phases of the design process via workshops, both online and in-person, facilitated by the unique opportunity provided by the PathoCERT project through so-called communities of practice (CoPs) (Wenger-Trayner and Wenger-Trayner, 2015).
We collected information about the teams, roles, equipment, and software, as well as communication methods of the FR to understand their context of use. Personas and a usage scenario were created based on this information to gain insights into the stakeholders and information flows related to FR and command post. User and functional requirements were gathered using the MoSCoW method (Waters, 2009).
Storyboards were developed to visualize how PathoVIEW could be used in disaster situations, and low-fidelity prototypes were created for different UIs and target devices. Feedback was gathered through CoP meetings, where par- ticipants provided insights and priorities for required information. The need for multi-modal feedback in various working environments was identified, with a focus on minimizing distractions for FR. Different components and features were iteratively conceived and developed with feedback and input from the FR interactions.
The final prototype was then evaluated through four pilot case studies, simulating realistic scenarios such as a contamination event resulting from the mixture of wastewater and drinking water after an earthquake, or the need to evacuate a flooded area with potential water contamination due to heavy rainfall and cascading effects. FR received training before indepen- dently using the technologies in large-scale on field exercises that included multiple stakeholders, organisations, and co-ordination across multiple loca- tions of interest. Each pilot case in turn led to adjustments and improvements for the following on-field evaluation.
PATHOVIEW SYSTEM The final result of the multiple design and development cycles resulted in the integrated PathoVIEW system that includes a set of apps and functionalities for integrating different wearable devices. The core of it is a comprehensive web app for cross platform usage and an app that can be installed on any Android smartphone or tablet. The Android app can connect with a smart- watch for notifications on the wrist, as well as a haptic vest for on body vibrations as alerting system via Bluetooth. Vibrations and sounds provided by the mobile or the connected devices can communicate important infor- mation without distracting FR from the task at hand, especially in the visual field.
PathoVIEW allows users to store and access relevant data mostly leverag- ing FIWARE Smart Data Models5 through the PathoWARE platform devel- oped as a part of the project. It incorporates the Open-Source Identity and
5https://www.fiware.org/smart-data-models/
82 Knecht and Kosuru
Access Management system, Keycloak6 for authentication and authorization. The app provides real-time updates and notifications using Server-Sent Events as Publish-Subscribe mechanism. The UI development followed a mobile-first approach since the main target devices are smartphones on the field. It uses Leaflet7 for interactive maps with the free Open Street Map tiles8 and Open Weather Map9 for displaying current weather. For Android devices, the app is enhanced using the cross-platform native runtime Capacitor10, which pro- vides access to native hardware and software features like camera, native notifications (e.g. vibration and sound on success or error) and background location-tracking. We used a framework for background location-tracking that is also power efficient to ensure it can work longer without draining battery11. It leverages the motion-detection interfaces (using accelerometer, gyroscope, and magnetometer) to detect when the device is moving or sta- tionary to ensure optimum performance. A variable distance filter, such as updating only after every 10 meters, enables adaptive tracking tailored to the specific requirements of different roles and situations. The Android applica- tion facilitates integration with the bHaptics vest through a plugin12, which synchronizes the list of paired devices and their Bluetooth connection statuses with the app. It also enables the registration and activation of ‘tact files’. Tact files are tactile patterns that specify the activation parameters for the vest’s motors - such as location (front or back, left or right), intensity, dura- tion, and sequence - thereby allowing the app to deliver varied information through distinct vibration patterns. These patterns can be further customized or expanded to meet the FR needs according to the scenarios encountered.
Dashboard PathoVIEW provides a dashboard with details about the emergency event. It also displays important times like time to sunset or time since the emer- gency occurred. In the “news” area, the last alert, mission, and comment are displayed. Users can also see the geo-coordinates of their current position. Weather data can help FR to better assess the current situation and plan the according actions. Additionally, the users and roles involved in the current emergency event are shown (see Figure 1 on the left).
Alerts Alerts are the main functionality of PathoVIEW for making FR aware of dangerous situations. Alerts are raised automatically based on specific sensor measurements. PathoVIEW additionally offers a user-friendly form for man- ual input, enabling commanders at a mobile command centre, or any FR with the appropriate permissions, to share alerts with the centralized platform. The FIWARE Smart Data Models ensure a standardised exchange system for other apps or IMS systems that implement the same. PathoVIEW ensures that any new alert raised complies with the Smart Data Model. One of the most important properties is the ability to define the location either as a point, or complex polygons which is crucial for delivering in-situ information.
Context-Aware Alerts Context-aware alerts are an important functionality of PathoVIEW to warn FR in critical situations which extensively depend on the location. The con- text service receives all alerts that are related to a selected emergency event and either checks whether the user is near or is inside an alert location peri- odically (web) or on location change (Android). It uses the GPS location if available otherwise it tries to narrow the position down with the help of the IP address and other data. There are different warning levels, depending on the distance to the dangerous area, the alert severity and category/sub- category. The warning level affects the visual representation of the alert, the alert sound, the used haptic pattern (e.g. the activation of the vest motors), as well as the intensity and the number of consecutive repetitions of the vibrations.
84 Knecht and Kosuru
Figure 2: Context-aware alert notifications (left = visual information in app, right = pat- tern representation for haptic vest).
Missions Similar to IMS platforms, PathoVIEW provides a basic mission management system. Missions can be assigned to certain roles. Additionally, missions can be used by FRs to provide updates on the current situation with texts and pictures from the field. The goal here is to provide especially a cross agency light-weight multi-way communication platform to support FRs without direct access or need for an IMS.
Map Alerts, missions, and FR device positions can also be displayed directly on a map. The different model types can be filtered. Each model type uses a dedicated icon and colours for better differentiation. On clicking an item, a popover with information appears (see Figure 1 on the right).
Instructions and Pathogen Information In addition to the main functionalities PathoVIEW also provides interactive instructions with texts and pictures for setting up different devices that allow the detection of microbiological contamination as well as an encyclopae- dia for different pathogens including description, transmission, symptoms, complications, diagnosis, treatment, prevention, and control measures.
EVALUATION RESULTS AND DISCUSSION PathoVIEW was tested with actual First Responders in simulated field pilots in four European countries. These exercises covered various possible scenar- ios that occur in a real emergency and the FR incorporated PathoVIEW into their equipment arsenal. Due to the scale of the simulations which covered multiple agencies and multiple technologies in play, the evaluations cannot be compared to classical in lab studies with detailed feedback. However, they provided a realistic insight into the use of the technology by users with
Protecting the First Responders: Improving FR Situational Awareness 85
minimum training input as well as richer feedback due to the integration in the context of use. The overall high level quality feedback was generally positive (see Table 1) with the participants also providing input for further enhancements. Some general comments – “It is useful for the First Respon- ders for giving instructions, coordination and cooperation.”; “...vest is ideal as a warning system in darkness”. The users noted that the system does not consider altitude when calculating warning distances. They also suggested that it should alert users if, for example, if the vest has lost its connection or is low on battery during a mission.
Table 1. PathoVIEW FR high level quality feedback from four pilots on a scale from 1-5 (very negative, negative, undecided, positive, very positive).
Table available in the original PDF paper.
N Average STD Min Ease of Use 36 4,44 0,64 3 Usefulness for role 31 4,19 0,93 2 Usefulness for communication and coordination 35 4,40 0,73 3 Likeliness of use 34 4,21 0,93 2
The recurring theme from the very onset of the user requirement gathering stage was the need to ensure intuitive use and seamless integration in the con- text of use. Any new technology should provide a tangible benefit without a major affect on the current routines and established patterns of work. The use of contextual alerts only relevant to the task and current location along with feedback modalities such as the vibrating vest contributed to the positive reac- tion to the PathoVIEW system. The ability to provide relevant information timely but unobtrusively to the FR thus improves their situational awareness and contributing to their safety. This focus is otherwise overlooked at times in evaluation of similar systems so far which was noted in the FR feedback.
The pilots took place over a period of 10 months with an opportunity for iterative improvements to align with user expectations. However, there is fur- ther scope for optimizing its functionality to improve performance, usability, and robustness.
CONCLUSION AND FUTURE WORK Providing relevant information timely to the First Responders is crucial for improving their situational awareness. The use of digital tools and technolo- gies goes beyond just informing the users about the missions and what they need to do but contributes highly to their own safety and wellbeing. Although mobile apps and Incident Management Systems have seen rapid development and proliferation through the years, there are multiple facets to First Respon- der needs due to the diverse organisational structures and procedures that vary across countries.
This paper presents PathoVIEW, a First Responder interface ecosystem that allows users access relevant information and warnings over multiple modal- ities for a holistic view of the situation. The multi-modal approach enhances
86 Knecht and Kosuru
the situational awareness of First Responders in hazardous conditions, allow- ing them to focus on their immediate tasks while being warned of potential dangers that may not be immediately visible. The system was evaluated on the field in simulation exercises to show the need and suitability of such systems. Although the system was rated very positively, significant work is still required to bridge the gap for daily use. On one hand, server-side avail- ability and scalability must be guaranteed, for instance, through redundancy while on the other hand it is also essential that First Responders maintain access to the system even in the event of cell phone network failure, such as when infrastructure is damaged during a natural disaster. For example, this may necessitate the use of wireless, battery-operated gateways. Devices like vibration vests or smartwatches, being commercial products, may require replacement or enhancement with rugged systems designed to endure harsh usage conditions. This includes ensuring all devices are waterproof and have sufficient battery capacity for extended operations. However, rapid advance- ments in both hardware and software are increasingly making them easily and affordably accessible to organizations worldwide. In any case, first respon- ders’ safety is crucial and should be incorporated into the development of FR apps for smartphones.
ACKNOWLEDGMENT
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 883484.
REFERENCES
Bakratsas, M., Moumtzidou, A., Gialampoukidis, I., Lioumbas, I., Iliadis, D., Spyrou, D., Christodoulou, C., Katsiapi, M., Vourvachis, I., Vrochidis, S., Kompatsiaris, I. (2022). Assisting First Responders in Water-borne Hazards in Northern Greece Through PathoCERT Solutions. Zenodo. doi: 10.5281/zen- odo.7304218. Chan, E., Wang, Y., Seyed, T., Maurer, F. (2016). ERWear: Wearable System Design
through the Lens of First Responders, in: Proceedings of the 2016 ACM on Interactive Surfaces and Spaces - ISS ‘16, ACM Press, Niagara Falls, Canada, pp. 489–492. Chung, C., Egan, D., Jain, A., Caruso, N., Misner, C., Wallace, R. (2013). A Cloud- Based Mobile Computing Applications Platform for First Responders, in: 2013 IEEE Seventh International Symposium on Service-Oriented System Engineering. pp. 503–508. Duro, R. (2016). Results from the Survey on User Requirements in the domain of Crisis and Disaster Management. Retrieved April 04, 2021, from https://c2-sense.eu/index.php/news/results-from-the-survey-on-user-requir ements-in-the-domain-of-crisis-and-disaster-management/. Franklin, J., Howell, G., Ledgerwood, S., Griffith, J. (2020). Security Analysis of
First Responder Mobile and Wearable Devices. National Institute of Standards and Technology. Fuchs-Kittowski, F., Jendreck, M., Meissen, U., Rösler, M., Lukau, E., Pfennigschmidt, S., Hardt, M. (2017). ENSURE - Integration of Volunteers in Disaster Management, in: Environmental Software Systems. Computer Science for Environmental Protection, IFIP Advances in Information and Communication Technology. Springer International Publishing, Cham, pp. 247–262.
Protecting the First Responders: Improving FR Situational Awareness 87
Hines, K., Lages, W., Somasundaram, N., Martin, T. (2015). Protecting workers with smart e-vest, in: Proceedings of the 2015 ACM International Joint Confer- ence on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers - UbiComp ‘15. Osaka, Japan. pp. 101–104. International Organization for Standardization (2019). ISO 9241-210:2019 Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems. Molino Sr., L. N. M. (2006). Emergency Incident Management Systems: Fundamen-
tals and Applications. John Wiley & Sons. Sakhakarmi, S., Park, J., Singh, A. (2021). Tactile-based wearable system for improved hazard perception of worker and equipment collision. Automation in Construction. Volume 125. Sarshar, P., Nunavath, V., Radianti, J. (2015). On the Usability of Smartphone
Apps in Emergencies: An HCI Analysis of GDACSmobile and SmartRescue Apps, Human-Computer Interaction, Part II, HCII 2015, LNCS 9170. Springer International Publishing Switzerland. pp. 765–774. Shaghaghi, N., Patel, S., Pabari, B., Francis, M. (2019). Implementing Communications and Information Dissemination Technologies for First Responders, in: 2019 IEEE Global Humanitarian Technology Conference (GHTC). pp. 1–4. Sklar, A. E., Sarter, N. B. (1999). Good Vibrations: Tactile Feedback in Support
of Attention Allocation and Human-Automation Coordination in Event-Driven Domains. Hum. Factors 41. pp. 543–552. Steen-Tveit, K., Radianti, J. (2019). Analysis of Common Operational Picture and Situational Awareness during Multiple Emergency Response Scenarios. Von Der Linde, M., Nospickel, P. M., Göcke, M., Thielsch, M. T. (2022). Emer-
gency Alerting Through Smartphone Apps: Functions, Usage and Usability Issues from the First Responder’s Point of View, in: Mensch Und Computer 2022. ACM, Darmstadt Germany. pp. 442–446. Waters, K. (2009). Prioritization using moscow. Agile Plan. Volume 12. Wenger-Trayner, E., Wenger-Trayner, B. (2015). Introduction to communities of practice: a brief overview of the concept and its uses. Available from authors at https://www.wenger-trayner.com/introduction-to-communities-of-practice.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Safety Management and Human Factors
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-27-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional