Justin Cheung, Maegan Schmitz, Thomas Strybel, Panadda Marayong, Vernol Battiste, Stacey Ahuja, Kim-Phuong L. Vu, Praveen Shankar (2024). Development of Android-based Messaging Application for Onboard Communication for UAM Simulation. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 159, pp. 1554–1564. https://doi.org/10.54941/ahfe1005723
The Urban Air Mobility (UAM) concept utilizes vertical take-off and landing (VTOL) aircraft for on-demand passenger and cargo air transportation services in metropolitan areas. Various designs of UAM vehicles have been developed, but research is needed to achieve their integration with the existing national airspace system to ensure flight safety and efficiency. The concept of simplified vehicle operations (SVO) can be applied to UAM to allow for novice pilots to operate the vehicle more easily and reduce the operation complexity and demands on trained personnel for mission management. Voice-based pilot-ATC communications have been shown as a major source of workload, which will also likely impact UAM pilots. Text-based communication could potentially reduce pilot’s workload and voice communication errors. In this work, we report on the development of a messaging application for pilot and vertiport (VP) manager/air traffic controller (ATC) communications. The system consisted of an Android tablet installed with a custom application for pilots and a Windows-based application for VP manager/ATC. The text-messaging application was compared to voice communications in a user study involving pilots taking off and landing at six vertiports around San Francisco Bay area while communicating with an ATC/VP managers to acknowledge their flight plans and flight plan changes. Participants were interviewed following the study to gain insights on their perceptions of the UAM operations tested and the messaging application. The results showed that the communication mode did not affect workload, but pilots reported higher situation awareness with voice communication. A qualitative thematic analysis was conducted on the debriefing data, which provided recommendations on the design refinement. Based on the user feedback, an automated feature was added to the messaging application to facilitate the transition between vertiports. The revised interface will be evaluated in a future user study.
Development of Android-Based Messaging Application 1555
INTRODUCTION
Global economic growth and urbanization is expected to increase the demand for efficient local transportation. The Urban Air Mobility (UAM) program, conceptualized as a future transportation system of low-altitude air vehicles for on-demand transportation of people and goods, is expected to address this need. Many novel designs of low-altitude vertical take- off and landing (VTOL) aircrafts have been developed. This proliferation of vehicle designs and display interfaces are significant barrier to the safe and efficient implementation of the UAM system with the current national air transportation system. Recent UAM conceptualizations rely mostly on autonomous systems; however, the near-term operations will likely employ onboard or remote pilots flying under current-day visual or instrument flight rules (VFR or IFR). Considerable work is required to determine the impact of various urban airspace concepts, air traffic management procedures and new roles and responsibilities on traffic operations in urban class “B, C, D, and G” airspace. In recent years, investigations of simplified vehicle operations based on NASA’s EZ-Fly Concept have been reported for vehicles that are easier to operate which lessens the complexity and demands on pilots and trained operators for mission management (Lombaert et al., 2020; Wing et al., 2020).
One area of concern for onboard, piloted UAM vehicles is communications. Pilots flying in urban areas will be required to communicate with vertiport (VP) managers when approaching and departing from each vertiport. Additional communications may be required for communications with air traffic controllers (ATC) when the vehicle enters different classes of airspace, and when traffic managers issue flight plan changes designed to maintain separation from other UAM vehicles, and fixed-wing aircraft in the area. In current day aircraft, voice-based pilot-ATC communications have been shown to be a major source of workload; it is likely that this would impact UAM pilots as well. Text-based communications is expected to reduce pilot workload, readback errors and frequency congestion (Hah et al., 2011). In today’s NAS most passenger and cargo flights typically require two onboard pilots, but UAM vehicles will contain a single pilot, and therefore simplified vehicle operations should include improvements to pilot communication interfaces.
Our research group has developed a simulation of a VTOL aircraft operating in the San Francisco Bay Area using a CAVE virtual reality (VR) system to provide a flexible testbed for the evaluation of potential UAM operational concepts and human-automation teaming configurations (Marayong et al., 2020; Shankar et al., 2022; Strybel et al., 2022). In this paper, we report on the development of a messaging application for pilot – VP manager/ATC communications and its design refinement based on a user study with pilots. The messaging application simplifies the pilot’s tasks, given the constraints of flying in dense urban areas, to ensure that workload and situation awareness are within acceptable limits. We examined the effectiveness of text messaging as compared to the traditional voice communication for UAM operation. The messaging application was designed for a user study involving pilots taking off and landing at six (6) VPs
around San Francisco Bay area that required communicating with an ATC to acknowledge their flight plans and clearances, and to obtain permission from VP managers for landing and departure. The pilots flew two routes around the bay area. As shown in Figure 1, the clockwise (CW) route started east-bound from the Ferry Terminal (Ferry) to Oakland International Airport (OAK), Hayward (HWD), San Francisco International Airport (SFO), Daly City, and Presidio. The counter-clockwise (CCW) route started west-bound from the Ferry Terminal toward Presidio and stops at the same VPs in the reverse order. Pilots were required to communicate with the VP manager/ATC at different locations along the route marked by communication markers. The map overlay includes the flight path (blue line), vertiport locations (marked as V), route markers (grey circles labelled with R), communication markers (grey circles labelled with C),and combination route/communication markers (grey circles labelled RC). Route markers indicate locations where the pilot must make a turn and/or change freeways at an interchange. The red outlines around the vertiports are only shown for illustrative purpose in Figure 1 to highlight their locations. The following sections describe the messaging applications, the user study, results, and design modifications.
Figure 1: Map used in the messaging application which depicts the clockwise flight route showing the flight path, vertiport locations (V), and route (R) and communication (C) markers.
SYSTEM OVERVIEW The CAVE system consists of four panel displays with an approximate 3D volume of 12.5’W x 8’D x 9’H. It is integrated with motion capture and control software for generation of immersive VR environment with real-time body tracking. We have developed a VR simulation of a VTOL
Development of Android-Based Messaging Application 1557
aircraft operating in the San Francisco Bay Area. The vehicle can be operated autonomously or manually with a single control joystick. Multiple user interfaces, such as a virtual cockpit with minimap and vehicle status displays, and a tactile alerting system, have been added for evaluation of different system configuration and operational concepts (Haneji et al., 2023). The following sections describe the components and functionalities of the new design of the messaging applications. The messaging apps were used in a simulation with pilots (Strybel et al., 2024).
Messaging Applications The communication system consisted of two messaging applications, one for the pilot and one for the ATC/VP manager. The pilot app was employed on a Samsung A8 tablet as a custom messaging application. The tablet was attached to the pilot’s thigh using an adjustable strap, similar to a knee board used by pilots in small aircraft. The other application designed for ATC and VP manager usage was Windows-based and was employed on a standard desktop computer. This ATC/VP manager application provided a simple, efficient interface for experimenters to interact with the pilots, and was not intended to be representative of operational ATC or VP manger interfaces. Both messaging applications were developed in Unity3D (Unity Software Inc., San Francisco, CA) with the Photon Engine API (Exit Games Inc., Portland, OR). Figure 2 provides the layout of the pilot’s and the ATC’s user interfaces. The six vertiports were listed on both interfaces. The sections below describe the graphical user interface (GUI) and the functionality of the applications.
Figure 2: User interfaces on the messaging applications (Left) pilot’s interface (Right) ATC’s interface.
Pilot’s GUI The Pilot GUI is essentially a Chatroom that consisted of a textbox for pre- generated messages, a chat box for live messages, vertiport buttons, related map- reference buttons, and message-reply to buttons. The top right corner of the GUI included a home menu used for returning to the home page. This function was used to change routes during the experiment. It allows the
scenarios to be easily changed between test flights without needing to restart the program. The home menu button was partially occluded on the pilot’s app to prevent participants from clicking on it accidentally during flight.
Selecting the vertiport buttons simulated changing the frequency channel of the Chatroom in order to communicate with the next VP manager or ATC. Each vertiport included a list of pre-generated messages that the pilot could send during take-off and landing. Once a Chatroom was selected, the program loaded the list of pre-generated messages in the textbox. The pilot could select a message using touch input, and then would click on the “Send” button to send the selected message to the ATC or VP manager. The pilot could respond to ATC/VP messages by clicking on “Roger”, “Wilco”, or “Reject”, which would transmit the message to the ATC/VP manager. Live messages between the pilot and ATC were displayed on the chat box located to the left of the screen. Each vertiport’s chatroom included their own set of canned messages that were to be sent and received from their respective VP manager. When the pilot received a message from ATC and the application was not currently on the correct ATC channel, the pilot received a sound notification and the vertiport button was highlighted.
Next to each vertiport button was a map reference button. Selecting the button displayed a static map of the current flight leg to the respective vertiport. The arrow button on the bottom left of the screen displayed the map image of the entire route with all vertiports.
ATC-VP Manger GUI The GUI for the ATC was similar to the Pilot application but without the map buttons and the pre-set message (“Roger,” “Wilco,” and “Reject”) buttons. Note again that this application was not intended to simulate actual ATC or VP manger interfaces. Instead, it simply provided an efficient method for sending and receiving messages to/from the participant pilot and the experimenter playing the role of the ATC/VP manager. The ATC could change the route between the west-bound (CCW) and the east-bound (CW) direction by pressing a button located on the top left corner. The text on the button displayed the current route: “CW” or “CCW”.
The vertiport buttons located on the left of the screen changed the active vertiport frequency. Canned messages were stored in the box at the center of the screen based on the currently active vertiport. Alongside pre-generated messages, there was also a text box at the bottom of the screen where the ATC/VP manger could input custom messages. After selecting and inputting their message, the ATC would click on the “Send”button to send their custom message. The live messages were displayed in the chat box located at the top of the screen. The chat box displayed all messages sent and received from the pilot application regardless of the current vertiport chatroom that the messages were sent from.
Development of Android-Based Messaging Application 1559
USER STUDY Experimental Protocols
Six certified pilots participated in the study. Two were Certified Flight Instructors II, three were instrument rated pilots, and all reported 200–3500 hours of flight time (M = 981 hrs, SD = 1288 hrs). Upon arrival at the facility, participants were briefed on UAM, received instruction on the flight routes, flight parameters, trained on communication protocols, and walked through the messaging application. Participants then completed 15 minutes of practice flying the simulated vehicle along a designated path while practicing communication procedures using the messaging app. After their training, participants piloted the simulated UAM aircraft along the path depicted in Figure 1, through the San Francisco metropolitan area, stopping at 6 vertiports. Participants flew two routes, one CW and one CCW, using voice or text communication. The flight path and vertiport locations were the same for CW and CCW routes, though communication points differed due to the airspace structure, which were illustrated on the minimap with the “C’, “R”, and “RC” markers. This protocol resulted in a total of four flights per participant, with conditions counterbalanced across participants. For additional details on the simulation and results see Strybel et al. (2024).
Figure 3: Sample text communication view on (Left) pilot’s interface (Right) VP manager/ATC’s interface.
Participants received flight communications from ATC/VP manager through the messaging application in both voice and text communication modes. The communication points and the content of communication statements were consistent across the communication modes. Communications from the participant were required at vertiport departure, approach, landing, when transponder changes (Squawks) occurred, and specific reporting points (see Figure 1). Examples include, “Ferry Vertiport, BeachCave23, ready for departure” and “OAK Vertiport, BeachCave23, at 880 abeam Lake Merritt, inbound for landing with Alpha”. In the text communication mode, these messages were shown on the pilot messaging application that were selected and sent via touch input. Figure 3 shows a
sample of the messaging application during text communication between the pilot and the VP manager at Presidio. The pre-programed messages for the pilot and the VP manager were loaded in the textbox (shown in the middle) of the respective user interface. The sent messages appeared with a timestamp in the live chat boxes. In the voice communication mode, the canned messages were read and spoken by the participant via Zoom (Zoom Video Communications, Inc.). ATC responded to participant communications either by text or voice. Examples of ATC responses include, “BeachCave23, SFO Vertiport, cleared for clockwise departure, depart west to 101, winds 260 at 9, Squawk 3203 while in Class Bravo” and “BeachCave23, Daly City Vertiport, cleared to land Vertiport slot 2, winds 090 at 8”. In the text communication mode, these messages were pre-programmed. In the voice communication mode, these messages were read aloud by ATC.
ATC also issued airspeed and altitude change directives. Examples of ATC directives include, “Descend and maintain 550 feet” and “Increase airspeed 10 knots”. In the text communication mode, these directives were manually typed into a textbox and sent using a “Send” button. In the voice communication mode, these directives were issued aloud by ATC. Finally, in response to ATC directives, participants responded with “Wilco”, “Roger”, or “Reject”. In the text communication mode, participants responded to ATC directives using the “Wilco”, “Roger”, and “Reject” buttons which automatically sent that response. In the voice communication mode, these responses were read aloud by the participant.
Following the completion of all experimental trials, each participant was interviewed using a semi-structured format. The goal of the interview was to gain insight regarding pilot perceptions of UAM operations, UAM barriers, the simulation test’s fidelity, communication modes, and the messaging application’s ease of use.
Data Analysis Performance data, subjective workload, and situation awareness were obtained and analysed. For a complete description of these results, see Strybel et al. (2024). Two findings of note here, subjective workload (NASA TLX) was unaffected by the mode of communication. However, situation awareness (SART) was higher in the voice compared with text communication modes. A qualitative thematic analysis was used to organize the interview data. To achieve this, two independent coders reviewed the interview transcripts and organized statements into thematic categories. The coders then deliberated, settling on themes and operationalizing which statements fit into each theme. One theme that emerged was regarding the Messaging Application. The theme included statements regarding the messaging application’s user interface, user experience, level of automation, and communication mode.
Development of Android-Based Messaging Application 1561
Pilot Feedback Results
Table 1 provides a summary of participants’ feedback and recommendations by theme regarding different aspects of the messaging application. Examples of specific comments from the participants are shown in Table 2.
Pre-programmed text messages could cause complacency
Implement geolocation that automatically switches chat room
Loss of skill from automation
Equation reproduced from the original PDF.
Display map and chat rooms simultaneously
Use real charts and maps instead of illustrated maps
Too many windows when toggling between home page, navigational maps, and chat rooms
Interface was disorganized
Map button organization was not intuitive
Have a single chat room which houses all text communications from all VP Managers User experience (n = 5)
Make the vertiport destination (icon/name) as the main access point which houses all relevant communications and maps
Switching between windows (home page, navigational map, chat room) resulted in missed ATC directives, lose track of which chat room should be active, and fatigue
Provide text notification alert when user is viewing navigational map
Too many ‘clicks’ required to navigate messaging application
Maps were helpful, but challenging to use due to workload
Allow users to type custom communications
Equation reproduced from the original PDF.
Voice was more natural as that was the communication mode in which pilots are trained
Voice communication was realistic
Provide alternatives when typing custom communications is not ideal
Text was easier to miss than verbal communication
Text was more efficient
Text allowed for simultaneous conversations
Text made it easier to anticipate what was going to happen
Table 2. Example comments from pilot participants regarding the messaging application.
Table available in the original PDF paper.
Comments Participant Number 1 “...the text-based system was good because I didn’t have to write anything down. The information was always on screen. I can scroll up and just look, “Oh what was the frequency I just took down?” but at the same time I can see automation making pilots complacent.” 5 “...having one window versus a popup window, because the interface could be done with one page…The less clicks the better.” 6 “If you got a notification when you were on the map mode, that would have been nice. It also would have been nice if there was a window where all the communications were, and like you just changed who you’re talking to. You could see everything you said in the past that might be a continuous log.”
Figure 4: The revised design of the pilot’s messaging application based on the user study.
Design Modifications As a result of the pilots’ feedback from the user study, we have implemented two design changes to the pilot messaging application. First, when a message is sent to the pilot, the application will automatically change the interface to the active vertiport channel. Second, the vertiport frequency will automatically change when the vehicle enters a new vertiport control area accompanied by a popup message for a few seconds before disappearing to notify the pilot of the change (see Figure 4). Additionally, messages in the chat box will be highlighted if the pilot has not responded to it. Automatic mode changes may result in missed information and other errors that may be unacceptable to pilots. Thus, the modified pilot messaging application
Development of Android-Based Messaging Application 1563
will be evaluated in an upcoming simulation of UAM vehicles flying in an extended urban area (e.g., Ferry VP to San Jose Airport and return) containing additional air traffic.
CONCLUSIONS
Considerable work is required to evaluate urban airspace concepts and air traffic management procedures for a safe and effective integration of UAM in the national airspace system. The initial implementations of UAM will most likely involve onboard pilots, yet possible shortages of certified VFR/IFR pilots is a known barrier. Therefore, we are testing novel cockpit applications to reduce the amount of training required for future UAM operators to ensure that workload and situation awareness are within acceptable limits. Use of a messaging application is one example of simplifying the pilot’s communication tasks, given the constraints of flying in dense urban areas. In this work, we describe the development of a messaging application that allows text-based communication between pilot and the ATC/VP manager. Our study provided a proof-of-concept for feasible UAM operations, with pilots performing take-offs and landings at six vertiports around San Francisco Bay area. Our study also provides an example of the communications that will be required between ATC/VP managers to acknowledge UAM pilot flight plans using a messaging application as compared to voice communication. Pilot reactions to the messaging application were favourable, but they noted several limitations of the current design. Based on the feedback received, a few design modifications were made to automate the transition between the vertiports to facilitate the operation. The revised design will be tested in another user study to evaluate its effectiveness.
ACKNOWLEDGMENT
This research was supported in part by NSF Major Research Instrumentation award (Award #1626655) and San Jose State Research Foundation Grant #2116145736 (PI: Sean Laraway). We also thank Andrew Alfaroarevalo for his help with the data analysis.
REFERENCES
Haneji, K., Leung, K., Tran, A., Cheung, J., Deabaapilux, W., Marayong, P., Vu, K., Shankar, P., Strybel, T. Z., & Battiste, V. (2023). Use of Tactile Alerts in Urban Air Mobility Vehicles. In: Gesa Praetorius, Charlott Sellberg and Riccardo Patriarca (eds) Human Factors in Transportation. AHFE (2023) International Conference. AHFE Open Access, vol. 95. AHFE International, USA. https://doi.org/10.54941/ ahfe1003836 Lombaerts, T., Kaneshige, J., & Feary, M. (2020, June 15–19). Control concepts for
simplified vehicle operations of a quadrotor eVTOL vehicle. In AIAA AVIATION 2020 FORUM, virtual event. Marayong, P., Shankar, P., Wei, J., Nguyen, H., Strybel, T., & Battiste, V. (2020).
Urban Air Mobility system testbed using CAVE virtual reality environment, IEEE Aerospace Conference (virtual).
Shankar, P., Marayong, P., Strybel, T., Battiste, V., Nguyen, H., Cheung, J., Viramontes, J. (2022). Urban Air Mobility: Design of a Virtual Reality Testbed and Experiments for Human Factors Evaluation. ASME International Mechanical Engineering Congress and Exposition, Columbus, OH. Strybel, Z., Battiste, V., Marayong, P., Shankar, P., Viramontes, J., Nguyen, H., &
Cheung, J., (2022, October 10–14). Preliminary validation of a virtual UAM vehicle and simplified cockpit interface. Abstract. Human Factors & Ergonomics Society 66th International Annual Meeting, Atlanta, GA, United States. Strybel, Z., Battiste, V., Vu, K., Marayong, P., Ahuja, S., Schmitz, M., Cheung, J., Culver, C., Alfaroarevalo A., & Shankar, P. (2024). Evaluation of voice vs. text communication modes in simulated UAM operations. AHFE International Conference on Human Factors in Design, Engineering, and Computing. In press. Willems and Shulz: Hah, S., Willems, B., and Schulz, K. (2011) What happens
when data communications fail? - A simulation study. Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting, 51–65. Wing, D. J., Chancey, E. T., Politowicz, M. S., & Ballin, M. G., (2020). Achieving resilient in-flight performance for advanced air mobility through simplified vehicle operations. AIAA, 1–17. https://doi.org/10.2514/6.2020-2915
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional