Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Preserving the Human Element in Pilot Weather Reports (PIREPs)

Federal Aviation Administration, Civil Aerospace Medical Institute, Oklahoma City, OK 73169, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1004159

Abstract

Pilot reports (PIREPs) are reports describing in-flight weather conditions submitted by pilots, and provide crucial weather information to other pilots for pre-flight and in-flight planning. However, the current PIREP system is antiquated, prone to error, and has been identified as a safety concern according to a 2017 National Transportation Safety Board Special Report. The current paper describes some preliminary results from a proof of concept study investigating the feasibility, utility, and usability of a PIREP submission and retrieval process that uses VHF radio, cloud computing, and artificial intelligence (AI) technologies. In this concept, pilots were able to submit and retrieve PIREPs without talking to an air traffic controller but by communicating with an automated VHF ground station via a voice-user interface. Pilots were also able to retrieve PIREPs online (website and mobile app), where they could listen to voice recordings of PIREPs or read voice-to-text transcriptions of PIREPs submitted in their local area. Most aircraft today are equipped with a VHF radio, which is the primary communication tool used to file PIREPs via communicating either with a Flight Service or Air Traffic Control facility. The use of a dedicated radio frequency for submitting and retrieving PIREPs has the potential to provide an additional mechanism for pilots to communicate these reports reducing congestion on other FSS and ATC frequencies. For the period of time between November 4, 2022 and April 30, 2023; over 200 pilot-participants who were flying within or overflying the areas with radius of ~ 50 nm from two airports—Will Rogers World Airport (KOKC) in Oklahoma and Nenana Municipal Airport (PANN) in Alaska submitted and retrieved~5,000 PIREPs. Two PIREP ground station identifiers were used for this research: (a) “PIREP Watch Alaska” and (b) “PIREP Watch Oklahoma”. For this initial phase of the research, the system did not have the capability to map ADS-B equipped aircraft to aircraft type, location or altitude without querying the pilot. Pilots identified a number of areas of growth for this concept, such as the need for aviation-specific vocabulary in the voice-to-text transcription aspect of the technology. Overall, pilots were highly enthusiastic about the proof of concept study and supported the feasibility, utility, and usability of this novel method of submitting and retrieving PIREPs. This innovative system preserves the human element in modern weather information sources by allowing pilots to use free form when reporting their direct weather observations in-flight while eliminating the need to talk to an air traffic controller or flight service specialist. Furthermore, it offers data-driven support for modernizing the PIREP system.

INTRODUCTION

After a thorough investigation, the National Transportation Safety Board (NTSB) identified deficiencies in the handling of Pilot Reports (PIREPs) infor- mation that resulted in delays, errors, and data losses. Released in the spring of 2017, the NTSB’s special investigation report included safety recommen- dations to the Federal Aviation Administration (FAA) addressing two broad categories of issues that reduce the effectiveness of PIREPs: submission issues and dissemination issues.

In June of 2020, the FAA Civil Aerospace Medical Institute (CAMI) con- ducted a series of focus groups with pilots and air traffic controllers. The main objective of the focus groups was to improve stakeholder understanding of the deficiencies in the PIREP system from the standpoint of the end- user populations – pilots and controllers. During the discussions, pilots and controllers demonstrated their appreciation of the PIREPs’ significance for aviation safety and stressed the importance of PIREPs during all phases of flight and across the full spectrum of flight operations (Kratchounova, 2020).

The feedback during the focus groups further solidified the perception on the part of both end-user populations (pilots and controllers) that soliciting,

© 2023. Published by AHFE Open Access. All rights reserved. 467

468 Kratchounova and Baumgartner

submitting, capturing, and disseminating PIREPs is not a priority for either pilots or controllers. Task saturation, lack of standardization, and outdated format and technology used were identified as major contributors to such perception. Controllers were convinced that only “official” PIREPs are on decline (Kratchounova, 2020).

All participants agreed that pilots and controllers are aware of the impor- tance of PIREPs. However, they also stressed an immediate need for a major system overhaul. In summary, this feedback highlighted the need for (a) improved user interface design and overall user experience by eliminating the antiquated forms and format simplifying the process of submission and dissemination, and (b) sensible use of automation and new technologies and the need for a hybrid solution involving both automation and human input (Kratchounova, 2020). In light of the focus groups results, in the fall of 2022, FAA CAMI launched a proof-of-concept study to make an initial assessment of a subset of the PIREP system improvement ideas proposed by the focus group participants.

PROOF-OF-CONCEPT RESEARCH The objective of this exploratory-in-nature proof-of-concept research was to examine the feasibility, utility, and usability of a PIREP submission and retrieval concept using VHF radio as the transmission/retrieval medium and natural language processing (NLP) and cloud computing as a method for soliciting, processing, storage, retrieval, and dissemination of PIREPs.

METHOD

Participants Two-hundred thirty-seven pilots participated in this study including student pilots, instructor pilots, general aviation pilots, regional air carrier pilots, as well as air transport pilots. Participants were recruited via email, the web, newsletters, online articles, and in-person. Pilots completed an online sign-up process that included informed consent, demographics, and a participant briefing online. All procedures were approved by the FAA CAMI Institutional Review Board (#202230).

Procedures This study used a PIREP submission and retrieval concept using VHF radio, an internet connection, an automatic ground station (AGS), and a cloud-based web services where PIREPs were processed, stored, and dis- played on web and mobile applications. Two radio communication fre- quencies were used as dedicated PIREP frequencies for the duration of the study – one in Oklahoma and one in Alaska. The pilots interacted with the system via a voice user interface for both submittal and retrieval of PIREPs.

Data collection for this study took place between November 4, 2022, and April 30, 2023. Participants were encouraged to submit at least one PIREP per flight and retrieve the experimental PIREPs any time they were flying or overflying the areas within 50 miles of Will Rogers World Airport (KOKC)

Preserving the Human Element in Pilot Weather Reports (PIREPs) 469

in Oklahoma and Nenana Municipal Airport (PANN) in Alaska via the web and mobile app developed for this study.

PIREP Submittal This prototype generated a voice-to-text transcription using a NLP service as well as an audio recording of the PIREP submitted via VHF radio. Using an internet connection, the AGS sent the text and audio to a cloud service for processing, storage, and display on web and mobile applications.

PIREP Retrieval Pilots in flight were able to use VHF radio to request available PIREPs within a certain geographical area. Outside the flight deck, pilots were able to view all PIREPs submitted as part of this proof-of-concept study as experimental PIREPs as well as all available PIREPs in the National Airspace (NAS) at the time of viewing, by using the web or mobile versions of CAMI’s aviation weather human factors research platform. The experimental PIREPs were available for retrieval by tapping/clicking directly on the icon and in three different modes: a) audio recording, b) voice-to-text transcription (VTT), and c) plain text.

User Surveys Every two-weeks, participants were asked to complete a brief survey assess- ing the feasibility, utility, and usability of the following aspects of this experimental PIREP system:

1) Submission via VHF radio 2) Retrieval via VHF radio (audio playback) 3) Retrieval via website/mobile app (audio playback) 4) Retrieval via website/mobile app (VTT) 5) Retrieval via website/mobile app (plain text).

Participants used a 5-point Likert scale where 1 = Very poor; 2 = Poor; 3 = Adequate; 4 = Very good and 5 = Excellent. In addition to the ratings, participants had the option of including open-ended feedback on any of these 5 system aspects.

RESULTS

Throughout the data collection period, iterative updates to the PIREP sub- mission and retrieval process were made based on user feedback. Therefore, at the time of this writing, to evaluate the final iteration of the program sur- vey responses from the last full month of data collection (March, 2023), are quantitatively described below (n = 131). Additionally, informative open- ended responses are included addressing the overall impact of each aspect of the experimental PIREP submission and retrieval process.

470 Kratchounova and Baumgartner

PIREP Submission via a VHF Radio Call Overall, participants rated the PIREP submission via VHF radio call as one of the best features of the experimental PIREP system. On a scale of 1-to-5, with 5 being excellent, participants rated the submission via VHF radio func- tion a mean score (and standard deviation) of 4.50 ± 0.68 for Usefulness, 4.19 ± 0.95 for Ease of Use, 4.18 ± 0.83 for Ease of Interpretation, 4.21 ± 0.79 for User Experience, and 4.24 ± 0.86 for Overall Operational Viability / Practicability.

Participants appreciated the potential utility of this experimental system and were supportive of its future implementation: “I really like the system. At first, I was sceptical of its ease of use, but after a few practices with it, I find that it is really easy to give and retrieve PIREPs. It’s often difficult to get in touch with ATC on congested frequencies, especially when the weather is bad, so this would be a perfect way to be able to capture that weather data and provide it to pilots in a way that will not be a burden on busy air traffic controllers. Especially for VFR traffic, a pilot is more likely to call the automated system rather than try to contact an ARTCC center controller for PIREPs.”

PIREP Retrieval via a VHF Radio Call Using the VHF radio call function, participants were able to retrieve the audio recordings of recently submitted PIREPs while in flight. Overall, participants rated the PIREP retrieval via VHF radio call positively. On a scale of 1-to-5, with 5 being excellent, participants rated the retrieval via VHF radio func- tion a mean score (and standard deviation) of 4.27 ± 0.98 for Usefulness, 4.28 ± 0.93 for Ease of Use, 4.12 ± 0.94 for Ease of Interpretation, 4.08 ± 1.03 for User Experience, and 4.17 ± 1.02 for Overall Operational Viability / Practicability.

Ratings for retrieval via a VHF radio call were slightly lower than rat- ings for submission via a VHF radio call, likely due to the system having difficulties at times correctly identifying the location for which a user was requesting PIREPs. For example, participants commented: “PIREP retrieval is very helpful for any pilot operating in all environments. The voice recogni- tion software had a little bit of difficulty recognizing the station I was asking about, but that could have happened for a variety of reasons,” and “Great to hear reports in pilots’ own words and voice. System seems to still have some trouble with locations.”

Overall, participants viewed this component of the experimental PIREP submission and retrieval process as essential, noting: “The PIREP retrieval is just as important as the PIREP submissions in my opinion. This will enable pilots to get weather updates even when the ARTCC frequencies are crowded during periods of marginal weather. It very easy to use and it is a huge plus to be able to hear the audio file from the pilot submitting the PIREP first-hand.”

PIREP Retrieval Online via Audio Playback Using the website or mobile app, participants were able to retrieve the audio playback for individual PIREPs by clicking on a PIREP icon from the map.

Preserving the Human Element in Pilot Weather Reports (PIREPs) 471

Overall, participants rated the PIREP audio retrieval via website/app as one of the best features of the experimental PIREP system. Participants rated the retrieval via website function a mean score (and standard deviation) of 4.33 ± 0.89 for Usefulness, 4.37 ± 0.80 for Ease of Use, 4.19 ± 0.83 for Ease of Interpretation, 4.25 ± 0.79 for User Experience, and 4.31 ± 0.86 for Overall Operational Viability / Practicability.

In particular, participants positively commented on the utility of being able to listen to PIREPs through the web or app function during pre-flight plan- ning. For example, participants noted “This is perhaps the glowing gem of the retrieval system thus far. Very useful in pre-flight preparation!”, and “I was sitting trying to make a decision on my go/no-go the other day and I thought to myself, “Wow, this really would be incredibly beneficial to have this as a legal means of weather.”

PIREP Retrieval Online via Voice-to-Text Transcription Using the online website or mobile app, participants were able to retrieve the voice-to-text transcription of individual PIREPs by clicking on a PIREP icon from the map. Overall, participants rated the PIREP retrieval via website transcription positively, though this was the lowest rated component of the experimental PIREP submission and retrieval system. Participants rated the retrieval via VHF radio function a mean score (and standard deviation) of 4.22 ± 1.06 for Usefulness, 4.28 ± 0.85 for Ease of Use, 3.93 ± 1.07 for Ease of Interpretation, 4.08 ± 0.95 for User Experience, and 4.13 ± 0.94 for Overall Operational Viability / Practicability.

While participants largely viewed this component of the retrieval system positively, negative user feedback centered on the limitations in the accu- racy of the voice-to-text transcriptions by the NLP. For example they wrote “Voice to text transcriptions are not always accurate as to what was stated. Interpretation of some words seems to be dependent on the proper language used in giving the PIREP. If the pilot does not have good diction and pro- nunciation of some words, the translation seems to be off. I see a need for improvement with the interpretation to text.”

Still, users overall commented that they overall appreciated this voice-to-text function as a means of retrieving PIREPs. For example, par- ticipants noted “The voice to text is probably the most useful part of the system. Although the transcription is very inaccurate,” and “If the transcrip- tion becomes more accurate, this will be my preferred way to receive the PIREPS and can more easily be integrated into the current NAS and pilot app systems.”

PIREP Retrieval Online via Plain Text Using the online website or mobile app, participants were able to retrieve the individual PIREPs in the plain text format used by many current online and mobile applications. Overall, participants rated the PIREP retrieval via web- site plain text comments positively. Participants rated the retrieval via VHF radio function a mean score (and standard deviation) of 4.24 ± 0.95 for Use- fulness, 4.23 ± 0.86 for Ease of Use, 3.96 ± 0.98 for Ease of Interpretation,

472 Kratchounova and Baumgartner

4.10 ± 0.96 for User Experience, and 4.15 ± 0.95 for Overall Operational Viability / Practicability.

Pilots reported limitations to the plain text retrieval function that were also related to the accuracy of voice-to-text transcriptions by the NLP. Given pilots’ familiarity with the plain text system already, some reported that this would be a preferred means of accessing PIREPs if the transcription accuracy were improved. For example, participants commented that “As long as the text is translated correctly, it is a quicker method to retrieve the PIREP.” In addition, the pilot participants recommended that “The accuracy of the plain text needs to get better. I listen to various pilots talking and leaving PIREP. Some speak clearly and use normal terminology; others are not so clear and use nonstandard descriptions of weather and associated phenomena. More pilot training and experience will yield better results in the long run.”

In summary, participant feedback on the feasibility, utility, and usability of this novel PIREP submission and retrieval concept was very positive. The PIREP submission via a VHF radio call and PIREP retrieval online via audio playback were the most favourably rated components of the process. The VTT and plain text features was identified as needing improvement in order to be viable for implementation in the NAS in the future.

DISCUSSION

While the overall feedback from the pilot participants was favourable regard- ing the feasibility, utility, and usability of this concept, we identified several technical challenges that negatively impacted VTT accuracy and the ability to correctly parse it into plain text, as follows:

1) The use of non-aviation specific Natural Language Processing (NLP). 2) Noise generated from a variety of sources. 3) The limited ground station transceiver power.

In addition to the primary goal of the study and as part of the data col- lected, several elements of PIREP entry were recorded and stored for future analyses including, (a) the source audio from the cockpit, and (b) the tran- scribed text. These data provided a valuable insight to how automated voice recognition systems and VTT, developed specifically for aviation, could opti- mize this concept as well as provide a quality overall user experience for all its end-users.

CONCLUSION

The NTSB special investigation report (NTSB, 2017), as well as the pilots and controllers’ feedback collected during the focus groups (Kratchounova, 2020) proved invaluable for identifying the most error-prone parts of the PIREP system as it exists today. The preliminary results from this research indicate a strong value to doing additional qualitative research, to identify and assess the benefits and impact of automating those specific elements of the current system. A potentially viable approach would be the transition to an aviation- based machine learning model as a first step towards a more sophisticated

Preserving the Human Element in Pilot Weather Reports (PIREPs) 473

artificial intelligence model. While preserving the human element (i.e., the direct weather conditions observation made by a pilot in flight), this could lead to a more efficient and effective human-system integration that will have a positive impact on aviation safety.

REFERENCES

Kratchounova, D. (2020). Pilot Reports (PIREPs) end-user (Pilots and Controllers) focus groups (No. DOT/FAA/AM-20/11). United States. Department of Trans- portation. National Transportation Safety Board. (2017). Improving pilot weather report sub-

mission and dissemination to benefit safety in the national airspace system (Special Investigation Report No. NTSB/SIR-17/02 PB2017-101424).

Cover of the proceedings volume Human Systems Engineering and Design (IHSED 2023): Future Trends and Applications

Explore the complete digital volume

Human Systems Engineering and Design (IHSED 2023): Future Trends and Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-88-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →