Steven Tze Fung Lam, Alan H.S. Chan (2024). Application of Artificial Intelligence, Machine Learning and Deep Learning in Piloted Aircraft Operations: Systematic Review. In: Artificial Intelligence and Social Computing. AHFE Open Access, vol. 122, pp. 284–305. https://doi.org/10.54941/ahfe1004666
1 Industrial Centre, The Hong Kong Polytechnic University, Hong Kong SAR, China; Department of Systems Engineering, City University of Hong Kong, Hong Kong SAR China ↗
2 Department of Systems Engineering, City University of Hong Kong, Hong Kong SAR China ↗
Aviation research on artificial intelligence (AI), machine learning (ML), and deep learning (DL) has seen significant growth as these emerging technologies hold immense potential for supporting both human-centred and technology-centred aspects of civil aircraft operations. This systematic review, following the guidelines of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020, was registered on the Open Science Framework (DOI 10.17605/OSF.IO/ZR7A3) and focused specifically on the use of AI, ML, and DL in human-centric flight operations. The review conducted a comprehensive search of databases including Scopus, Web of Science, IEEE Xplore, as well as online repositories (ResearchGate and Aerospace Research Central) to identify relevant articles published between 2013 and 2023. In total, 32 studies were included, which explored various applications of AI, ML, and DL in aircraft pilots and flight operations. The studies were categorized into four main areas: (i) assessment and management of human factors risks, including AI-assisted data analysis of pilot performance, crew resource management, and ML-based support for pilots’ cognitive workload monitoring, (ii) detection of human errors, with support systems based on ML-based approaches for real-time monitoring and DL models for biometric monitoring of cockpit pilots were identified for the detection of human errors in flight safety, (iii) reduction and prediction of human errors, categorized into AI-assisted predictive analytics in flight accidents, and ML-based pattern recognition to predict unstable approaches, and (iv) prevention of human errors in aviation through ML utilization for pilot training enhancement, and AI-supporting flight automation and decision support systems for flight operation. Analysis of the included studies revealed a rising trend in the publication of articles after 2020, albeit at a slow rate. It is worth noting that the majority of studies focused on conceptual applications, with fewer studies involving empirical testing. The findings of this review highlight the potential for future research in developing and testing improved human factors risk assessment (HRA) models assisted by computational intelligence in piloted aircraft operations, with the ultimate aim of enhancing flight safety.
Application of Artificial Intelligence, Machine Learning and Deep Learning 285
INTRODUCTION
Pilot errors continue to pose a significant challenge within the general avia- tion sector, contributing to a majority of accidents. Data from the Interna- tional Civil Aviation Organization’s (ICAO) integrated Safety Trend Analysis and Reporting System, covering the period from 2013 to 2022, reveals that out of 815 reviewed and validated general aviation accidents, 70.6% are attributed to human errors (ICAO Accident Statistics 2013-2022, ICAO Safety Report 2023). These errors predominantly stem from pilot-related mistakes during flight operations, encompassing factors such as misjudge- ments of abnormal flight operations, pilot fatigue, diminished situational awareness, and suboptimal decision-making.
Artificial Intelligence (AI) encompasses the creation of computer systems with the ability to carry out tasks that traditionally demand human intelli- gence. Machine Learning (ML) involves the crafting of algorithms or systems that learn from specific training data to automate analytical model construc- tion and address related tasks. Deep Learning (DL), a subset of machine learning, relies on artificial neural networks characterised by multiple layers (Janiesch, Zschech, and Heinrich, 2021).
According to a number of reviews, the research related to AI, ML and DL has expanded significantly. The expansion is attributed to the significant potential these emerging technologies hold in enhancing various aspects of civil aircraft operations, encompassing both human-centric and technology- centric domains. However, little is known about how AI can be utilized to improve different aspects of pilot safety. AI applications can provide decision-making support, real-time monitoring, and analyzing large datasets to identify patterns and potential risks. ML algorithms can be trained on var- ious datasets such as historical flight data and human performance metrics to detect patterns related to potential hazards, errors, or critical situations. DL can be applied to analyze complex data sources, such as physiological sig- nals or cockpit images, to identify subtle patterns that may indicate potential safety issues.
Hence, this review aims to systematically examine the current status of research on the application of AI, ML and DL in piloted aircraft operations for flight safety. The results of this review provide insights for practition- ers and researchers on AI’s theoretical and practical uses of AI applications in aviation safety, highlighting areas for improvement like training, proce- dures, and equipment design. The study was categorized into four main areas: (i) assessment and management of human factors risks, (ii) detection of human errors, (iii) reduction and prediction of human errors, and (iv) prevention of human errors to enhance flight operation safety.
METHODS
This systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines and was registered on the Open Science Framework (DOI 10.17605/OSF.IO/ZR7A3). A com- prehensive database search of Scopus, Web of Science, IEEE Xplore, and two
286 Lam and Chan
online repositories (ResearchGate and Aerospace Research Central) was con- ducted to find all articles relevant to the topic. See Table 1 for the search strategy. The search was performed on 20 September 2023 and included two sets of keywords. The first set of keywords were “artificial intelli- gence,” “machine learning”and “deep learning”. The second set of keywords included “aircraft cockpit,” “flight operation,” “aircraft operation,” “air- craft pilot,” “airline pilot” and “flight crew”. This PRIMSA-based review comprised articles published between January 2013 and September 2023.
Table 1. Search strategy and search results.
Table available in the original PDF paper.
Search String # of titles and abstracts
Scientific database (2013–2023 September)
363
Scopus Article title, Abstract, Keywords (“Artificial Intelligence” OR “Machine Learning” OR “Deep Learning”) AND Article title, Abstract, Keywords (“aircraft cockpit” OR “flight operation” OR “aircraft operation” OR “aircraft pilot” OR “airline pilot” OR “flight crew”)
2465
Web of Science #1 TI= (“Artificial Intelligence” OR “Machine Learning” OR “Deep Learning”) OR AB= (“Artificial Intelligence” OR “Machine Learning” OR “Deep Learning”) #2 TI= (“aircraft cockpit” OR “flight operation” OR “aircraft operation” OR “aircraft pilot” OR “airline pilot” OR “flight crew”) OR AB= (“aircraft cockpit” OR “flight operation” OR “aircraft operation” OR “aircraft pilot” OR “airline pilot” OR “flight crew”) #1 AND #2
129
IEEE Xplore ((Document Title: Artificial Intelligence OR Document Title: Machine Learning OR Document Title: Deep Learning) OR (Abstract: Artificial Intelligence OR Abstract: Machine Learning OR Abstract: Deep Learning)) AND ((Document Title: aircraft cockpit OR Document Title: flight operation OR Document Title: aircraft operation OR Document Title: aircraft pilot OR Document Title: airline pilot OR Document Title: flight crew) OR (Abstract: aircraft cockpit OR Abstract: flight operation OR Abstract: aircraft operation OR Abstract: aircraft pilot OR Abstract: airline pilot OR Abstract: flight crew))
Equations and mathematical formulas are available in the original paper PDF.
ResearchGate 23 Aerospace Research Central Total abstracts and titles reviewed: 2986. Total abstracts and titles reviewed minus duplicates: 2552. First selection of studies (after title and abstract review): 669. Second selection of manuscripts/studies (after full text review): 32.
This study screened titles and abstracts based on the following criteria: (a) usage of AI systems and/or ML algorithms and/or DL models in
Application of Artificial Intelligence, Machine Learning and Deep Learning 287
human-piloted aircraft; (b) AI, ML and DL assisting in human operations to identify, analyze, and prevent human errors related to flight operations and (c) being published in English language peer-reviewed articles or conference proceedings. The exclusion criteria were as follows: (a) introduction of any AI, ML and/or DL application in aviation, even including aircraft operations; (b) analysis using AI, ML and/or DL for Human-Machine Interface (HMI) to optimize pilot-aircraft interaction alone; (c) using AI, ML and/or DL to develop cockpit aids for fully autonomous flight; (d) case studies, textbooks, and dissertations and (e) full article not available in electronic document.
The search results were imported into EndNote 21, and duplicates were eliminated. The authors then checked each record based on the title and abstract, following the inclusion and exclusion criteria. Any records that did not meet the criteria were removed using the Covidence systematic review software. The remaining records were subjected to full-text screen- ing, and any excluded records were provided with reasons. Two authors independently screened the records to identify relevant articles, and any disagreements were resolved through group discussion with a third reviewer.
Data were extracted according to a predetermined format that included authors, publication year, study purpose, methods, framework or model for computational intelligence, outcomes, techniques, domains, and results. Additionally, the selected articles were analyzed based on the themes of the studied fields and were classified into four main areas related to human fac- tors for pilots: (i) human factors risk assessment and management, (ii) human error detection, (iii) human error reduction and prediction, and (iv) human error prevention.
RESULTS
Search Results A total of 2948 records were retrieved from searching three databases: Sco- pus (n = 363), Web of Science (n = 2465), and IEEE Xplore (n = 129). After removing 396 duplicates, 2559 records were left for title and abstract screen- ing. 1908 records were excluded after screening the titles and abstracts. Out of the remaining 644 records, 616 were removed because they focused on AI/ML/DL application in aviation only (n = 179), AI/ML/DL for Human- Machine Interface (HMI) (n = 164), used AI/ML/DL for fully autonomous flight and unmanned flight operation (n = 265), were case studies, textbooks, dissertations (n = 5), or had no full-text available (n = 3).
In addition, 38 records were found in two online repositories, Research- Gate (n = 23) and Aerospace Research Central (n = 15). Out of these records, 21 were excluded due to duplicates (n = 7), the introduction of AI/ML/DL application in aviation (n = 2), AI/ML/DL for Human-Machine Interface (HMI) (n = 6), using AI/ML/DL for fully autonomous flight and unmanned flight operation (n = 4), case studies, textbooks, and dissertations (n = 1), and full article not available (n = 1). Finally, 32 papers were deemed relevant to the systematic review (Figure 1). Figure 1 shows the PRISMA process of literature search and selection.
288 Lam and Chan
Figure 1: PRISMA process used to identify and select relevant studies.
ASSESSMENT AND MANAGEMENT OF HUMAN FACTORS RISKS Seven articles focused on the assessment and management of human factors risks in aviation, covering diverse facets such as AI-assisted data analysis (n = 3), crew resource management (n = 1), and cognitive workload moni- toring (n = 3). In the realm of AI-assisted data analysis, Jiang et al. (2021) employ a multi-dimensional data fusion approach and an LSTM model with an attention mechanism to achieve a remarkable 94% detection accuracy for pilots’ mental workload. Paces and Insaurralde (2021) contribute by intro- ducing an AI-based system for manual pilot performance assessment, empha- sizing its potential in pilot training and underscoring the need for ongoing considerations in AI-assisted evaluation. Lounis, Peysakhovich, and Causse (2021) explore the influence of pilot expertise on visual scanning strategies through a flight simulator study, revealing insights into expert pilots’ supe- rior perceptual efficiency and distribution of attention, with implications for aviation training and safety. In the realm of crew resources management, Piera et al. (2022) introduce a socio-technical simulation model to enhance pilot performance in single-pilot cockpits, emphasizing the model’s achievements in understanding Pilot-AI interaction, the impact of extra cognitive tasks, and effective simulation of abnormal flight scenarios.
Application of Artificial Intelligence, Machine Learning and Deep Learning 289
For cognitive workload monitoring, Ji et al. (2022) investigate EEG pat- terns during critical flight phases, highlighting the significance of β-wave changes and the potential applications of their findings in pilot workload management and aviation safety. Mohanavelu et al. (2022) contribute with an ML-based approach to identify pilots’ mental workload, emphasizing the effectiveness of HRV and EEG features and the importance of subject- independent classification in real-time systems. Finally, Taheri Gorji et al. (2023) focus on discriminating cognitive workload levels in pilots using EEG and ML, achieving high accuracy and suggesting future research directions for refining the discrimination of cognitive workload levels. In conclusion, this review provides valuable insights into recent advancements in assessing and managing human factors risks in aviation. The collective findings under- score the pivotal role of AI, ML, and advanced technologies in enhancing pilot performance, improving safety measures, and optimizing cockpit design. The holistic approach showcased in these studies, which integrates techni- cal, social, and cognitive aspects, contributes significantly to the evolving landscape of human factors research in aviation.
DETECTION OF HUMAN ERRORS Eight articles, including AI-supporting real-time monitoring (n = 3) and bio- metric monitoring (n = 5), were identified for the detection of human errors in flight safety. AI-powered systems for real-time monitoring track various parameters, enabling the detection of unusual patterns or deviations. Binias, Myszor, and Cyran (2018) contribute a study utilizing EEG signals to detect pilots’ reactions to unexpected events, aiming to enhance cognitive cockpit performance. The study provides insights into cognitive responses to unfore- seen situations, utilizing ML methods such as Common Spatial Pattern and various classification algorithms. Wu et al. (2019) introduce a novel approach integrating spherical Haar wavelet transform and DLfor tracking a pilot’s attention, demonstrating its potential for improving pupil center detection. Wang et al. (2020) decode pilot behavior consciousness through a multimodal approach incorporating EEG, ECG, and eye movements, analyzing various physiological characteristics using SVM.
AI and DL analyze pilots’ biometric data to detect signs of fatigue, stress, or other conditions contributing to human errors. Lee et al. (2020) pro- pose an MFB-CNN for classifying four pilot mental states based on EEG signals. Deng et al. (2020) emphasize the importance of precisely detecting pilot fatigue using a DL network and EEG signals. The study introduces a novel model, DCSAEN-SoftMax, for fatigue identification, achieving high recognition rates. Han, Kwak, Oh, and Lee (2020) developed a robust sys- tem for detecting pilots’ mental states using multimodal biosignals and a multimodal deep-learning network. The proposed method achieves the high- est accuracy, emphasizing the significance of considering multiple modalities for understanding and predicting pilots’ mental conditions. Li et al. (2023) focus on detecting pilot stress through physiological signals, employing a Transformer-based DL model. The proposed model demonstrates high accu- racy in identifying stress based on various physiological signals. Lee et al.
290 Lam and Chan
(2023) introduce MentalNet, a hybrid deep neural network for continuous monitoring and analysis of pilots’ mental states using EEG signals. The model exhibits superior accuracy compared to conventional models, successfully classifying seven mental states across all subjects.
REDUCTION AND PREDICTION OF HUMAN ERRORS Seven articles were identified for the reduction and prediction of human errors in aviation, categorized into AI-assisted predictive analytics (n = 5) and pattern recognition (n = 2). In the realm of predictive analytics, studies demonstrate the potential of AI in analyzing large datasets to predict and mitigate human errors. Notably, Jain, Misra, and Truong (2022) leverage DL techniques to predict unstable approaches for gen- eral aviation aircraft, employing a recurrent neural network (RNN) model. Cantero et al. (2022) concentrate on predicting go-arounds in non-stabilized approach scenarios, using a ML regression model trained from pilots’ expertise. Odisho, Truong, and Joslin (2022) contribute a comprehensive study on runway safety improvement, developing predictive models to iden- tify and mitigate factors leading to runway excursions, including Unstable Approaches and Rejected Landings. Additionally, Dhief et al.’s (2022) inves- tigation into a predictive model for go-around events using ML within pilot-in-the-loop simulations demonstrates the potential for real-time sup- port. D. Gil et al. (2021) introduce the E-pilot system, a predictive model designed to forecast hard landings during the approach phase of commer- cial flights, employing advanced ML techniques. Madeira et al. (2021) explore the application of ML and natural language processing (NLP) techniques to predict human factors in aviation incident reports, enhanc- ing safety by identifying and categorizing human factors contributing to incidents.
For pattern recognition, Rajendran et al. (2021) focus on predicting situa- tional awareness in pilots using electrocardiogram (ECG) signals, developing a predictive model based on physiological data collected in a flight simulator. Baomar and Bentley (2016) introduce an Intelligent Autopilot System (IAS) that utilizes ML and artificial neural networks to learn piloting skills from human pilots through imitation, aiming to enhance autopilot capabilities and reduce crew dependence. These articles collectively showcase the diverse applications of AI in analyzing large datasets, identifying patterns, and pre- dicting potential errors in aviation. From predicting unstable approaches to enhancing runway safety and predicting human factors, these studies con- tribute valuable insights and tools to improve aviation safety by proactively addressing and mitigating human errors.
PREVENTION OF HUMAN ERRORS This review identified ten articles on the prevention of human errors in aviation. The articles can be categorized into two main themes: training enhancement (n = 3), automation and decision support systems (n = 6). In the realm of training enhancement, AI-based training programs are explored
Application of Artificial Intelligence, Machine Learning and Deep Learning 291
as a proactive measure to prevent errors. Bach et al. (2021) introduce a novel hierarchical structure of finite state machines (FSMs) for interactive pilot training, emphasizing the creation of dynamic and realistic scenarios. The study focuses on the preflight procedure for a Boeing 737 captain, employing dynamic hyperlinks to simulate realistic scenarios. The analysis of hyper- text arrangements and user traversals helps determine threshold values for dynamic hyperlinks, contributing to adaptive learning experiences. Källström et al. (2022) explore the integration of ML agents into pilot training simu- lations. Intelligent agents capable of learning and adapting within scenarios enhance training effectiveness. Experiments compare teams with synthetic agents to those with mixed human and synthetic agents. Human pilots find value in situation awareness maps, but concerns are raised about synthetic agents’ behavior in conflicts. The study concludes by emphasizing the impor- tance of designing learning agents with architectures that generalize across various environments. Mao, Ren, and Zhao (2022) propose an off-axis flight vision display system using ML. The system, designed for take-off and landing, utilizes a deep neural network to predict free-form surface models, streamlining the design process. The ML approach demonstrates effectiveness in predicting surface coefficients, saving design time and improving efficiency in optical engineering.
Automation and decision support systems are explored as tools to reduce human errors. Lyu, Nandiganahalli, and Hwang (2018) address mode confu- sion detection in the flight deck using a Generalized Fuzzy Hidden Markov Model (GFHMM), providing successful detection and proposing enhance- ments. Álvarez, González, and Gracia (2020) delve into the integration of automation in flight procedures, introducing the Cockpit Automation Proce- dures System (CAPS) to address safety challenges. Würfel, Djartov, Papenfuß, and Wies (2023) present the Intelligent Pilot Advisory System (IPAS), an AI-based decision support system aiming to enhance decision-making on air- line flight decks. Ramos et al. (2023) discuss the conceptual design of the Integrated Flight Advisory System (IFAS), emphasizing improved decision- making, situational awareness, reduced workload, and increased safety. Watkins, Gallardo, and Chau (2018) introduce a Pilot Support System employing ML for real-time decision support and pattern recognition to enhance pilot decision-making. Zhang, Sun, and Zhang (2021) employ a system dynamics approach to investigate the evolutionary game and col- laboration mechanism in future intelligent aircraft cockpits. Pitchammal and Sadda (2013) discuss the application of AI techniques in the criti- cal mission computer (MC) of fighter aircraft, outlining the development of an Intelligent Mission Computer (IMC) to reduce pilot workload and enhance decision-making. Together, these articles contribute significantly to advancing aviation safety through innovative training methodologies, AI integration in decision support, and automation technologies in the flight deck, emphasizing a holistic approach targeting human factors and real-time adaptability.
292 Lam and Chan
(Continued)
behaviour and progress during runway approaches was
validated.
Professional pilots, compared to novices, exhibit superior
perceptual efficiency, better attention distribution, and more
complex scanning patterns. Flight performance data support
expertise in better flying performance. ML achieves 93%
accuracy in classifying expertise.
Socio-technical simulation model proves effective in reducing the
negative effects of interruptions and improving overall
performance in demanding flight scenarios, with implications
for designing new Pilot-AI interactions.
A substantial increase in the proportion of β-wave during
critical phases of take-off and landing.
during flight training The effectiveness of the AI algorithms in evaluating pilots’
expertise on visual scanning strategies in the
cockpit.
Electroencephalogram (EEG) data LSTM model integrated with an attention mechanism achieved a
future single-pilot cockpit through the
development of a socio-technical simulation
model.
patterns of pilots during critical phases of
take-off and landing.
(AI/ML/DL) Domain Results
KNN) Investigation of the influence of pilot
I. Assessment and management of human factors risks
Table 2. Studies of AI, ML, and DL applications in piloted aircraft operations (n = 32).
Table available in the original PDF paper.
Author
(Publication year) Application
AI-assisted data analysis
Jiang et al. (2021) DL (LSTM) Assessment of pilots’ mental workload using
94% detection accuracy for 2-second EEG data. Paces & Insaurralde
(2021) AI Assessment of manual pilot performance
Lounis, Peysakhovich &
Causse (2021) ML (Cosine
Crew resource
Piera et al. (2022) AI Enhancement of pilot performance in the
Cognitive workload
Ji et al. (2022) ML Investigation of Electroencephalogram (EEG)
Application of Artificial Intelligence, Machine Learning and Deep Learning 293 (Continued)
There is effective differentiation between low and high mental
workload conditions, with HRV features excelling in takeoff and
landing, and EEG features performing better during cruise.
Combining power spectral density (PSD) and log energy entropy,
achieved an accuracy of 91.67%, precision of 93.89%, recall of
91.67%, and an F-score of 91.22%, which effectively discriminate between low, medium, and high cognitive workload
states in pilots during flight.
The study compared different classification algorithms and
found that the Neural Network classifier achieved the best mean
accuracy, outperforming Linear Discriminant Analysis (LDA)
and other classifiers by almost 5% and 8%, respectively.
The proposed DHCAEN-FGSVM presents a detection accuracy
of 94.8% with SOHO wavelet input features, demonstrating
effectiveness through experiments on public and actual datasets.
pilots during flight operations, with a specific
focus on fighter pilots engaged in air-to-air
combat.
Electroencephalography (EEG) to discern and
discriminate cognitive workload levels in
aircraft pilots during flight.
Employing the ML approach to detect a
pilot’s reaction to unexpected events using
Electroencephalographic (EEG) signals
Development of a novel approach for tracking
a pilot’s attention by the method integrates
spherical Haar wavelet transform and DL
techniques.
(AI/ML/DL) Domain Results
k-NN, SVM,
RFs, ANNs)
DAEN-SVM,
DSAEN-SVM,
DCAEN-
FSVM,
DCAEN-
FGSVM,
DHCAEN-
FGSVM.)
Table 2. Continued
Table available in the original PDF paper.
(Publication year) Application
Mohanavelu et al.
(2022) ML Assess and classify the mental workload of
Taheri Gorji et al.
(2023) ML Utilizing ML method and
II. Detection of human
errors
AI-supporting real-time
Binias, Myszor, and
Cyran (2018) ML (LDA,
Wu et al. (2019) DL (SVM,
294 Lam and Chan
(Continued)
distraction, respectively.
Experimental setups employing FIR filters extract δ, θ, α,
and β waves, establishing their correlation with fatigue.
DCSAEN-SoftMax utilizes a contractive loss function and sparse
Joslin (2022) ML Development of predictive models using
Dhief et al. (2022) ML Development of a predictive model for
D. Gil et al. (2021) ML (LSTM) Development of the E-pilot system, a
Madeira et al. (2021) ML (LS, SVM,
Application of Artificial Intelligence, Machine Learning and Deep Learning 297 (Continued)
The study reveals that the Decision Tree (DT) classifier,
particularly emphasizing maximum and minimum ECG values,
outperforms other classifiers. It achieves 98% accuracy and
97% Area Under the Curve (AUC) across ten repetitions. This indicates that the DT model, when applied to electrocardiogram
(ECG) signals, is effective in predicting pilot awareness states,
specifically distinguishing between distracted and non-distracted
states with high accuracy.
Experimental results indicate that the IAS performs take-off,
climb, and slow ascent tasks with high accuracy, even when
provided with limited examples. This is demonstrated by the
Mean Absolute Error and Mean Absolute Deviation metrics.
The IAS is capable of imitating low-level sub-cognitive abilities,
such as rapid and continuous stabilization attempts during
turbulent weather, as well as high-level strategic skills, including
the sequence of sub-tasks necessary to pilot an aircraft from the
stationary position on the runway to a steady cruise.
the utilization of ML to predict situational
awareness in pilots using electrocardiogram
(ECG) signals. The research focuses on
enhancing aviation safety by developing a
predictive model based on physiological data
collected in a flight simulator. ECG signals are
utilized to infer the cognitive state of pilots
and predict their levels of situational
awareness.
System (IAS) that utilizes ML and ANNs to
learn piloting skills from human pilots
through imitation. The study demonstrates
the IAS’s effectiveness in calm and stormy
weather conditions. In calm weather, the IAS
accurately imitates human pilot actions, while
in stormy conditions, it generalizes and
imitates rapid stabilization actions,
outperforming the simulated Aircraft Flight
Controller (AFC).
(AI/ML/DL) Domain Results
NB, GBM,
Table 2. Continued
Table available in the original PDF paper.
(Publication year) Application
Rajendran et al. (2021) ML (DT, RF,
Baomar and Bentley
(2016) ML (ANNs) Development of an Intelligent Autopilot
298 Lam and Chan
(Continued)
The research involves simulations, focusing on the prefight pro-
cedure for a Boeing 737 captain, with 65 commands and 20,000
simulated trainees.
The study doesn’t explicitly provide the numerical or
quantitative results obtained from the simulations. Overall, the
study suggests that the system has the potential to enhance pilot
training by recommending relevant information through
dynamic hyperlinks, improving the efficiency of the learning
process in flight simulators.
The study included a survey of human pilots, revealing that sit-
uation awareness maps were valuable for coordinating with syn-
thetic pilots. While mixed teams and pure agent teams achieved
an 80% success rate, human pilots expressed concerns about syn-
thetic agents’ behaviour in conflicts and suggested modifications
to the agents’ action and observation spaces.
The results of the study indicate similar performance between
the two types of teams, with mixed teams performing slightly
better in certain scenarios.
The results of the study demonstrate the effectiveness of the
ML approach in predicting free-form surface coefficients. The
network’s performance is evaluated using mean absolute error
(MAE), explained variance regression score function (EV), and
R2 regression score function.
Results indicate small MAE, with EV and R2 values close to one,
demonstrating superior performance in regression prediction.
However, the actual numerical results for the evaluation metrics
(MAE, EV, R2) are not explicitly mentioned.
state machines (FSMs) framework, in creating
dynamic and realistic scenarios for effective
pilot training.
training simulations, provides a more realistic
and personalized experience for pilots,
ultimately improving their decision-making
abilities.
ZPL-macro programming to define optical
specifications and corresponding surface
expressions for the flight vision display
system, enhancing pilot training during
take-off and landing by designing a free-form
surface for the flight vision display system.
(AI/ML/DL) Domain Results
Table 2. Continued
Table available in the original PDF paper.
(Publication year) Application
Bach et al. (2021) AI (FSMs) Development and implementation of the finite
Källström et al. (2022) ML Implementing ML algorithms into pilot
Mao, Ren, and Zhao
(2022) ML Development of ZEMAX software and
Application of Artificial Intelligence, Machine Learning and Deep Learning 299 (Continued)
The algorithm proposed for detecting mode
confusion was tested and demonstrated using the
real-life example of the Asiana 214 accident in
2013. The results indicate that if the divergence of intent could have been detected 35 seconds before
the impact, the pilot would have had enough time
to respond and potentially prevent the accident.
The study discussed the methods employed for the
prototype development, including the creation of a
CAPS database and its integration with the aircraft
simulator. The validation process involves
crew-in-the-loop evaluations in various flying
scenarios with system failures. However, the results
or outcomes of the CAPS prototype were not
mentioned.
The proposed concept model of IPAS with eight
detailed use cases, categorizing them into
Information Support and Decision Support. The
system model includes modules for data collection,
AI-based situation assessment (AICOM), and
presenting results to pilots (AICIS). However, the
IPAS was not validated and no results were
presented in this study.
The conceptual design of IFAS was described, for
enchaning pilots’ situation awareness (SA) and
reducing workload for safer flights. The IFAS
model was not validated and no results were
presented in this study.
Hidden Markov Model (GFHMM) for flight deck
mode confusion detection within an intent-based
framework to model the behaviour of both the
automation system and pilot interactions.
(CAPS) was designed to automate procedures in a
simulated civil Airbus A330 aircraft, offering crew
assistance and augmentation during system failures
and associated procedures.
System (IPAS), which is an AI-based decision
support system for enhancing decision-making on
the flight deck by providing intelligent advisories to
pilots based on real-time data and analysis.
Integrated Flight Advisory System (IFAS) based on
AI models. The focus is on providing real-time
guidance and advice to flight crews during flight
operations, with the aim of enhancing
decision-making processes, improving situational
awareness, reducing workload, and increasing
overall safety in aviation.
(AI/ML/DL) Domain Results
(GFHMM) Propose and implement a Generalized Fuzzy
AI Development of an Intelligent Pilot Advisory
Table 2. Continued
Table available in the original PDF paper.
(Publication year) Application
Lyu, Nandiganahalli,
and Hwang (2018) ML
Álvarez, González, and
Gracia (2020) AI The Cockpit Automation Procedures System
Würfel, Djartov,
Papenfuß, and Wies
(2023)
Ramos et al. (2023) AI Development and conceptual design of an
300 Lam and Chan
The system’s methodology, algorithms, and applications, were
presented on mitigating pilot errors through real-time analysis of
flight data but lacks validation of its results in the study.
The study provided preliminary proof of the proposed method
with a dynamic simulation of human-computer interaction in an
intelligent aircraft cockpit, triggered by an error mode. Findings
indicate fluctuations in task load and human error probability,
with a tendency for the system to favour collaborative states.
Sensitivity analyses show that higher human error probability
amplification factors and task load change rates influence
decision-making strategies.
The IMC highlights the role of cognitive automation in
improving aircraft performance and mission capabilities. Testing
methods such as rules and cross-reference verification, line of
reasoning assessment, and empirical testing were employed,
however, the validation of the results in the study was
insufficient.
enhance pilot decision-making was
introduced. The system analyzes various flight
data and employs a Pattern Recognition
Function to identify patterns similar to those
before previous accidents, triggering alarms in
cases of potential pilot impairment or
inattention.
investigate the evolutionary game and
collaboration mechanism of human-computer
interaction in future intelligent aircraft
cockpits. The study establishes an
evolutionary game model involving a flight
crew and an intelligent decision-making
system, providing insights into modelling
complex interactions in intelligent cockpit
environments.
the critical mission computer (MC) of fighter
aircraft to address the increasing complexity
of modern cockpit environments. The paper
outlines the development of an Intelligent
Mission Computer (IMC) and its potential to
reduce the pilot’s workload, improve
decision-making, and assist in various
mission-related tasks.
(AI/ML/DL) Domain Results
Table 2. Continued
Table available in the original PDF paper.
(Publication year) Application
Watkins, Gallardo, and
Chau (2018) ML A Pilot Support System employing ML to
Zhang, Sun, and Zhang
(2021) AI Applied a system dynamics approach to
Pitchammal and Sadda
(2013) AI Discusses the application of AI techniques in
Application of Artificial Intelligence, Machine Learning and Deep Learning 301
DISCUSSION
The systematic review provides a comprehensive overview of recent advance- ments in assessing, managing, detecting, reducing, predicting, and preventing human factors risks in aviation through the use of AI, ML, and DL in human-centric flight operations for continuous airworthiness.
The studies in the assessment and management of the human factors risk category underscored the importance of AI in diverse aspects of human fac- tors risks, from mental workload detection to crew resource management. The findings suggest that AI can significantly contribute to enhancing pilot performance, safety measures, and cockpit design by addressing technical, social, and cognitive aspects. Jiang et al. (2021) achieved a 94% detection accuracy for pilots’ mental workload using multi-dimensional data fusion and an LSTM model. Paces and Insaurralde (2021) introduced an AI-based system for manual pilot performance assessment, emphasizing its potential in pilot training. Piera et al. (2022) also introduced a socio-technical simulation model to enhance pilot performance in single-pilot cockpits.
Articles for the detection of human errors highlighted the role of AI and ML in real-time monitoring and biometric analysis for the detection of human errors, including the analysis of physiological signals such as EEG, ECG, and other bio-signals, is a recurring theme in the literature. They demon- strate the potential of AI in tracking physiological parameters to identify signs of fatigue, stress, or other conditions that may contribute to errors. Binias, Myszor, and Cyran (2018) used EEG signals to detect pilots’ reactions to unexpected events, providing insights into cognitive responses. Lee et al. (2023) introduced MentalNet, a hybrid deep neural network for continuous monitoring and analysis of pilots’ mental states using EEG signals.
Concerning the reduction and prediction of human errors, research studies explored the potential of AI in analysing large datasets, predicting poten- tial errors, and mitigating risks. They ranged from predicting unstable approaches to enhancing runway safety, showcasing diverse applications of AI in improving aviation safety. Jain, Misra, and Truong (2022) lever- aged DL techniques to predict unstable approaches for general aviation aircraft. Odisho, Truong, and Joslin (2022) conducted a comprehensive study on runway safety improvement, developing predictive models for runway excursions.
Articles under the category of the prevention of human errors empha- sized the proactive measures of preventing human errors, including AI-based training programs and decision support systems. They highlighted the poten- tial of AI and DL in improving training effectiveness, decision-making on flight decks, and overall situational awareness. Bach et al. (2021) introduced a novel hierarchical structure of finite state machines for interactive pilot training. Lyu, Nandiganahalli, and Hwang (2018) addressed mode confu- sion detection in the flight deck using a Generalized Fuzzy Hidden Markov Model. It is worth noting that the majority of studies in this category focused on conceptual applications, with fewer studies involving empirical testing.
This systematic review identifies research gaps and suggests future trends in AI applications for aviation safety. Studies in this systematic review were conducted in simulated or controlled environments, which implies the need
302 Lam and Chan
for potential research that integrates AI applications with human factors considerations and impacts decision-making for crew operations in real oper- ational settings to ensure their practical utility under the complexities of actual flight conditions. It involves collaboration with airlines and aviation authorities to implement and assess AI systems in actual flight scenarios. The adaptability and robustness of AI models in handling unforeseen or novel scenarios are limited in existing studies from this review.
Understanding how AI systems respond to unexpected situations is essen- tial for ensuring safety across a wide range of circumstances. Given the dynamic nature of flight operations like turbulence, bad weather conditions and wind shear effects while landing which are very common for crew to face in every flight. Future research is likely to focus on enhancing the adaptability of AI systems to unforeseen scenarios. It involves developing AI models capa- ble of handling novel situations, providing real-time insights and ensuring the robustness of these systems in diverse operational conditions.
Generalization across different civil aircraft types from different aircraft manufacturers such as Boeing and Airbus, researchers may need to explore the generalizability of AI models across different types of aircraft and opera- tional contexts, for example, can the AI application be adapted to Boeing B787 and Airbus A350 in each aspect to access, assist or monitor pilot behaviour during their flight operation. Understanding how the applications can be adapted and optimized for various aircraft types will be crucial for their widespread implementation.
The literature emphasizes the importance of understanding the dynam- ics of human-AI collaboration and establishing trust between pilots and AI systems. The studies included in the review likely use diverse AI technolo- gies, algorithms, and models. Heterogeneity in these approaches makes it challenging to draw direct comparisons or make overarching conclusions about the effectiveness of AI in aviation safety. Moreover, the ethical impli- cations of AI in aviation, including issues related to flight operation data privacy, algorithmic bias, and airliner flight database transparency, require further exploration. Future research should delve into the ethical dimensions and contribute to the development of robust regulatory frameworks for AI in aviation, investigate optimal interfaces and communication strategies to ensure effective collaboration, the adoption of safety-critical AI and mitigate AI trustworthiness issues (Will Hunt, 2020).
Moreover, this review highlights the potential for future research in devel- oping and testing improved human factors risk assessment (HRA) models assisted by computational intelligence in piloted aircraft operations, with the ultimate aim of enhancing flight safety.
CONCLUSION
The systematic review suggests that future research should focus on validat- ing the effectiveness of computational intelligence models in real operational settings, assessing their adaptability to unforeseen scenarios, exploring gen- eralizability across different aircraft types, and understanding the dynamics of human-AI collaboration to advance continuous airworthiness and human factors research in piloted aircraft operations.
Application of Artificial Intelligence, Machine Learning and Deep Learning 303
REFERENCES
Álvarez, C., González, M. I., & Gracia, A. (2020, October). Flight procedures automation: Towards flight autonomy in manned aircraft. In 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC) (pp. 1–8). IEEE. EU Commission, ‘EU Commission — Proposal for a Regulation of the European
Parliament and of the Council laying down harmonised rules on artificial intel- ligence (Artificial Intelligence Act) and amending certain Union legislative acts, COM/2021/206 final’, 2021. Bach, M., Werner, A., Mrozik, M., & Cyran, K. A. (2021). A hierarchy of finite state machines as a scenario player in interactive training of pilots in flight simula- tors. International Journal of Applied Mathematics and Computer Science, 31(4), 713–727. Baomar, H., & Bentley, P. J. (2016). An Intelligent Autopilot System that learns pilot-
ing skills from human pilots by imitation. In 2016 International Conference on Unmanned Aircraft Systems (ICUAS) (pp. 1023–1031). IEEE. Binias, B., Myszor, D., & Cyran, K. A. (2018). A machine learning approach to the detection of pilot’s reaction to unexpected events based on EEG signals. Computational intelligence and neuroscience, 2018. Cantero, J., Colomer, A., Launet, L., Duchevet, A., De La Hogue, T., Imbert, J. P., &
Naranjo, V. (2022). Go-Around Prediction in Non-Stabilized Approach Scenarios Through a Regression Machine-Learning Model Trained from Pilots’ Exper- tise. In International Conference on Intelligent Data Engineering and Automated Learning (pp. 491–499). Cham: Springer International Publishing. Deng, P. Y., Qiu, X. Y., Tang, Z., Zhang, W. M., Zhu, L. M., Ren, H., & Sheng, R. S.
(2020). Detecting fatigue status of pilots based on deep learning network using EEG signals. IEEE Transactions on Cognitive and Developmental Systems, 13(3), 575–585. Dhief, I., Alam, S., Lilith, N., & Mean, C. C. (2022). A machine learned go-around prediction model using pilot-in-the-loop simulations. Transportation Research Part C: Emerging Technologies, 140, 103704. Gil, D., Hernandez-Sabate, A., Enconniere, J., Asmayawati, S., Folch, P., Borrego-Carazo, J., & Piera, M. À. (2021). E-pilots: A system to predict hard landing during the approach phase of commercial flights. IEEE Access, 10, 7489–7503. Han, S. Y., Kwak, N. S., Oh, T., & Lee, S. W. (2020). Classification of pilots’ mental
states using a multimodal deep learning network. Biocybernetics and Biomedical Engineering, 40(1), 324–336. International Civil Aviation Organization. (2023). International Civil Aviation Organization Safety Report 2023. Montréal, Canada. ICAO. 999 Robert-Bourassa Boulevard Montréal, Quebec, Canada. Jain, T., Misra, S., & Truong, D. (2022). Using Deep Learning to Predict Unsta-
ble Approaches for General Aviation Aircraft. Journal of Aerospace Information Systems, 19(12), 811–817. Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning.
Analysis of pilots’ EEG map in take-off and landing tasks. Biomedical Engineer- ing/Biomedizinische Technik, 67(5), 345–356. Jiang, G., Chen, H., Wang, C., & Xue, P. (2022). Mental Workload Artificial Intelligence Assessment of Pilots’ EEG Based on Multi-Dimensional Data Fusion and LSTM with Attention Mechanism Model. International Journal of Pattern Recognition and Artificial Intelligence, 36(11), 2259035.
304 Lam and Chan
Källström, J., Granlund, R., & Heintz, F. (2022). Design of simulation-based pilot training systems using machine learning agents. The Aeronautical Journal, 126(1300), 907–931. Lee, D. H., Jeong, J. H., Kim, K., Yu, B. W., & Lee, S. W. (2020). Continuous EEG
decoding of pilots’ mental states using multiple feature block-based convolutional neural network. IEEE access, 8, 121929–121941. Lee, D. H., Jeong, J. H., Yu, B. W., Kam, T. E., & Lee, S. W. (2023). Autonomous System for EEG-Based Multiple Abnormal Mental States Classification Using Hybrid Deep Neural Networks Under Flight Environment. IEEE Transactions on Systems, Man, and Cybernetics: Systems. Li, Y., Li, K., Chen, J., Wang, S., Lu, H., & Wen, D. (2023). Pilot Stress Detec-
tion Through Physiological Signals Using A Transformer-Based Deep Learning Model. IEEE Sensors Journal. Lounis, C., Peysakhovich, V., & Causse, M. (2021). Visual scanning strategies in the cockpit are modulated by pilots’ expertise: A flight simulator study. PLoS one, 16(2), e0247061. Lyu, H., Nandiganahalli, J. S., & Hwang, I. (2018). Automation intent infer-
ence using the GFHMM for flight deck mode confusion detection. Journal of Aerospace Information Systems, 15(3), 172–177. Madeira, T., Melício, R., Valério, D., & Santos, L. (2021). Machine learning and natural language processing for prediction of human factors in aviation incident reports. Aerospace, 8(2), 47. Mao, S., Ren, Z., & Zhao, J. (2022). An off-axis flight vision display system design
using machine learning. IEEE Photonics Journal, 14(2), 1–6. Mohanavelu, K., Poonguzhali, S., Janani, A., & Vinutha, S. (2022). Machine learning-based approach for identifying mental workload of pilots. Biomedical Signal Processing and Control, 75, 103623. Odisho, E. V., Truong, D., & Joslin, R. E. (2022). Applying machine learning to
enhance runway safety through runway excursion risk mitigation. Journal of Aerospace Information Systems, 19(2), 98–112. P. Paces and C. C. Insaurralde (2021) Artificially Intelligent Assistance for Pilot Performance Assessment. IEEE/AIAA 40th Digital Avionics Sys- tems Conference (DASC), San Antonio, TX, USA, 2021, pp. 1–7, doi: 10.1109/DASC52595.2021.9594487. Piera, M. A., Muñoz, J. L., Gil, D., Martin, G., & Manzano, J. (2022). A socio-
technical simulation model for the design of the future single pilot cockpit: An opportunity to improve pilot performance. IEEE Access, 10, 22330–22343. Pitchammal, R., & Sadda, V. (2013). Making the Mission Computer Intelligent – A Step Ahead. Defence Science Journal, 63(2), 174–180. https://doi.org/10.14429/ dsj.63.4260. Rajendran, A., Kebria, P. M., Mohajer, N., Khosravi, A., & Nahavandi, S. (2021,
December). Machine learning based prediction of situational awareness in pilots using ecg signals. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 1–6). IEEE. Ramos, M. A., Sankaran, K., Guarro, S., Mosleh, A., Ramezani, R., & Arjounilla, A. (2023). The need for and conceptual design of an AI model-based Integrated Flight Advisory System. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 237(2), 485–507. Taheri Gorji, H., Wilson, N., VanBree, J., Hoffmann, B., Petros, T., & Tavakolian,
K. (2023). Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight. Scientific Reports, 13(1), 2507.
Application of Artificial Intelligence, Machine Learning and Deep Learning 305
Würfel, J., Djartov, B., Papenfuß, A., & Wies, M. (2023). Intelligent Pilot Advisory System: The journey from ideation to an early system design of an AI-based decision support system for airline flight decks. AHFE 2023. Wang, X., Gong, G., Li, N., Ding, L., & Ma, Y. (2020). Decoding pilot behavior
consciousness of EEG, ECG, eye movements via an SVM machine learning model. International Journal of Modeling, Simulation, and Scientific Computing, 11(04), 2050028. Watkins, D., Gallardo, G., & Chau, S. (2018). Pilot support system: A machine learning approach. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC) (pp. 325–328). IEEE. Will Hunt (2020). The Flight to Safety-Critical AI: Lessons in AI Safety from the
Aviation Industry. CLTC White Paper Series, UC Berkeley Center for Long-Term Cybersecurity. https://cltc.berkeley.edu/research/research-library/ Wu, E. Q., Zhou, G. R., Zhu, L. M., Wei, C. F., Ren, H., & Sheng, R. S. (2019).
Rotated sphere haar wavelet and deep contractive auto-encoder network with fuzzy Gaussian SVM for pilot’s pupil center detection. IEEE transactions on cybernetics, 51(1), 332–345. Zhang, X., Sun, Y., & Zhang, Y. (2021). Evolutionary game and collaboration mechanism of human-computer interaction for future intelligent aircraft cock- pit based on system dynamics. IEEE Transactions on Human-Machine Systems, 52(1), 87–98.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Artificial Intelligence and Social Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-98-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional