Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Effectiveness of Knowledge Models for Visual Object Detection

1 Electronic Navigation Research Institute, Tokyo, 182-0012, Japan ↗
2 Graduate School of Engineering, The University of Tokyo, Tokyo, 113-8656, Japan ↗ ↗
Publication year
Digital Object Identifier10.54941/ahfe1005957

Abstract

There are various effective methods of image processing for detecting moving objects from video information, such as background subtraction, optical flow, and edge extraction. Some image processing techniques such as image difference detection can detect targets small pixels. On the other hand, these methods do not recognize and distinguish the target itself. Image recognition such as machine learning can also detect objects from video information. However, if there are not enough pixels to recognize the target, it is difficult to find it by image recognition. In this study, we tried to develop a mechanism for automatic detecting function of aircraft approaching to the airport by using visual cameras. In the situation where aircraft flying to the airport for landing, the image of aircraft appears as a small dot. And, this target image does not have enough pixel information to identify the aircraft by image recognition techniques. However, a tower controller can identify whether a small dot is an aircraft or not under the similar condition. This is because air traffic controllers make their decisions based on their operational knowledge and experience. Therefore, we aim to model the knowledge and experience of air traffic controllers explicitly to judge the situation as rules for high accuracy target detection.In order to detect and track a specific object continuously, another mechanism is needed for identifying the object. In this study, we propose a rule-based model as an air traffic controllers' operational knowledge for detecting the target from the frame of video image. We present techniques for the detection and selection of target aircraft that appear as small dots on the image. And then, we discuss the results of our validation of the effectiveness of our proposed approach.

INTRODUCTION

Air traffic control services at airports are provided by “tower controllers” who directly observe the air and ground traffic from the top of a control tower. Recently, a digital/remote tower concept has been developed whereby controllers provide services using live video images from the airport, replacing or supplementing an existing control tower or allowing services to be provided to airports that hitherto lacked such a facility (Fürstenau, 2022).

The use of video provides an opportunity to introduce new functions that can support the controller. One example is “box-and-follow”, which automatically recognises moving objects, tracks their motions and superimposes a frame around them on the video display to support situational awareness. However, it is necessary for the function to discriminate between objects that are likely to be of interest to controllers, such as vehicles or persons in certain locations, and non-relevant objects, to avoid “nuisance” indications that increase display clutter.

As part of ENRI’s remote tower research, we have been working to improve box-and-follow by increasing the reliability of moving object

© 2025. Published by AHFE Open Access. All rights reserved. 88

Effectiveness of Knowledge Models for Visual Object Detection 89

detection while suppressing irrelevant box-and-follow indications. A particularly challenging class of relevant objects is aircraft approaching the airport. These initially they appear as small dots in the sky. Since the number of object pixels is small, visual characteristics such as shape cannot be discerned and so object recognition techniques cannot be applied. On the other hand, classical image processing methods such as background subtraction, optical flow, and edge extraction can detect such small dot-like objects, but cannot discriminate them from objects such as birds and clouds. Even when aircraft are too distant to appear as more than dots, however, air traffic controllers can detect and distinguish them as aircraft. It can be thought that controllers make such determinations based not only on object appearance, but on a synthesis of information such as the positions of dots compared with the expected positions of traffic, their apparent speed and direction, and behaviours, applying experience and knowledge.

This paper reports on a study aiming to improve the performance of box-and-follow for aircraft approaching an airport by combining object detection techniques with rule-based processing based on a model that captures controller’s formal knowledge. In this following, we first give an overview of image processing-based object detection methods used by our remote tower, and their drawbacks for the object class of interest. We then describe our knowledge-based model to recognise and track such objects, and report the results of initial experiments using our remote tower installation at Sendai airport. Finally, we discuss the effectiveness of the proposed approach.

METHODS OF TARGET DETECTION The basic object detection methods applied in this study are background subtraction (Kalsotra, 2021) and edge extraction (Xu, 2020). In our remote tower system, background subtraction is applied as the first processing step. Background subtraction finds moving objects from differences between successive video images. The difference calculation is performed pixel-by- pixel and an object is recognised as a moving object if its change exceeds a certain threshold value. We set 2x4 pixels as the threshold value in our case studies.

The motions of some objects of interest may be too small to be detectable by background subtraction; for example, distant aircraft flying almost directly towards or away from the camera will have little or no apparent motion between successive video images. To address such cases, we apply edge-based image extraction to supplement background subtraction. Edge extraction requires a filter function to determine continuity to determine targets simultaneously.

The sensitivity of these differential detection methods can be tuned by a parameter value. Increasing the sensitivity allows finer image differences to be detected, but too high a sensitivity may also detect non-relevant objects such as clouds. Background subtraction supplemented by edge extraction has the advantage of low computer processing requirements, and so can operate in real-time; however, it does not provide object recognition and so cannot

discriminate between operationally relevant and non-relevant objects, which will lead to display clutter in the box-and-follow application.

The specific target of this study is the important but difficult case of detecting aircraft approaching an airport for landing. When an aircraft approaches an airport, it is detected in images in which it appears as a dot at distances from 10 NM (18,520 m) to 2 NM (3,740 m). These dots have insufficient pixel information to distinguish shape, so object recognition techniques such as machine learning (Fujiyoshi, 2019) and reinforcement learning (Alrebdi, 2022), cannot be applied. We attempt improve the detection of such objects and their discrimination as objects of operational interest by applying a formal knowledge model for aircraft identification based on an analysis of the controller’s operations.

KNOWLEDGE MODEL FOR RULE BASED PROCESSING Approaching Aircraft Behaviour

Figure 1 shows a plan-view radar-type display image of two aircraft approaching Sendai airport to land. Sendai airport is located near the coast, and its runways are shown in the figure in blue. Sendai airport has two runways, and under normally prevailing wind conditions, aircraft approach from over the sea to land in a westerly direction. In the discussion that follows, we assume aircraft landing to the west on runway 27. The extended centerline of the runway is drawn as an orange line from the runway along the direction of its axis to a distance of approximately 10 NM (18,520 m) from the runway threshold. The runway’s instrument approach path is delineated by the magenta arrow-shaped outline.

Figure 1: Plan-view radar display-type image of traffic approaching runway 27 at Sendai airport.
Original PDF Figure 1:

Flights such as airliners operating under instrument flight rules (IFR) are typically guided by radar to intercept the extended centerline in level flight, and then intercept and follow a 3-degree sloping “glideslope” to the runway. This gives a descent along the glideslope of about 300 ft (91.4 m) per 1 NM (1,852 m) horizontal distance. In the radar image, aircraft and their tracks

Effectiveness of Knowledge Models for Visual Object Detection 91

(previous positions at approximately 1 second intervals) are shown as red dots. There are two aircraft inbound from the south, and one has turned west to intercept the extended centerline and glideslope.

For traffic operating under visual flight rules (VFR), aircraft typically intercept the approach course at an angle of between 45–90 degrees or so from a much closer distance, say 2 NM (3,704 m) or less, at a height of around 500–1,000 ft (152–305 m). VFR aircraft are typically small propeller-driven airplanes or helicopters and are much smaller than airliners.

From the above discussion, the motions of aircraft approaching the airport can be broken down into the following phases: (i) Initial approach: Approach the runway centerline from an approximately perpendicular direction; (ii) Intercept Final Approach Course: turn onto the runway extended centerline; (iii) Final approach: fly level along the extended centerline until intercepting the glideslope, then descend along the glideslope.

Mapping Approach Phases to Display Areas in Remote Tower Video Images The areas of the sky in which these phases occur can be mapped into areas in the remote tower video images, as shown in Fig. 2, which shows the areas superimposed on a view from our test remote tower camera installation at Sendai airport. For (i), aircraft objects will be moving across the field of view but appear as only small point-like dots. In (iii), the aircraft will be approaching approximately along the line of sight, and so have little lateral apparent motion in video images, but will descend and also grow larger as they approach the camera position. (ii) is a transition phase.

(Note that in Fig. 2, our cameras are located on a 30 m-tall tower at ENRI’s branch office adjacent to Sendai airport, which would not be the optimum location for a remote tower for air traffic control use. A better location would be on top of the existing control tower, which can be seen in the image.)

Figure 2: Example image of “Area” and “Path”, and plotting flight path to Runway27 of Sendai airport.
Original PDF Figure 2:

Knowledge Model for Identifying Aircraft Approaching Airport This article describes a knowledge model for the situation of air traffic inbound to an airport. The knowledge used by air traffic controllers to identify such aircraft was extracted based on task analysis, and can be described as decision rules based on six conditions: (1) area of appearance,

(2) continuity of flying objects, (3) flight path, (4) size, (5) amount of movement, and (6) direction of movement.

The application of these rules to the detection of inbound aircraft in video images is described below.

1. Area of appearance

The location in which a potential aircraft object appears is checked (the area in the video image corresponding to (i) in Fig. 2), and whether the object is continuously observed in the area (i.e., whether it is discrete). 2. Continuity of flying objects Continuity determines whether a potential aircraft object is detected continuously. Detection must be continuous for a certain period of time to distinguish it from non-relevant nuisance objects.

Aircraft movement and flight direction differ in the Initial Approach and Final approach phases and so require different continuity detection algorithms which are switched at the transitory phase. 3. Flight path Inbound traffic to an airport usually arrives by fixed routes within an area. The routes are identified within the area in the image based on the determined paths and patterns of operation. The final decision is used in conjunction with the relationship between parameters such as the amount of movement in (5) and the direction of movement in (6) to make a decision. 4. Size of object

The approximate size range of the types of aircraft operating at Sendai airport are known for each area to be observed. Objects outside this size range (plus a margin) are excluded. 5. Apparent movement The amount of apparent movement (the difference in object position between successive video frames) is determined by speed range of object and its direction of motion. The approach and landing speed ranges of each aircraft type are known, as is their approximate direction of motion in each area. In the case of aircraft, the amount of movement can be observed and discerned as a value within a certain range. 6. Direction of movement

The approximate direction is known in each area. Landing aircraft can be used as a reference to determine different moving objects, as the direction of movement is known for each area.

Rule-Based Process The combination of the above six conditions can be used to decide whether an object detected in images is an aircraft by applying a rule-based process. The process is shown in Figure 3. The rule flow determines the conditions along with the order of the knowledge model items which described above. First, the process checks the area in which an object is first detected found in the area determination. The system then determines whether the object continues to exist in the area. Thereafter, the continuity of the object is determined. Finally, the object is determined to be an aircraft from the airway, size, movement and direction parameters set for each area.

Effectiveness of Knowledge Models for Visual Object Detection 93

Figure 3: Rule-based flow based on controllers’ knowledge model for detecting approaching aircraft.
Original PDF Figure 3:

EXPERIMENT – EFFECTIVENESS OF RULE-BASED PROCESSING A preliminary experiment to evaluate the effectiveness of the proposed method was conducted using ENRI’s remote tower camera system at Sendai Airport for arrivals to runway 27.

First, the airborne aircraft detection performance of the system using only the differential detection image processing described in METHODS OF TARGET DETECTION was investigated to determine the conditions under which nuisance mis-detections are likely to occur. Reliable detection of aircraft and rejection of nuisance objects such as clouds could be achieved within ranges of approximately 2 NM (3,406 m) from the camera solely by adjusting the sensitivity parameters of the image processing methods, due to differences in size between objects of interest and nuisance objects and other factors. However, when the detection sensitivity was increased to detect aircraft at ranges beyond 2 NM, the rate of nuisance mis-detections increased. Figure 4 shows an example of true aircraft detection (left image) and a nuisance mis-detection of a cloud (right image).

Increasing the sensitivity of the differential detection enables even distant aircraft that appear as little more small dot-like clusters of pixels to be detected, but at the cost of increased nuisance object detections. We therefore tested the application of knowledge mode-based rule-based processing to suppress nuisance detections with the higher sensitivity.

Figure 4: Example of detection (left: approx. 10NM aircraft, right: nuisance mis- detection (clouds)).
Original PDF Figure 4:
Table 1: The result of rule-based processing for detecting aircraft.
Original PDF Table 1:

First, we adjusted the sensitivity of the image processing-based object detection to detect aircraft at a range of approximately 10 NM (18.5 km), which also resulted in nuisance mis-detections. Table 1 shows the results of applying rule-based processing to ten cases where image processing- based detection alone mis-tracked clouds and was unable to track objects continuously from frame to frame. In six out of the cases, mis-detections of clouds were eliminated. In particular, application of the rule-based processing eliminated false object detections at ranges of 3.5–6 NM (6.5–11.1 km), and continuous detection of moving aircraft was possible. In a further three cases, there was momentary mis-detection of cloud detection, but rule-based processing eventually was able to reject the clouds as objects of the target class and supress the nuisance mis-detection. In the last case in table 1, highlighted in orange, the close proximity of an aircraft to cloud in the image resulted in their pixels being clustered together and the rule-based process was unable to distinguish them.

CONCLUSION

An experiment that combining image-processing based differential detection of objects with rule-based processing based on knowledge models improved

Effectiveness of Knowledge Models for Visual Object Detection 95

the detection performance of aircraft approaching an airport from a distance, increasing the range of remote tower “box-and-follow” function and reducing nuisance mis-detections of non-relevant objects. Compared to object recognition techniques such as machine learning, this technique has cost advantages in terms of knowledge construction, and allows aircraft to be discriminated from nuisance objects such as clouds even when they appear only as dot-like clusters of pixels with no shape information discernible. On the other hand, we were unable to eliminate all mis-detections. Therefore, we consider that further detailed rule creation and model refinement are needed for practical use.

REFERENCES

Alrebdi, N. Alrumiah, S. Almansour, A. and Rassam, M. A. (2022). Reinforcement Learning in Image Classification: A Review, proceeding of 2nd International Conference on Computing and Information Technology (ICCIT). Fujiyoshi, H., Hirakawa, T. and Yamashita, T. (2019) Deep Learning-Based Image

Recognition for Autonomous Driving. IATSS Research, 43, 244–252. Fürstenau, N. (2022) Virtual and Remote Control Tower. Research, Design, Development and Validation, 2nd Eds, Springer, Research Topics in Aerospace. Kalsotra, R. and Arora, S. (2021). Background subtraction for moving object

detection: Explorations of recent developments and challenges, The Visual Computer: International Journal of Computer Graphics, Volume 38, Issue 12, pp. 4151–4178. Xu, P., He, Z., Qiu, T. and Ma, H. (2020). Quantum image processing algorithm using edge extraction based on Kirsch operator, Optics Express. 2020 Apr 27;28(9): 12508–12517.

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET-FS 2025): Future Systems and Artificial Intelligence Applications

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET-FS 2025): Future Systems and Artificial Intelligence Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-72-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →