JP7764784B2 - Information processing device, information processing method, program, and system - Google Patents

Description

The present invention relates to an information processing device, an information processing method, a program, and a system.

Various technologies have been proposed to allow users to monitor target objects. One example is technology for air traffic controllers to monitor aircraft. For example, Patent Document 1 proposes a control support device that uses an electronic flight sheet, which is an electronic version of the paper flight sheet used in air traffic control operations.

JP 2016-134032 A

However, it is desirable to provide technology that reduces the workload placed on users who monitor monitored objects. Therefore, the present invention aims to provide technology that reduces the workload placed on users who monitor monitored objects.

In order to solve the above problem, according to one aspect of the present invention, an information processing device is provided, comprising: a monitored object detection unit that detects the position of a monitored object in a captured image or screen; a monitored area calculation unit that calculates the position of a user's monitored area in the captured image or screen based on information about the user's point of gaze generated by a gaze sensor ; and a first ancillary information placement unit that determines the display position of first ancillary information, which is information related to the monitored object, at a position that corresponds to the position of the monitored object and the position of the monitored area and is close to the position of the monitored area .

The information processing device may include an emphasis line calculation unit that calculates the position of an emphasis line, which is at least a part of a line that passes through the position of the monitored object and the position of the focused monitoring area, and the first incidental information placement unit may determine the display position of the first incidental information based on the position of the emphasis line.

The information processing device may also include a second auxiliary information placement unit that determines a display position of second auxiliary information, which is information related to the monitored object and different from the first auxiliary information, at a position near the position of the monitored object.

The second incidental information placement unit may determine a display position of the second incidental information at a position based on a position of the emphasis line and close to a position of the monitored object .

The second incidental information placement unit may determine the display position of the second incidental information based on whether the user's point of gaze is within or near the image area of the monitored object in the captured image or on the screen.

The information processing device may include a transmission control unit that transmits data in which the first incidental information is arranged at the display position of the first incidental information determined by the first incidental information arrangement unit to the presentation control device via a communication unit.

According to another aspect of the present invention, an information processing method is provided, comprising: detecting the position of a monitored object in a captured image or screen; calculating the position of a user's monitoring focus area in the captured image based on information about the user's point of gaze generated by a gaze sensor ; and determining a display position of first ancillary information, which is information related to the monitored object, at a position corresponding to the position of the monitored object and the position of the monitoring focus area and close to the position of the monitoring focus area .

According to another aspect of the present invention, a program is provided that causes a computer to function as an information processing device comprising: a monitored object detection unit that detects the position of a monitored object in a captured image or screen; a monitored area calculation unit that calculates the position of a user's monitored area in the captured image or screen based on information about the user's point of gaze generated by a gaze sensor ; and a first ancillary information placement unit that determines the display position of first ancillary information, which is information related to the monitored object, at a position that corresponds to the position of the monitored object and the position of the monitored area and is close to the position of the monitored area.

According to another aspect of the present invention, a system is provided which includes an information processing device including a monitored object detection unit that detects the position of a monitored object in a captured image or screen, a monitored area calculation unit that calculates the position of a user's monitored area in the captured image or screen based on information about the user's point of gaze generated by a gaze sensor , and a first auxiliary information placement unit that determines a display position of first auxiliary information, which is information related to the monitored object, at a position corresponding to the position of the monitored object and the position of the monitored area and close to the position of the monitored area, and a presentation control device that includes a control unit that controls the display unit so that the first auxiliary information is displayed at the display position.

As described above, the present invention provides technology that reduces the workload of users monitoring monitored objects.

1 is an explanatory diagram showing an example of a schematic configuration of an air traffic control assistance system according to an embodiment of the present invention; FIG. 2 is a block diagram showing an example of the hardware configuration of an air traffic control support server, an airport camera, an air traffic control information presentation device, and an aviation information management system according to the embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of the traffic control information presentation device according to the embodiment. 10 is an explanatory diagram illustrating an example of air traffic control support information displayed on a screen (display unit) of the air traffic control information presentation device. FIG. 10 is an explanatory diagram for explaining an example of the display positions of the first incidental information and the second incidental information on the screen. FIG. FIG. 2 is a block diagram showing an example of a functional configuration of a traffic control assistance server 100 according to the embodiment. FIG. 2 is an explanatory diagram showing an example of an operation flow of the air traffic control assistance system according to the embodiment. FIG. 2 is an explanatory diagram showing an example of an operation flow of the air traffic control assistance system according to the embodiment.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configuration will be designated by the same reference numerals, and redundant explanations will be omitted.

In addition, in this specification and drawings, multiple components having substantially the same or similar functional configurations are distinguished by using the same reference numeral followed by a different alphabet. However, if there is no need to particularly distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numeral will be used.

(0. Overview)
Air traffic control is the process by which a control agency gives instructions on flight methods and other matters to ensure the safe operation of aircraft (Kojien, 6th edition), and it is a task that carries the risk of even a small human error leading to a serious aircraft accident.

Air traffic controllers (hereafter referred to simply as "controllers") responsible for air traffic control duties must simultaneously monitor the status of, in some cases, 10 or more aircraft, and issue instructions and communications with the right timing and content to the pilots of each aircraft and other controllers. This work is known to be mentally demanding. As a result, controllers sometimes give incorrect instructions due to human error or fall asleep due to fatigue, resulting in aircraft accidents and serious incidents. The workload of these controllers and the resulting risk of human error and accidents and incidents must be addressed as issues.

By the way, with the development of information and communications technology, there are signs of two major changes in the environment in which air traffic controllers work.

The first change is the digitization of flight strips used by air traffic control, i.e., the introduction of an electronic flight strip system. Examples of electronic flight strips are described in Patent Document 1 (JP 2016-134032) and elsewhere.

The second change is the realization of remote airport control systems (hereafter referred to as remote airport control) using remote towers, etc. (e.g., Electronic Navigation Research Institute, Research on the Practical Application of Remote Airport Operations Support Systems, https://www.enri.go.jp/research/kenkyu/M-17-01.html). In a remote airport control system, imaging devices such as wide-angle cameras are installed at airports, and video imagery of the airport captured by these cameras is presented to air traffic controllers located far from the airport via low-latency network communications. The controllers view video imagery including airport runways and aircraft displayed on large display devices, and remotely communicate with pilots of aircraft at remote airports to give instructions and communicate with them.

The aforementioned remote airport control functions using electronic flight sheets and video communication are expected to be widely adopted in future airport control systems.

Embodiments of the present invention apply the technical elements of the "electronic flight sheet" and "remote airport video communication" described above to realize a new remote airport control support system that can reduce the workload of controllers.

One factor in a controller's workload is the high cognitive and memory processing load. Controllers must simultaneously grasp the status of multiple aircraft, look at paper flight sheets on their desks or electronic flight sheets displayed on display devices, memorize the information, and mentally link the status of the multiple aircraft with the information on the flight sheets (call signs, aircraft types, estimated departure and arrival times, etc.).

At the same time, controllers must visually monitor changes in the status of aircraft scattered at distances from one another within the airport, i.e., detect visual changes within the wide field of view. Through these and other tasks, controllers are constantly checking to see if the situation within the entire airport is changing according to plan.

Here, the former action of linking and memorizing the status of each aircraft with the information on the flight schedule can be reduced by using AR (Augmented Reality) technology to visually link and display video images of the aircraft with the electronic flight schedule information corresponding to each aircraft. In other words, by linking and displaying video images of the aircraft with the electronic flight schedule information corresponding to each aircraft on the display, the action of memorizing the linking becomes unnecessary.

Furthermore, the latter action of detecting visual changes within the wide-field information of an airport can reduce the mental burden on the controller by presenting AR information on the electronic flight schedule, which changes in response to visual changes within the wide-field information, near the controller's key monitoring area (described below). In other words, by presenting AR information on the electronic flight schedule near the controller's key monitoring area (described below), the controller's visual search range is reduced, allowing the controller to notice changes in the situation of the object of monitoring with a low visual burden.

Here, we will also consider situations in which multiple people use the air traffic control support system according to an embodiment of the present invention. Air traffic control operations are often performed collaboratively by multiple air traffic controllers. For example, at airports with heavy air traffic, such as hub airports, there are cases in which air traffic controllers with multiple roles, such as airport controller, ground controller, control approval transmitter, assistant controller, and general controller, work together for each runway. In such cases of air traffic control performed collaboratively by multiple people, for example, two controllers, one controller acts as airport controller and ground controller, and the other controller acts as assistant controller or general controller, coordinating with other related organizations, we will consider the following case.

Since the first controller plays the role of airport controller or ground controller, it is useful to utilize the function of presenting AR information on the airport video near the first controller's priority monitoring area (described below). On the other hand, the second controller plays the role of assistant controller or supervising controller, and may intermittently view unspecified locations on the airport video in accordance with liaison and coordination work with other related organizations. In other words, since the priority monitoring area (described below) of the first controller is different from the priority monitoring area of the second controller, in this case, the location of the AR information presented near the first controller's priority monitoring area (described below) is not particularly useful for the second controller, and it is suggested that presenting AR information near the video images of each aircraft would be more useful due to the location of the AR information.

In summary, embodiments of the present invention propose technology for realizing a new remote airport control support system, which detects video images of monitored objects such as aircraft on video images of a remote airport, links the detected video images of the monitored objects with related information such as electronic flight sheets and presents AR information, and further controls the presentation position of the AR information so as to reduce the visual search range of the first controller. In addition, embodiments of the present invention propose technology for realizing a remote airport control support system, which controls the presentation of AR information having content equivalent to the AR information for the second controller so that it is also presented near the video images of the monitored objects.

In the embodiments of the present invention, an aircraft will be used as an example of a monitored object. The aircraft may be a manned aircraft or an unmanned aircraft (a so-called unmanned aircraft). However, the monitored object may also be an object other than an aircraft. For example, the monitored object may be a person, an animal other than a person, or an inanimate object (such as a ship, vehicle, or robot).

Furthermore, in the embodiments of the present invention, the occupation of a user monitoring a monitored object is described as an air traffic controller. However, the user monitoring a monitored object does not have to be an air traffic controller. For example, the occupation of a user monitoring a monitored object may be changed as appropriate depending on the type of monitored object.

(1. Description of the configuration)
First, referring to FIG. 1 , a schematic configuration of an air traffic control support system according to an embodiment of the present invention will be described. FIG. 1 is an explanatory diagram showing an example of the schematic configuration of an air traffic control support system according to the present embodiment. Referring to FIG. 1 , the air traffic control support system includes an air traffic control support server 100, an airport camera 200, a line-of-sight sensor 300, an air traffic control information presentation device 400, an aviation information management system 500, and a LAN 99. FIG. 1 also illustrates an airport 10, a remote control room 30, an aircraft 60, and an air traffic controller 90 as space, object, and person elements for explaining the air traffic control support system. In this embodiment, the airport camera 200 and the aircraft 60 are located within or around the airport 10, and the line-of-sight sensor 300, the air traffic control information presentation device 400, and the air traffic controller 90 are located within the remote control room 30.

Figure 2 is a block diagram showing an example of the hardware configuration of the air traffic control support server 100, airport camera 200, air traffic control information presentation device 400, and aviation information management system 500 according to this embodiment (hereinafter, these may be referred to as "devices according to this embodiment" without distinguishing between them). Note that not all of the above devices need to be equipped with all of the hardware configuration described below (for example, the air traffic control support server 100 does not need to be directly equipped with sensors), and each device may be equipped with an appropriately limited number of hardware modules that can realize the functional configuration of each device described below.

Referring to FIG. 2, the device according to this embodiment includes a bus 801, a CPU (Central Processing Unit) 803, a ROM (Read Only Memory) 805, a RAM (Random Access Memory) 807, a storage device 809, a communication interface 811, a sensor 813, an input device 815, a display device 817, and a speaker 819. The CPU 803 executes various processes in the device according to this embodiment.

The ROM 805 also stores programs and data for causing the CPU 803 to execute processing in the device according to this embodiment. The RAM 807 also temporarily stores programs and data when the CPU 803 is executing processing. The bus 801 interconnects the CPU 803, ROM 805, and RAM 807. The bus 801 is also connected to a storage device 809, a communication interface 811, a sensor 813, an input device 815, a display device 817, and a speaker 819. The bus 801 includes, for example, multiple types of buses. As one example, the bus 801 includes a high-speed bus connecting the CPU 803, ROM 805, and RAM 807, and one or more other buses slower than the high-speed bus.

Storage device 809 stores data to be saved temporarily or permanently within the device of this embodiment. Storage device 809 may be, for example, a magnetic storage device such as a hard disk, or may be nonvolatile memory such as EEPROM (Electrically Erasable and Programmable Read Only Memory), flash memory, MRAM (Magnetoresistive Random Access Memory), FeRAM (Ferroelectric Random Access Memory), and PRAM (Phase Change Random Access Memory).

The communication interface 811 is a communication means provided in the device according to this embodiment, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication, in which case it may include, for example, a communication antenna, an RF circuit, and other circuits for communication processing. The communication interface 811 may also be an interface for wired communication, in which case it may include, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing.

The sensor 813 is, for example, a camera, microphone, biosensor, other sensor, or a combination thereof. The camera captures an image of a subject and includes, for example, an optical system, an image sensor, and an image processing circuit. The microphone collects ambient sound, converts the sound into an electrical signal, and then converts the electrical signal into digital data. The input device 815 is, for example, a touch panel, a mouse, or a gaze detection device. The display device 817 displays an output image (i.e., a display screen) from the device of this embodiment and can be realized using, for example, a liquid crystal display, an organic light-emitting diode (OLED), a cathode ray tube (CRT), or the like. The speaker 819 outputs sound, converting digital data into an electrical signal and then converting the electrical signal into sound.

First, we will explain an example of the functional configuration of the "airport camera 200" related to this embodiment.

The airport camera 200 is a network camera that captures images of the airport 10 and aircraft 60 and outputs and transmits the generated video data to the air traffic control support server 100 via LAN 99. To perform the above processing, the airport camera 200 may include a communication unit, memory unit, control unit, and other functional components in addition to an imaging unit (sensor unit), although these are not shown. The airport camera 200 may be a panoramic camera capable of capturing images with an ultra-wide angle of view of 180° or 360°, or a PTZ (Pan-Tilt-Zoom) camera with pan, tilt, and zoom functions. The airport camera 200 may be implemented using a sensor 813.

Next, we will explain an example of the functional configuration of the "gaze sensor 300" according to this embodiment.

The gaze sensor 300 generates "point of gaze" information indicating where on the screen (display) of the air traffic control information presentation device 400 (described below) the air traffic control officer 90 in the remote control room 30 is looking, and transmits this information to the air traffic control support server 100 via the LAN 99. The point of gaze information includes, for example, position information indicated by two-dimensional (X, Y) coordinates on the screen (display) of the air traffic control information presentation device 400 and information on the time the position information was generated.

Methods of gaze sensing that use special hardware devices include the corneal reflex method using near-infrared light, the scleral reflex method, electrooculography, and the search coil method, as described in Non-Patent Document 1 (Hashimura Masaru, Iizuka Hiromi, Li Jun, 2015, "Measurement of Eye Movements," Ergonomics, 51(6), pp. 406-410).

Furthermore, gaze sensing methods that combine general camera devices with image processing techniques are also known. Methods that combine general camera devices with image processing techniques include appearance-based methods and machine learning-based estimation methods, as described in Non-Patent Document 2 (Non-Patent Document 2: Yusuke Kanno, 2020, "Research Trends in Gaze Measurement and Estimation Technology," Journal of the Institute of Image Information and Television Engineers, 74(3), pp. 495-500).

In an embodiment of the present invention, any of the above gaze sensing methods may be used as long as it can generate information on the gaze point of the air traffic controller 90A. Furthermore, there are two types of gaze sensors: environmentally installed and head-mounted. In this embodiment, the explanation will be centered on an environmentally installed gaze sensor (for example, installed near the screen of the air traffic control information presentation device 400, described below). However, the gaze sensor may also be head-mounted. Furthermore, the environmentally installed gaze sensor 300 may have information input in advance by the system administrator regarding the relative positional relationship in three-dimensional space between itself and the plane of the screen (display unit) of the air traffic control information presentation device 400.

Note that depending on the biological shape of the eyes and the operating environment of the gaze sensor, it may not be possible to measure the gaze with high accuracy. Alternatively, even if the gaze can be measured, there is a method to improve the estimation accuracy of gaze information by combining the gaze measured by the gaze sensor with facial direction information to estimate the gaze. Therefore, the gaze sensor 300 may have a function to estimate facial direction as alternative information to the gaze of the controller 90. One example of a facial direction sensing method is known, as described in Non-Patent Document 3, which estimates facial direction using feature point information of a facial image obtained from a general camera device (Non-Patent Document 3: Yamazaki Akito, Pongsathornraksilansak, and Otake Motoki, 2017, "Extracting the Gaze Area by Estimating the Driver's Facial Direction Using an In-Vehicle Camera," Transactions of the Society of Automotive Engineers of Japan, 48(5), pp. 1113-1119).

The gaze sensor 300 may calculate position information of the intersection between the estimated face direction, for example, a three-dimensional vector extending in the direction of the front of the face, and the plane of the screen (display unit) of the air traffic control information presentation device 400, and replace the position information of the intersection with the above-mentioned gaze point information. Note that the gaze sensor 300 may be implemented by sensor 813.

Next, we will explain an example of the functional configuration of the "aviation information management system 500" related to this embodiment.

The aviation information management system 500 has a set of various information data necessary for the airfield control operations of the controller 90, and may transmit some of this data to the airfield control support server 100 in response to a request from the airfield control support server 100. For example, the set of various information data necessary for the airfield control operations may be obtained by the aviation information management system 500 from a system or database that handles the data.

The set of various information data required for the airport control operations may include data handled by the systems and databases described below. Specifically, the systems may be FDP (Flight Data Processing System), RDP (Radar Data Processing System), MLAT (Multilateration), etc. Furthermore, the databases may be aircraft information databases, sector information databases, etc.

Here, the FDP handles operational information for each aircraft within the airport 10 or the surrounding airspace controlled by the controller 90 (for example, each aircraft's destination, planned flight distance, spot (parking) location, takeoff location, scheduled spot departure time, scheduled takeoff time, etc.). The RDP handles information such as the flight number and altitude of each aircraft. The MLAT handles the three-dimensional position and identification information (for example, call sign) of each aircraft 60. For example, the three-dimensional position of each aircraft 60 may be latitude, longitude, and altitude information, or three-dimensional relative position information between the airport camera 200 and each aircraft 60.

The aircraft information database handles data such as the name of each aircraft 60, air routes, and flight performance information for each aircraft model. The sector information database handles data on air routes within the controlled airspace (sector), as well as data indicating the positions of FIXs such as junctions, target points, and radio beacons.

In particular, the aviation information management system 500 may transmit to the air traffic control support server 100 the latitude, longitude, and altitude information of each aircraft 60 within the airport 10 or surrounding airspace that is under the control of the controller 90, or the three-dimensional relative position information between the airport camera 200 and each aircraft 60, in association with the identification information and operation information of each aircraft 60. The latitude, longitude, and altitude information and three-dimensional relative position information of each aircraft 60 may be generated using the MLAT described above.

Next, an example of the functional configuration of the "traffic control information presentation device 400" according to this embodiment will be described with reference to Figure 3. The traffic control information presentation device 400 includes a communication unit 410, a storage unit 420, a control unit 430, and a display unit 450. Note that the traffic control information presentation device 400 can function as an example of a presentation control device.

The communication unit 410 communicates with other devices. For example, the communication unit 410 is directly connected to the LAN 99 and communicates with the air traffic control support server 100. More specifically, the communication unit 410 receives air traffic control support information data, which is generated by the air traffic control support server 100 (described below) and which includes captured video images of the airport 10 with additional information added in a manner characteristic of an embodiment of the present invention. The communication unit 410 can be implemented by the communication interface 811.

The memory unit 420 stores programs and data for the operation of the traffic control information presentation device 400. The memory unit 420 may be implemented by the storage device 809.

The control unit 430 provides various functions of the traffic control information presentation device 400. The control unit 430 can be implemented by a CPU 803, a ROM 805, and a RAM 807.

The display unit 450 presents information to the controller 90. For example, the display unit 450 has a screen, and displays the air traffic control support information on the screen based on the air traffic control support information data received by the communication unit 410, and presents it to the controller 90. Note that the display unit 450 can be realized, for example, by the display device 823.

Figure 4 is an explanatory diagram illustrating an example of air traffic control support information A40 displayed on the screen (display unit) of the air traffic control information presentation device 400. Note that the dashed lines and center of gravity shown in Figure 4 are graphic elements used for explaining the present invention, and may not actually be displayed on the screen (display unit) of the air traffic control information presentation device 400.

Referring to Figure 4, there are shown a screen A40 in which air traffic control support information according to an embodiment of the present invention is displayed on a video image of an airport, a video image A401 of airport 10, an image A441 of aircraft 60, a monitored object detection area A442, a monitoring priority area A451, an example of a gaze point A452, a center of gravity A461 of the monitored object detection area, a center of gravity A462 of the monitoring priority area, a priority line A463, first auxiliary information A471, second auxiliary information A481, and a priority line A483 (those shown in multiple figures are numbered A, B, C, etc.).

As an overview of Figure 4, first ancillary information A471 is visually linked to an image A441 of each aircraft 60 and displayed in AR, and the first ancillary information A471 is further displayed in a position near the controller 90A's key monitoring area A451. A detailed explanation of each of the above illustrated elements will be provided in the explanation of the air traffic control support server 100, which will be described later.

In this embodiment of the present invention, it is primarily assumed that the controller 90 views a captured video image A401 of the airport 10 displayed on screen A40. However, the captured video image A401 of the airport 10 does not have to be displayed. That is, the controller 90 may be located within the airport 10, which is in real space (not remote), or in a control tower located nearby, and view the airport 10 directly. In this case, the display unit 450 is configured as a transparent display, and the controller 90 views the airport 10, which is in real space, through the transparent display instead of the airport 10 displayed on screen A40. The supplementary information is then superimposed on the airport 10, which is in real space, instead of being superimposed on the airport 10 displayed on screen A40.

Figure 5 is an explanatory diagram illustrating an example of the display positions of the first incidental information A471 and the second incidental information A481 on the screen A40. Referring to Figure 5, similar to Figure 4, the screen A40, a captured video image A401 of the airport 10, an image A441 of the aircraft 60, a monitored object detection area A442, a monitoring priority area A451, an example of a gaze point A452, a center of gravity A461 of the monitored object detection area, a center of gravity A462 of the monitoring priority area, the first incidental information A471, and the second incidental information A481 are shown.

Furthermore, referring to Figure 5, the center of gravity A472 of the first auxiliary information area and the center of gravity A482 of the second auxiliary information area are shown. Also shown is the center of gravity A474 of the first auxiliary information area.

Next, an example of the functional configuration of the "control assistance server 100" according to this embodiment will be described with reference to Figure 6. Note that the control assistance server 100 can function as an example of an information processing device.

The air traffic control support server 100 acquires and receives, via LAN 99, video image data of the airport 10 from the airport camera 200, information on the gaze point of the air traffic controller 90A from the line-of-sight sensor 300, and three-dimensional position information, identification information, and operation information of each aircraft 60 from the aviation information management system 500. The air traffic control support server 100 then generates data for air traffic control support information (information in which supplementary information has been added to the video image of the airport 10 using a method characteristic of this embodiment of the present invention) and transmits it to the air traffic control information presentation device 400.

Figure 6 is a block diagram showing an example of the functional configuration of the traffic control assistance server 100 according to this embodiment. Referring to Figure 6, the traffic control assistance server 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

The communication unit 110 communicates with other devices. For example, the communication unit 110 is directly connected to the LAN 99 and communicates with the airport camera 200, the line-of-sight sensor 300, the aviation information management system 500, etc. The communication unit 110 may be implemented by the communication interface 811.

The memory unit 120 stores programs and data for the operation of the air traffic control support server 100. The memory unit 120 may be implemented by the storage device 809.

The control unit 130 provides various functions of the control support server 100. The control unit 130 includes a monitored object detection unit 140, a monitoring focus area calculation unit 150, a priority line calculation unit 160, a first incidental information placement unit 170, a second incidental information placement unit 180, and a transmission control unit 190. The control unit 130 can be implemented by a CPU 803, a ROM 805, and a RAM 807.

The monitored object detection unit 140 detects the position of each aircraft 60 in the captured video based on the captured video of the airport 10 received from the airport camera 200. More specifically, the monitored object detection unit 140 identifies the display position of the image of each aircraft 60 shown in the captured video based on the captured video or the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500.

The monitored object detection unit 140 then links the identification information and flight information corresponding to the aircraft 60 with the position of the aircraft 60 for each aircraft 60, and stores the linking information in the memory unit 120. That is, the monitored object detection unit 140 calculates the on-screen area position of the monitored object detection area A442 shown in FIG. 4 above, and performs a process of linking the on-screen area position of the monitored object detection area A442 with the identification information and flight information of the aircraft shown in the monitored object detection area A442.

The display position of the image of the aircraft 60 may be determined by object detection processing of each aircraft image. Note that object detection processing is processing to determine the location of objects of a certain category in an image. Specific methods for object detection processing may include any known method such as YOLO. For example, YOLO is described in Redmon, J. et al., 2016, "You Only Look Once: Unified, real-time object detection," Proceedings of CVPR '16, pp. 779-788.

Alternatively, the display position of the image of the aircraft 60 may be determined by a process of calculating the two-dimensional position of the image of each aircraft 60 by projecting the three-dimensional position of each aircraft 60 onto a two-dimensional airport-captured video image based on the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500. The camera parameter information required for projective transformation of the three-dimensional position into two-dimensional space may be determined in advance using a known camera calibration method. As described above, the air traffic controller 90 may also view the airport 10, which is real space, through a transmissive display. At this time, the monitored object detection unit 140 detects the position of each aircraft 60 on screen A40. More specifically, the monitored object detection unit 140 detects the position of each aircraft 60 on screen A40 based on the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500. For example, the position of each aircraft 60 on screen A40 may be determined by a process of determining the two-dimensional position of each aircraft 60 on screen A40 by projecting the three-dimensional position of each aircraft 60 onto screen A40 based on the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500.

Here, we will assume a case where a method is used to identify the display position of the image of the aircraft 60 by combining object detection processing and projective transformation. Specifically, the monitored object detection unit 140 calculates a two-dimensional position by projecting the three-dimensional position onto the airport-captured video image using camera parameter information from the airport camera 200, based on the three-dimensional position information of each aircraft 60 obtained from the aviation information management system 500. The monitored object detection unit 140 then performs object detection processing on the aircraft image around the two-dimensional position on the airport-captured video image. The monitored object detection unit 140 performs labeling processing on the detected area of the aircraft image to temporarily assign label information, and then performs processing to link the aircraft's identification information and operation information with the area of the aircraft image to which the label information has been assigned.

Through the above processing, for example, the monitored object detection area A442A in Figure 4, the operation information of the aircraft 60 that is the monitored object related to the monitored object detection area A442A, and the example identification information (e.g., call sign) "OKI001" of the aircraft 60 are linked.

In Figure 4, the monitored object detection area A442 is represented as a "rectangle" circumscribing the area of the aircraft image, as an example. However, the shape of the monitored object detection area A442 may be any other polygon that includes the area of the aircraft image, or may be a curved closed area.

It is also anticipated that the above object detection process may be difficult. In such cases, the monitored object detection unit 140 may detect the two-dimensional position of each aircraft 60 obtained from the aviation information management system 500 projected onto the airport-captured video as the position of the monitored object detection area A442. Examples of cases in which object detection process is difficult include when the aircraft 60 is located far from the airport 10 and is therefore captured at a small size (for example, one pixel) in the airport-captured video, or when the shooting environment is poor due to weather or other reasons.

The fact that the aircraft 60 is captured at a small size in the airport video may be detected by the distance between the airport camera 200 and the aircraft 60 being greater than a threshold, calculated based on the three-dimensional relative position information between the airport camera 200 and the aircraft 60. An unfavorable shooting environment may also be detected by the fact that the shooting time is a predetermined time period (for example, nighttime), or that the weather at the location where the airport camera 200 is installed is predetermined weather (for example, rainfall, snowfall, etc.).

The monitoring priority area calculation unit 150 calculates the position of the monitoring priority area of the controller 90A in the captured video image based on the information of the controller 90A's gaze point generated by the aforementioned line-of-sight sensor 300. More specifically, the monitoring priority area calculation unit 150 calculates the position of the monitoring priority area of the controller 90A in the captured video image A401 (Figure 4) of the airport 10 based on the information of the controller 90A's gaze point. For example, the monitoring priority area calculation unit 150 calculates the position of the monitoring priority area from the accumulated information of the controller 90A's gaze point within a specified time period.

The monitoring priority area is a portion of the captured video of the airport that the controller considers important (e.g., a portion of the runway video). It is a two-dimensional area in the captured video A401 that the controller 90A monitors with a focus, i.e., a two-dimensional area in the captured video A401 where the controller 90A's gaze point is concentrated. (For example, if the shape of the captured video A401 (i.e., the shape of the screen A40) is three-dimensional, such as when the display unit of the air traffic control information presentation device 400 is a curved display, the monitoring priority area may be a three-dimensional area.) This is exemplified in Figure 4 as monitoring priority area A451. As mentioned above, the controller 90 may view the airport 10, which is real space, through a see-through display. In this case, the monitoring priority area calculation unit 150 calculates the position of the controller 90A's monitoring priority area on the screen A40. More specifically, the monitoring priority area calculation unit 150 calculates the position of the monitoring priority area of the controller 90A on the screen A40 based on information about the gaze point of the controller 90A.

The monitoring priority area may be, for example, a "convex hull" that includes the controller's 90 gaze points within a specified time period. In this case, the algorithm for calculating the convex hull may be a divide-and-conquer method, a Graham scan, or any other known method.

Furthermore, in order to reduce the amount of calculation, the gaze points to be subject to the convex hull calculation do not need to be all gaze points within a specified time. For example, the monitoring priority area calculation unit 150 may divide the screen A40 into any resolution, cumulatively calculate the number of gaze points and gaze dwell time contained in each element area of that resolution, and perform filtering processing of the element areas to be subject to the convex hull calculation based on the cumulative calculation results. For example, the monitoring priority area calculation unit 150 may exclude element areas with the number of gaze points equal to or less than N (N is a natural number) from the convex hull calculation. Note that the conditions and thresholds for the filtering processing may be set in advance by an administrator of the system according to an embodiment of the present invention.

The priority line calculation unit 160 calculates the position of a line corresponding to the position of the aircraft 60 detected by the monitored object detection unit 140 and the position of the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150 as the position of the priority line. More specifically, the priority line calculation unit 160 calculates the position of a line corresponding to the position of the aircraft 60 in the captured video A401 and the position of the monitoring priority area A451 in the captured video A401 as the position of the priority line.

For example, the priority line calculation unit 160 calculates the center of gravity A461 of the monitored object detection area from the monitored object detection area A442 calculated by the monitored object detection unit 140. The priority line calculation unit 160 also calculates the center of gravity A462 of the monitoring focus area from the monitoring focus area A451 calculated by the monitoring focus area calculation unit 150. The priority line calculation unit 160 then calculates the position of the priority line A463, which is at least a portion of a line passing through the two center of gravity positions, in the captured video image A401. As described above, the air traffic controller 90 may view the airport 10, which is a real space, through a transmissive display. In this case, the priority line calculation unit 160 calculates the position of a line corresponding to the position of the aircraft 60 on screen A40 and the position of the monitoring focus area A451 on screen A40 as the position of the priority line on screen A40.

The center of gravity of the monitored object detection area A442 (e.g., a rectangle) and the center of gravity of the monitoring priority area A451 (e.g., a convex hull polygon) can be calculated, for example, by dividing the polygon that is the shape of the area into multiple triangles, determining the center of gravity of each triangle, and then calculating a weighted average of the center of gravity positions of each triangle. However, the centers of gravity of these areas may also be calculated using any other known method. Note that while the center of gravity point A461 of the monitored object detection area, the center of gravity point A462 of the monitoring priority area, and the priority line A463 are shown in Figure 4 above for illustrative purposes, the center of gravity and priority line may not be displayed on the screen A40 of the control information presentation device 400 during operation.

The first incidental information placement unit 170 determines the display position of the first incidental information A471, which is information related to the aircraft 60, at a position corresponding to the position of the aircraft 60 and the position of the monitoring focus area A451. More specifically, the first incidental information placement unit 170 determines the display position of the first incidental information A471 based on the position of the focus line A463, which corresponds to the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area.

Referring to Figure 4, as an example, the first incidental information A471A includes aircraft identification information "OKI001," as well as examples of flight information such as spot position "SP01," takeoff position "RWY36," scheduled spot departure time "13:30," and scheduled takeoff time "13:45." However, the information included in the first incidental information A471 may be any information related to aircraft 60.

The display position of the first auxiliary information A471 is preferably determined to be a position linked to the image of the aircraft 60 associated with the first auxiliary information A471 (i.e., the monitored object detection area A442). This allows the first auxiliary information A471 and the image of the aircraft 60 associated with the first auxiliary information A471 to be visually linked and displayed, thereby reducing the mental burden on the controller 90A. More specifically, as shown in FIG. 4, the center of gravity A472 of the area of the first auxiliary information may be determined on the priority line A463, which is at least a part of the line passing through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring priority area.

Furthermore, as shown in FIG. 4, it is preferable that the display position of the first incidental information A471 be determined to be a position adjacent to the key monitoring area A451 (including a "circumscribing position"). This reduces the visual search range of the controller 90A for the first incidental information A471 based on the key monitoring area A451, thereby reducing the mental burden on the controller 90A.

The first incidental information placement unit 170 places the first incidental information A471 at the determined display position. More specifically, the first incidental information placement unit 170 places the first incidental information A471 at the position in the captured video A401 that has been determined as the display position of the first incidental information A471. In this manner, air traffic control support information data can be generated based on the placement of the first incidental information A471.

The second incidental information placement unit 180 determines the display position of the second incidental information A481, which is information related to the aircraft 60, for each aircraft 60 based on the position of the aircraft 60. More specifically, the second incidental information placement unit 180 determines the display position of the second incidental information A481 for each aircraft 60 based on the position of the monitored object detection area A442. For example, it is preferable that the display position of the second incidental information A481 be determined to be a position close to the monitored object detection area A442 (including a "circumscribing position"). This allows the controller 90A or 90B to move their line of sight only a short distance from viewing the image of the aircraft 60 to viewing the second incidental information A481 related to that aircraft 60, thereby reducing the mental burden on the controller 90A or 90B.

Referring to FIG. 4, as an example, the second incidental information A481A includes information with the same content as the information included in the first incidental information A471A. However, the information included in the second incidental information A481A does not have to be the same as the information included in the first incidental information A471A. The information included in the second incidental information A481A may be any information related to the aircraft 60.

In addition, an emphasis line A483 indicating the link between the second incidental information A481 (or its center of gravity A482) and the image A441 (or its center of gravity A461) of each aircraft 60 may be displayed. This makes it possible to more clearly visualize the relationship between the aircraft and the corresponding incidental information. In more detail, the emphasis line A483 may be at least part of a line passing through the center of gravity A482 of the area of the second incidental information and the center of gravity A461 of the monitored object detection area.

Furthermore, as shown in the example in FIG. 5, the center of gravity A482 of the second incidental information area may be determined on a line passing through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area. In this case, the priority line A483 and the priority line A463 are arranged parallel to each other. As a result, when controller 90A (first controller) moves his/her gaze from a position close to the center of gravity A462 of the monitoring focus area to the image A441 of each aircraft 60, the second incidental information A481 is located at the end of the extension of that line, which may make it easier to view the second incidental information A481 corresponding to the aircraft 60 together with the aircraft 60. On the other hand, if the second incidental information A481 were located in a location perpendicular to the priority line A463, the direction of gaze movement would have to be bent so as to be perpendicular, which could hinder smooth gaze flow.

The second incidental information placement unit 180 places the second incidental information A481 at the determined display position. More specifically, the second incidental information placement unit 180 places the second incidental information A481 at the position in the captured video A401 that has been determined as the display position of the second incidental information A481. In this manner, air traffic control support information data can be generated based on the placement of the second incidental information A481.

The transmission control unit 190 transmits the generated air traffic control support information data to the air traffic control information presentation device 400 via the communication unit 110, causing the air traffic control support information to be presented in an AR-like format on the screen A40 of the air traffic control information presentation device 400. Specifically, the center of gravity A472 of the first incidental information area is on the priority line A463 calculated by the priority line calculation unit 160, and the first incidental information A471 is displayed in a position close to (including a "circumscribing position") the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150.

As mentioned above, the priority line A463 does not have to actually be displayed on the screen A40 of the air traffic control information presentation device 400. However, as illustrated in FIG. 4, it is preferable that the first auxiliary information A471 and the image A441 of the aircraft 60 are presented on the screen A40 of the air traffic control information presentation device 400 in a manner that indicates a link between them. In the example of FIG. 5, the manner in which the link is indicated is expressed by displaying on the screen only the line segment that connects the monitored object detection area A442 (e.g., the center of gravity A461 of the monitored object detection area) and the first auxiliary information A471 (e.g., the center of gravity A472 of the area of the first auxiliary information) on the line that passes through the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring focus area.

In the conventional method of presenting only video footage of the airport to the controller, the controller 90 must frequently make large eye movements (visual search) between the numerous aircraft images (A441) scattered across the wide screen of the video footage of the airport and the paper flight sheet in front of them. Furthermore, this large eye movement creates a workload of having to memorize the linking information. For example, after looking at the paper flight sheet in front of them or on their desk (head down) and memorizing the contents, they must then make a large eye movement (head up) to the screen to search for the image of the aircraft corresponding to the flight sheet.

In contrast, in the method of presenting supplementary information that is a feature of the above-described embodiment of the present invention, a portion of the video image of the airport that is important to the controller 90 is designated as a priority monitoring area (A451), and information is presented so that supplementary information related to each aircraft (such as information on the electronic flight sheet) is concentrated in a position near the periphery of the priority monitoring area. This means that the range of the controller 90's line of sight search hardly extends beyond the priority monitoring area, and the visual workload can be reduced compared to conventional methods.

Furthermore, in the system according to an embodiment of the present invention, the incidental information and aircraft are presented in a manner linked by a priority line (A463), reducing the burden on the controller of having to memorize information from a paper flight chart at hand and search for the corresponding aircraft on the screen, as was done with conventional systems. Furthermore, the position of the incidental information automatically changes in response to changes in the display position of the aircraft image on the screen (for example, if an aircraft moves upward on the screen, the linked incidental information also moves upward accordingly). Therefore, even if the controller does not constantly directly monitor the group of aircraft images on the screen, he or she can notice changes in the situation of the monitored object simply by looking at the movement of the incidental information near the monitoring priority area. By utilizing the technology according to an embodiment of the present invention, controllers can obtain the benefits described above, for example.

(2. Description of Operation)
Next, an example of information processing operation according to this embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 and Fig. 8 are explanatory diagrams showing an example of the operation flow of the air traffic control assistance system according to this embodiment.

As shown in FIG. 7, the airport camera 200 captures video of the airport 10 and transmits the captured video data to the air traffic control support server 100 via the LAN 99 (S1101). The gaze sensor 300 also measures the gaze point of the controller 90A and transmits information about the gaze point of the controller 90A to the air traffic control support server 100 via the LAN 99 (S1103). Furthermore, the aviation information management system 500 transmits three-dimensional position information of each aircraft 60 to the air traffic control support server 100 via the LAN 99 (S1105). Additionally, the aviation information management system 500 transmits identification information and operation information of each aircraft 60 to the air traffic control support server 100 via the LAN 99.

In the air traffic control support server 100, the communication unit 110 receives captured video data of the airport 10 from the airport camera 200, receives information on the gaze point of the air traffic controller 90A from the line-of-sight sensor 300, and receives three-dimensional position information, identification information, and operation information of each aircraft 60 from the aviation information management system 500. Based on the captured video data of the airport 10, the monitored object detection unit 140 calculates the area of each aircraft 60 captured in the captured video data as the monitored object detection area A442 (S1107).

Next, the surveillance priority area calculation unit 150 calculates the surveillance priority area of the controller 90A in the captured video image A401 of the airport 10 based on information about the controller 90A's gaze point (S1109).

Next, the priority line calculation unit 160 calculates the center of gravity A461 of the monitored object detection area from the monitored object detection area A442. The priority line calculation unit 160 also calculates the center of gravity A462 of the monitoring priority area from the monitoring priority area A451 calculated by the monitoring priority area calculation unit 150. Furthermore, the priority line calculation unit 160 calculates the position in the captured video A401 of the priority line A463, which is at least a part of a line passing through the two center of gravity positions (S1111).

The first auxiliary information placement unit 170 determines the display position of the first auxiliary information A471 on an emphasis line A463 that corresponds to the center of gravity A461 of the monitored object detection area and the center of gravity A462 of the monitoring emphasis area.

Furthermore, it is preferable that the display position of the first incidental information A471 be determined to be a position adjacent to the key monitoring area A451 (including a "circumscribing position"). This reduces the range of gaze movement of the controller 90A to the first incidental information A471 based on the key monitoring area A451, thereby reducing the burden on the controller 90A. The first incidental information placement unit 170 places the first incidental information A471 at the determined display position (S1113).

The second incidental information placement unit 180 determines the display position of the second incidental information A481 to be a position adjacent to the monitored object detection area A442 (including a "circumscribing position"). This allows the controller 90A or 90B to move their line of sight only a short distance from viewing the image of the aircraft 60 to viewing the second incidental information A481 related to that aircraft 60, thereby reducing the mental burden on the controller 90A or 90B. The second incidental information placement unit 180 places the second incidental information A481 at the determined display position (S1114).

In this way, air traffic control support information data can be generated based on the arrangement of the first auxiliary information A471 and the second auxiliary information A481. The transmission control unit 190 transmits the generated air traffic control support information data to the air traffic control information presentation device 400 via the communication unit 110. In the air traffic control information presentation device 400, the control unit 430 controls the display unit 450 to display the air traffic control support information on the screen A40 (S1115). By displaying the emphasis line A463 on the screen, the first auxiliary information A471 and the image of the aircraft 60 related to the first auxiliary information A471 are visually linked and displayed, eliminating the need to link and memorize the correspondence between the two, thereby reducing the mental burden on the air traffic controller 90A.

(3. Various Modifications)
Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

(Variation 1)
In the above explanation, the representative points of the monitored object detection area and the monitoring focus area are taken as the centers of gravity (geometric centers) for determining the positions of the priority lines and supplementary information, but the representative points of these areas may be positions other than the centers of gravity.

Furthermore, for example, if multiple aircraft 60 are present in similar directions as viewed from the center of gravity A462 of the monitoring priority area (i.e., within a range where the angle formed between the center of gravity of the detected moving images of any two aircraft 60 and the center of gravity of the monitoring priority area falls within a specified angle), multiple pieces of incidental information will be displayed on the screen in similar positions, and in some cases the display ranges of multiple pieces of incidental information may overlap, impairing the visibility of the information. For this reason, the incidental information may be positioned in a position shifted from the priority line. Alternatively, the incidental information may be positioned in a position shifted away from the monitoring priority area based on a position circumscribing the monitoring priority area.

More specifically, it is assumed that multiple aircraft 60 are present within a range that falls within a predetermined angle based on the center of gravity A462 of the monitoring priority area. In such a case, the first auxiliary information placement unit 170 may determine the display position of the first auxiliary information A471 corresponding to at least one of the multiple aircraft 60 at a position shifted away from the first auxiliary information A471 corresponding to the other aircraft 60, or at a position shifted away from the monitoring priority area A451. This reduces the overlapping range between each piece of auxiliary information.

For example, in Figure 5, first incidental information A473B is shown at a position circumscribing the monitoring focus area A451, and first incidental information A473C is shown on the focus line A463C. However, using the center of gravity A462 of the monitoring focus area as a reference, the angle formed by connecting the center of gravity A461B of the monitored object detection area and the center of gravity A461C of the monitored object detection area falls within a range that falls within a specified angle.

In such a case, the first incidental information placement unit 170 may change the display position of the first incidental information A473B so that it is farther away from the focused monitoring area A451 (i.e., the first incidental information A473B may be changed to the first incidental information A471B). At this time, as shown in FIG. 5, the center of gravity A474B of the first incidental information area moves to the center of gravity A472B of the first incidental information area.

In such a case, the first incidental information placement unit 170 may change the display position of the first incidental information A473C (i.e., the first incidental information A473C may be changed to the first incidental information A471C) so that the first incidental information A473C is farther away from the other first incidental information A473B (in this case, since the first incidental information A473C is above the first incidental information A473B, the first incidental information A473C may be moved upward). Note that if the first incidental information A473C is below the first incidental information A473B, the display position of the first incidental information A473C may be moved downward. In this case, as shown in FIG. 5, the center of gravity A474C of the area of the first incidental information moves to the center of gravity A472C of the area of the first incidental information.

(Variation 2)
The second incidental information placement unit 180 may determine the display position of the second incidental information A481 corresponding to the aircraft 60 only when the air traffic controller 90A or the air traffic controller 90B is looking at the image A441 of the aircraft 60 (the point of gaze is within or near the area of the image A441). In other words, when the air traffic controller 90A or the air traffic controller 90B is not looking at the image A441 of the aircraft 60, the presentation of the second incidental information A481 corresponding to the aircraft 60 on the screen of the air traffic control information presentation device 400 may be stopped.

As a result, when controller 90A or controller 90B is not looking at image A441 of an aircraft 60, image A441 corresponding to that aircraft 60 is not displayed on the screen, thereby preventing the possibility that a large amount of additional information would be displayed on the screen when there are a large number of aircraft 60 taking off or landing, making visual search difficult.

For example, the second incidental information placement unit 180 can determine whether the controller 90A or 90B is looking at the image of the aircraft 60 based on information about the gaze point of the controller 90A or 90B measured by the gaze sensor 300, based on whether the gaze point of the controller 90B is on the image of the aircraft 60. For example, the second incidental information placement unit 180 can determine whether the controller 90A or 90B is looking at the image of the aircraft 60 based on whether the gaze point of the controller 90A or 90B has remained within the area of image A441 of the aircraft 60 for a predetermined period of time.

(Variation 3)
In the above description, the presence of the line-of-sight sensor 300 is essential for the operation of the monitoring priority area calculation unit 150. However, the monitoring priority area calculation unit 150 may operate in such a manner that the controller 90 or an administrator of the system according to the embodiment of the present invention inputs the setting information of the monitoring priority area into the system according to the embodiment of the present invention in advance.

Specifically, the monitoring priority area A451 in Figure 4 may be specified by, for example, the controller 90 specifying in advance an area of any position or range (e.g., a part of a runway image) on the captured video image A401 of the airport 10 using an input device (e.g., a pointing device such as a mouse). This allows the monitoring priority area calculation unit 150 and the system according to the embodiment of the present invention to operate even in situations or environments where the line-of-sight sensor 300 malfunctions and stops working, or where the line-of-sight sensor 300 is not provided.

(Variation 4)
In the above description, it has been mainly assumed that the system according to the embodiment of the present invention uses captured video images of the airport. However, instead of captured video images of the airport, the system according to the embodiment of the present invention may use captured still images of the airport. In other words, the system according to the embodiment of the present invention may use any captured images of the airport taken by the airport camera 200.

(4. Summary)
As described above, according to the embodiment of the present invention, when a controller performs airfield control work, an airfield control support system is realized that can reduce the workload of the controller by reducing the vast monitoring range that is required in the conventional airfield control work environment, such as looking directly at each aircraft or looking at a flight sheet on a desk, and the mental load required to link and memorize the status of each aircraft with the information on the flight sheet.

In addition, according to the present invention, when multiple controllers jointly perform airport control operations using a system according to an embodiment of the present invention, an airport control support system is realized that makes it easier for a second controller, different from the first controller who measures the key monitoring area, to view each aircraft along with the associated information.

REFERENCE SIGNS LIST 100 Air traffic control support server 110 Communication unit 120 Memory unit 130 Control unit 140 Monitoring target object detection unit 150 Monitoring focus area calculation unit 160 Priority line calculation unit 170 First incidental information placement unit 180 Second incidental information placement unit 190 Transmission control unit 200 Airport camera 30 Remote control room 300 Line of sight sensor 400 Air traffic control information presentation device 410 Communication unit 420 Memory unit 430 Control unit 450 Display unit 500 Aviation information management system 60 Aircraft 90 Controller

Claims (9)

a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen;
a monitoring focus area calculation unit that calculates the position of a monitoring focus area of the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor ;
a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area;
An information processing device comprising: The information processing device includes:
an emphasis line calculation unit that calculates the position of an emphasis line that is at least a part of a line that passes through the position of the monitored object and the position of the focused monitoring area;
the first auxiliary information placement unit determines a display position of the first auxiliary information based on the position of the emphasis line;
The information processing device according to claim 1 . The information processing device includes:
a second incidental information placement unit that determines a display position of second incidental information, which is information related to the object to be monitored and different from the first incidental information, at a position near the position of the object to be monitored;
The information processing device according to claim 2 . the second incidental information placement unit determines a display position of the second incidental information at a position based on the position of the emphasis line and close to the position of the monitored object ;
The information processing device according to claim 3 . the second incidental information placement unit determines a display position of the second incidental information based on whether the user's gaze point is within or near an image area of the monitored object in the captured image or the screen;
5. The information processing device according to claim 3. The information processing device includes:
a transmission control unit that transmits data in which the first incidental information is arranged at the display position of the first incidental information determined by the first incidental information arrangement unit to a presentation control device via a communication unit;
The information processing device according to any one of claims 1 to 5 . Detecting the position of a monitored object in a captured image or on a screen;
Calculating the position of the user's monitoring focus area in the captured image based on information about the user's gaze point generated by an eye-gaze sensor ;
determining a display position of first incidental information, which is information related to the object to be monitored, at a position according to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area;
An information processing method comprising: Computer,
a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen;
a monitoring focus area calculation unit that calculates the position of a monitoring focus area of the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor ;
a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area;
A program that causes the information processing device to function as an information processing device having the above. a monitoring target object detection unit that detects the position of a monitoring target object in a captured image or on a screen;
a monitoring focus area calculation unit that calculates the position of a monitoring focus area of the user in the captured image or the screen based on information about the user's gaze point generated by a line-of-sight sensor ;
a first incidental information placement unit that determines a display position of first incidental information, which is information related to the object to be monitored, at a position corresponding to the position of the object to be monitored and the position of the key monitoring area and in the vicinity of the position of the key monitoring area;
an information processing device comprising:
a control unit that controls the display unit so that the first auxiliary information is displayed at the display position,
a presentation control device;
A system comprising: