JP7625407B2 - Information processing device, control method thereof, and program - Google Patents

Description

The present invention relates to an information processing device, a control method thereof, and a program, and in particular to an information processing device, a control method thereof, and a program for controlling tracking and photographing a target subject by a camera.

Conventionally, a camera platform system has been widely known, in which a user remotely controls a camera using a control device to obtain the desired image. For example, the images of aircraft seen on television news are captured by a broadcasting station remotely controlling a camera platform device permanently installed on the roof of an airport.

An automatic tracking system has also been proposed that incorporates image recognition technology into the camera platform to recognize subjects in the image and automatically pan, tilt, and zoom to track the subject's movements. This makes it possible to automatically capture moving subjects without the user having to operate a controller.

In recent years, examples of image recognition technology that uses machine learning (AI) have become known. For example, Patent Document 1 proposes a system that detects and monitors subjects captured by a camera.

Patent No. 6184271

However, in the conventional technology disclosed in Patent Document 1, when the certainty (likelihood) of the machine learning processing results is low, the angle of the sub-camera is changed, so the main camera cannot track the subject and cannot capture continuous tracking footage. If the accuracy of subject detection could be improved and cases in which the likelihood is reduced could be eliminated, it would be possible to capture tracking footage using the main camera, but this would require a lot of learning.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information processing device, a control method thereof, and a program that can track a subject during shooting without requiring special learning, even in cases where the subject detection accuracy is reduced.

The information processing device of claim 1 of the present invention is an information processing device that tracks a target subject using an imaging device by controlling the pan, tilt, and zoom of the imaging device that captures moving images, and is equipped with an acquisition means for acquiring frames of moving images captured by the imaging device, an estimation means for estimating an object present in the acquired frames, and a control means for controlling the imaging device to track the object present in the acquired frames if the estimated object is a predetermined tracking object, and is characterized in that, if the size of the object present in the acquired frames is equal to or smaller than a first threshold , the control means resets the tracking object to an object whose actual size is smaller than the predetermined tracking object .

According to the present invention, even in cases where the accuracy of subject detection is reduced, it is possible to track a subject during shooting without requiring special learning.

1 is a diagram showing a configuration of an automatic tracking system including an information processing device according to an embodiment of the present invention. FIG. 2 is a diagram showing the hardware configurations of the information processing device, the camera platform device, and the operation device appearing in FIG. 1 . 2 is a diagram showing a software configuration realized by using hardware resources and programs included in the information processing device, the camera platform device, and the operation device in FIG. 1 . FIG. 4 is a conceptual diagram of an estimation process in an estimation unit in FIG. 3 . 10 is a flowchart showing a detailed flow of processing in a learning phase according to the embodiment of the present invention. 4 is a diagram showing an example of learning data input to a learning unit in FIG. 3; 5 is a diagram illustrating an automatic tracking and photographing operation in an automatic tracking system using the trained model of FIG. 4. 10 is a flowchart showing a detailed flow of processing in an estimation phase according to the embodiment of the present invention.

Figure 1 is a diagram showing an automatic tracking system 1 including an information processing device 100 according to an embodiment of the present invention.

In FIG. 1, the automatic tracking system 1 includes an information processing device 100, a camera platform device 200, an operation device 300, and a network NET.

The operation device 300 is a device that accepts user operations, and commands (control signals) corresponding to the accepted user operations are transmitted to the camera platform device 200 via the network NET and the information processing device 100.

The pan-and-tilt device 200 operates according to the contents of the control signal transmitted from the operation device 300. This allows the user to remotely control the pan-and-tilt device 200. Images captured by a camera 201 (FIG. 2) provided on the pan-and-tilt device 200 are input to the information processing device 100, where various calculations and recordings required for automatic tracking shooting are performed.

The network NET is a communication line such as a public telephone line or the Internet. The camera platform device 200 and the information processing device 100 are installed in spots such as airports, steel towers, and TV station rooftops, and the operation device 300 is installed inside the TV station, etc. In the following embodiment, a case will be described in which the information processing device 100 and the camera platform device 200 are installed in an airport, and the object to be automatically tracked and photographed by the camera platform device is an airplane.

Figure 2 is a diagram showing the hardware configuration of the information processing device 100, the camera platform device 200, and the operation device 300.

The information processing device 100 includes a RAM 101, a GPU 102, a CPU 103, an input unit 104, a memory unit 105, a serial communication unit 106, a network communication unit 107, and a UI unit 108.

The camera platform device 200 includes a camera 201, a drive unit 202, a serial communication unit 203, a CPU 204, and a memory unit 205.

The operation device 300 includes a network communication unit 301, an operation unit 302, a memory unit 303, a CPU 304, and a display unit 305.

The details of the hardware resources of the information processing device 100 are explained below.

RAM 101 is a volatile memory and is used as a temporary storage area such as the main memory and work area of CPU 103.

The CPU 103 controls each part of the information processing device 100, for example, according to a program stored in the memory unit 105, using the RAM 101 as a work memory.

Since GPU 102 can perform efficient calculations through parallel processing of data, the process of learning multiple times using a learning model through deep learning, as described below, is performed by GPU 102.

The input unit 104 (acquisition means) is an interface for inputting a video signal to the information processing device 100 via a wired video signal line, and is, for example, a variety of communication interfaces such as a USB.

The storage unit 105 is a non-volatile memory, and image data and other data, various programs for the operation of the CPU 103, etc. are stored in respective predetermined areas. The storage unit 105 is configured, for example, by a magnetic disk such as a HDD or flash memory.

The serial communication unit 106 is an interface for serial communication with the camera platform device 200 under the control of the CPU 103.

The network communication unit 107 is a communication interface for communicating with the operation device 300 via the network NET under the control of the CPU 103.

The UI unit 108 is a user interface for accepting user operations on the information processing device 100 and displaying information on the information processing device 100 to the user. Specifically, the UI unit 108 is configured with a touch panel, but may further include a keyboard, a mouse, a display, etc.

The hardware resources of the camera platform device 200 are described in detail below.

The camera 201 (imaging device) captures the surroundings around the camera platform device 200 and captures an image of a target subject. The camera 201 is equipped with an optical zoom lens that can change the imaging magnification. That is, the camera platform device 200 has an optical zoom function that changes the magnification of the captured image by transmitting an optical zoom control command for the optical zoom lens from the CPU 204 to the camera 201. Furthermore, the camera platform device 200 also has a digital zoom function that changes the magnification of the captured image by transmitting a digital zoom control command from the CPU 204 to the camera 201 to locally enlarge a part of the captured image. The digital zoom function is used when the magnification is insufficient with the optical zoom function, that is, when it is desired to further enlarge the captured image. The camera 201 is also connected to the input unit 104 of the information processing device 100 by a wired video signal line, and outputs the captured video signal to the information processing device 100.

The driving unit 202 is an actuator for rotating the camera platform device 200 in the pan and tilt directions, as well as its driving circuit and peripheral circuits. That is, the camera platform device 200 has a tracking shooting function that shoots tracking video by sending a tracking control command from the CPU 204 to the driving unit 202 to rotate the camera platform device 200 in the pan and tilt directions relative to a target subject.

The serial communication unit 203 is connected to the serial communication unit 106 of the information processing device 100, and is an interface for performing serial communication with the information processing device 100 based on the control of the CPU 204.

The CPU 204 controls each part of the camera head device 200 according to a program stored in the memory unit 205, for example.

The storage unit 205 is a non-volatile memory, and the setting data of the camera head device 200, other data, various programs for the operation of the CPU 204, etc. are stored in designated areas.

The hardware resources of the operation device 300 are described in detail below.

The network communication unit 301 is a communication interface for communicating with the information processing device 100 via the network NET based on the control of the CPU 304.

The operation unit 302 is a joystick, an operating lever, and various switches, and the user operates the operation unit 302 to perform rotation control, zoom control, gain adjustment, and the like of the camera platform device 200. Note that zoom control here refers to zoom control that utilizes the optical zoom function and digital zoom function described above.

The storage unit 303 is a non-volatile memory, and the setting data of the operation device 300, other data, various programs for the operation of the CPU 304, etc. are stored in respective designated areas.

The CPU 304 controls each part of the operation device 300 according to a program stored in the memory unit 303, for example.

The display unit 305 is equipped with an LED and a touch panel, and notifies the user of the status of the camera head device 200, warnings, etc.

Note that CPU 103, CPU 204, and CPU 304 may each be configured with one or more processors.

Figure 3 is a diagram showing a software configuration realized by using the hardware resources and programs provided in the information processing device 100, the camera platform device 200, and the operation device 300 shown in Figure 2.

The software configuration of the information processing device 100 is described below.

The information processing device 100 includes a learning unit 150, a data storage unit 151, an estimation target setting unit 152, a mode management unit 153, an image processing unit 154, an estimation unit 155, an estimation result processing unit 156, and a camera head control unit 157.

The learning unit 150 (learning means) executes a learning process to enable the estimation unit 155 to perform estimation processing. The details of the learning process executed by the learning unit 150 will be described later.

The data storage unit 151 records the video captured using automatic tracking and records the learning data.

The estimation target setting unit 152 manages the tags of objects output by the estimation unit 155. Here, the tag is part of the learning data input to the learning unit 150 and one of the data output by the estimation unit 155, and is a label indicating what the object is. Specific examples of tags include aircraft, dogs, cats, and birds. In the estimation target setting unit 152, the user can set in advance the tag of an object that he or she wishes to automatically track and photograph, and it is also possible to set multiple tags. In this embodiment, in order to perform automatic tracking and photographing of an aircraft, the tag of the aircraft is set as the tag of the object that he or she wishes to automatically track and photograph.

The mode management unit 153 manages the mode of the information processing device 100, and manages three modes: learning mode, automatic tracking mode, and manual mode. The details of each mode will be described later.

The image processing unit 154 processes the video signal (image; frame) received from the camera platform device 200. Specifically, it resizes the received image and adjusts the brightness.

The estimation unit 155 (estimation means) uses the image output from the image processing unit 154 as input data, inputs it to the trained model generated by the learning unit 150, and performs estimation processing. In this way, the subject (object) present in the frame received from the camera platform device 200 is estimated.

When the estimation result output from the estimation unit 155 includes a tracking target, the estimation result processing unit 156 performs various noise processing and averaging processing, and outputs the position of the tracking target in the image (current position of the subject). The image captured by the camera platform device 200 contains noise other than the tracking target (aircraft other than aircraft, parts of the background, clouds, etc. that can be mistaken for aircraft). The estimation result processing unit 156 has the role of processing this noise and outputting a highly reliable current position of the subject. Furthermore, if other tags are set in addition to the aircraft in the estimation target setting unit 152 described above, the noise processing in the estimation result processing unit 156 may be burdened, or the reliability of the current position of the subject to be output may decrease. For this reason, it is preferable to set the tags in the estimation target setting unit 152 to the minimum necessary.

The tripod head control unit 157 calculates a control signal for controlling the tripod head device 200 according to the aforementioned mode, and outputs it to the tripod head device 200. If the current mode is the automatic tracking mode, it calculates a control signal for controlling the tripod head device 200 based on the current position of the subject and the tracking target position (the position where the subject is to be held within the shooting screen during tracking shooting), and outputs it to the tripod head device 200. If the current mode is the manual mode, it outputs a control signal sent from the operation device 300 to the tripod head device 200.

The learning unit 150 uses the GPU 102 in addition to the CPU 103 for processing. Specifically, when executing a learning program including a learning model, the CPU 103 and the GPU 102 work together to perform calculations to perform the learning process. Similarly to the learning unit 150, the estimation unit 155 also works together with the CPU 103 and the GPU 102 to perform calculations to perform the estimation process. Depending on the calculation processing capabilities of the CPU 103 or the GPU 102, the processing of the learning unit 150 and the estimation unit 155 may be performed by the CPU 103 or the GPU 102 alone.

The software configuration of the camera head device 200 is explained below.

The camera platform device 200 includes a pan/tilt control unit 250, a camera control unit 251, a setting management unit 252, and a communication unit 253.

The pan/tilt control unit 250 outputs a signal for driving the pan and tilt to the drive unit 202 based on a control signal that controls the drive of the drive unit 202 received by the communication unit 253.

The camera control unit 251 outputs a signal for controlling the camera 201 to the camera 201 based on the control signal for controlling the camera 201 received by the communication unit 253.

The setting management unit 252 manages the settings of the operation device 300. Specific setting items managed by the setting management unit 252 include drive parameters such as the maximum pan and tilt speeds and the drivable range.

The communication unit 253 exchanges control commands and status information with the camera head control unit 157 in accordance with predetermined communication rules (protocol).

The software configuration of the operation device 300 is described below.

The operating device 300 includes a communication unit 350 and a display unit 351.

The communication unit 350 exchanges control signals and status information with the camera head control unit 157 in accordance with predetermined communication rules (protocol).

Figure 4 is a conceptual diagram of the estimation process in the estimation unit 155.

As shown in FIG. 4, the estimation unit 155 inputs input data 400 to a trained model 403, executes an estimation process, and outputs output data 401 from the trained model 403.

The input data 400 is image data captured by the camera 201 of the camera platform device 200 and processed by the image processing unit 154. Note that since the camera platform device 200 captures video, the data actually input to the trained model 403 as the input data 400 is one frame of the video.

The output data 401 includes tags, coordinates, and likelihoods of objects estimated to be present in the image, which is the input data 400.

The tags of the estimated objects are selected from the tags contained in the training data entered during training. Details of the training data will be described later.

The coordinates of the estimated object are the upper left coordinate (coordinate 1) and the lower right coordinate (coordinate 2) of the bounding box of the estimated object, as shown in image 402. From the coordinates of these two points, the object size and center point coordinates can be calculated.

The likelihood of an estimated object is a value between 0 and 1, and the higher the value, the higher the reliability of the estimation result by the trained model 403 for the output tag.

The trained model 403 is a neural network, and its internal parameters are generated by machine learning of the trained model by the training unit 150.

The learning unit 150 includes an error detection unit and an update unit.

The error detection unit obtains the error between the teacher data and the output data output from the output layer of the neural network in response to the input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the teacher data.

The update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. This update unit updates the connection weighting coefficients, for example, using the backpropagation method. The backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the above-mentioned error.

Figure 5 is a flowchart showing the detailed flow of processing in the learning phase according to this embodiment.

First, in step S500, the CPU 103 determines whether the current mode managed by the mode management unit 153 is the learning mode. If the CPU 103 determines that the current mode is the learning mode, the process proceeds to step S501. If not, the process proceeds to step S506.

In step S501, the CPU 103 reads one piece of learning data from the data storage unit 151 and proceeds to step S502. Here, the learning data will be described with reference to FIG. 6.

Figure 6 shows an example of learning data input to the learning unit 150.

The learning data is data in which an image and a tag (teaching data) of an object contained in the image are linked. Note that an image preferably contains one object, and the image size is preferably the same among multiple learning data. In this embodiment, to perform automatic tracking and shooting of an aircraft at an airport, an image of the aircraft is prepared as input data, and in addition, images of other objects that are likely to be captured by the camera 201 of the camera platform 200 during automatic tracking and shooting are prepared. Specific examples of other objects include birds and kites. Images of these objects may be photographed and recorded in advance using the camera platform 200. Among the learning data, the teaching data is set by visually determining what the object is from the image of the object described above. FIG. 6 shows learning data 601 to 603, which are examples of objects contained in the image that is the input data, such as an aircraft, a bird, and a kite.

Returning to FIG. 5, in step S502, the CPU 103 determines whether the learning data read from the data storage unit 151 conforms to the determined rules (is regular). As a result, if the CPU 103 determines that the learning data conforms to the rules, the process proceeds to step S503, and if not, the process returns to step S501.

In step S503, the CPU 103 inputs the learning data read from the data storage unit 151 into a learning model, which is a neural network in the learning unit 150.

In step S504, the CPU 103 and GPU 102 work together to perform learning processing in the learning unit 150 and update the internal parameters of the learning model.

Next, in step S505, the CPU 103 determines whether learning based on all learning data recorded in the data storage unit 151 has been completed, and if all learning has been completed, ends this process, otherwise returns to step S501.

By the above processing, the internal parameters of the trained model 403 on which the estimation process is performed by the estimation unit 155 are determined.

The process from step S506 onwards shows the process flow when the aforementioned mode is manual mode or automatic tracking mode.

In step S506, the CPU 103 determines whether the current mode is manual mode. If the CPU 103 determines that the current mode is manual mode, the process proceeds to step S507. If not, the process proceeds to step S509.

In step S507, the CPU 103 receives a control signal (command) from the operation device 300 and proceeds to step S508.

In step S508, the CPU 103 transmits the received control signal to the camera head device 200 and ends this process.

In step S509, the CPU 103 executes the auto-tracking mode process described below, and then ends this process.

Figure 7 is a diagram explaining the automatic tracking and shooting operation in the automatic tracking system 1 using the trained model 403 in Figure 4.

First, in step S701, a control signal for a tracking start command is transmitted from the operation device 300 to the information processing device 100. Note that this control signal is generated by the CPU 304 and transmitted to the information processing device 100 when a predetermined user operation is performed on the operation unit 302 of the operation device 300.

Next, in step S702, the information processing device 100 transmits a control signal for a tracking start position command to the camera-head device 200. The tracking start position is the pan, tilt, and zoom position of the camera-head device 200 at which automatic tracking shooting starts, and is stored in advance in the memory unit 205 of the camera-head device 200 by the user operating the operation unit 302.

In step S703, the CPU 204 of the camera head device 200 that has received the tracking start position command moves the pan, tilt, and zoom positions to the tracking start position.

In step S704, when the pan, tilt, and zoom positions reach the tracking start position, the CPU 204 of the camera head device 200 transmits to the information processing device 100 a notification that the tracking start position has been reached and a video signal captured by the camera 201.

In step S705, the CPU 103 of the information processing device 100, which has received the video signal from the pan-and-tilt device 200, starts a recording process to record the video signal from the pan-and-tilt device 200 in the memory unit 105.

Next, in step S706-1, the CPU 103 cooperates with the GPU 102 to perform an estimation process to estimate the subject contained in the received video signal (image).

Furthermore, in step S706-2, the CPU 103 calculates a tripod head control command using the estimation result processing unit 156 and the tripod head control unit 157, and in step S707, the CPU 103 transmits a control signal for the calculated tripod head control command to the tripod head device 200.

In step S708, the CPU 204 of the camera head device 200 controls the pan and tilt positions according to the control signal received by the drive unit 202, and also controls the zoom of the camera 201. Then, in step S709, the CPU 204 transmits the video signal captured by the camera 201 to the information processing device 100.

Then, the processing of steps S706-1 to S709 is executed, enabling automatic tracking and shooting of the target subject.

Figure 8 is a flowchart showing a detailed flow of processing in the estimation phase according to this embodiment. Note that this processing is also the automatic tracking mode processing of step S509 in Figure 5. Therefore, hereinafter, this processing will be referred to as automatic tracking mode processing.

The automatic tracking mode starts, and in step S800a, the CPU 103 receives an image (a frame of a moving image) from the camera head device 200 at the input unit 104 of the information processing device 100.

In step S800, the CPU 103 changes (resizes) the size of the received image in the image processing unit 154. The larger the size of the image data input to the trained model 403 as the input data 400, the longer the estimation process takes for the estimation unit 155. That is, the process of step S800 aims to reduce the time required for the estimation process by reducing the size of the input data 400. In this embodiment, the input data 400 is resized so that the time required for the estimation process after the size change in step S800 is about 50 ms. However, this is not the case because the time required for the estimation process is related to the computational processing capabilities of the CPU 103 and GPU 102.

Next, in step S801, the CPU 103 acquires a tag (tracking object) to be estimated from the estimation target setting unit 152. In this embodiment, an aircraft is set as the initial value of the estimation target setting unit 152, and therefore, in step S801, an aircraft is acquired as the tag.

Next, in step S802, the CPU 103 inputs the input data 400 resized in step S800 into the trained model 403.

In step S803, the CPU 103 and the GPU 102 cooperate to execute estimation processing in the estimation unit 155 and output the output data 401.

Then, the process proceeds to step S804, where the CPU 103 determines whether the output data 401 includes the tag acquired in step S801. As a result, if the CPU 103 determines that the output data 401 includes the tag acquired in step S801, the process proceeds to step S805, and if not, the process proceeds to step S890.

In step S890, the CPU 103 sends a control signal to the camera head device 200 to command a stop, and then proceeds to step S891.

In step S891, the CPU 103 changes the current mode managed by the mode management unit 153 to manual mode and ends the auto-tracking mode processing.

Thus, in step S804, if the output data 401 from the estimation unit 155 does not include the tag set by the estimation target setting unit 152, the CPU 103 determines that the tracking target has disappeared from the captureable range and ends the automatic tracking mode.

In step S805, the CPU 103 (calculation means) calculates the object size and center point coordinates of the estimated object in the screen from the upper left coordinates and lower right coordinates of the circumscribing frame of the estimated object included in the output data 401. Next, the CPU 103 (control means) calculates a pan and tilt control command based on the difference between the calculated center point coordinates and the tracking target position, and transmits a control signal of the command to the camera head device 200. Similarly, the CPU 103 (control means) calculates a zoom control command based on the calculated object size and tracking target size, and transmits a control signal of the command to the camera head device 200. The tracking target position and tracking target size may be registered in advance in the storage unit 105 of the information processing device 100. In this embodiment, the center coordinates in the screen are registered in advance as the tracking target position, and 30% of the screen size is registered in advance as the tracking target size, but this is not limited to this. The tracking target position and tracking target size may also be configured to be set in the information processing device 100 from the operation device 300.

By the above processing, the camera platform device 200 performs rotation and magnification control of the camera 201 according to the control signals of the pan and tilt control commands and the zoom control command transmitted from the information processing device 100. In addition, the image captured by the camera 201 during such rotation and magnification control is transmitted again to the information processing device 100. Except as described below (NO in step S808), the image transmitted again from the camera platform device 200 is used as input data 400, and the processing from step S800a is repeated. This allows the CPU 103 to control the automatic tracking shooting of the camera 201 based on the received frame each time it receives a frame from the camera 201.

When an aircraft taking off is photographed from a permanently installed pan head device 200, the aircraft moves away from the pan head device 200, so in the automatic tracking mode, the zoom control command in step S805 controls the aircraft to approach the tracking target size. However, if the camera 201 continues such automatic tracking photography, the size of the aircraft in the input data 400 becomes smaller, and image quality degradation occurs. As a result, the likelihood of the aircraft tag in the output data 401 from the estimation unit 155 decreases. In addition, if the camera 201 exceeds the optical zoomable area of the camera 201 and zooms in on the aircraft in the input data 400 by digital zoom, image quality degradation occurs. This also reduces the likelihood of the aircraft tag in the output data 401 from the estimation unit 155. In such a case, even if an aircraft is present in the subject included in the image transmitted from the pan head device 200, the accuracy of estimating the subject as an aircraft by the trained model 403 decreases, making it difficult to continue automatic tracking photography.

Therefore, in the processing from step S806 onwards, as described below, the estimation target is reset according to at least one of the calculation results from step S805, the output data 401 obtained in step S803, and the meta information of the frame received in step S800a. This enables the information processing device 100 to control automatic tracking and photography of the aircraft using the camera platform device 200.

In step S806, the CPU 103 determines whether the object size calculated in step S805 is equal to or smaller than a predetermined value (equal to or smaller than a first threshold value). If the CPU 103 determines that the object size calculated is equal to or smaller than the predetermined value, the process proceeds to step S809; otherwise, the process proceeds to step S807.

In step S807, the CPU 103 obtains the likelihood of the object from the output data 401, and if the likelihood is equal to or less than a predetermined value (equal to or less than a second threshold), the process proceeds to step S809; otherwise, the process proceeds to step S808.

In step S808, the CPU 103 determines whether or not there is a factor that reduces image quality in the camera 201. As a result, if the CPU 103 determines that there is a factor that reduces image quality in the camera 201, the process proceeds to step S809, and if not, the process returns to step S800a. In the embodiment, the CPU 103 determines that there is a factor that reduces image quality if the camera 201 used digital zoom when capturing the frame received in step S800a. When acquiring a frame from the camera 201, information on whether or not digital zoom was used when capturing the frame is also acquired as meta information for the frame.

The presence or absence of factors that degrade image quality may be determined based on the brightness of the captured frame, the gain of the camera 201 at the time of capture, and the time of capture, either alone or in combination. For example, when acquiring a frame from the camera 201, the brightness, gain, and capture time at the time of capture are also acquired as meta information for the frame. This allows the CPU 103 to determine from the acquired meta information that there is a factor that degrades image quality if the brightness at the time of capture of the frame was low, the gain of the camera 201 was high, and the capture time was at night.

In step S809, the CPU 103 resets the estimation target managed by the estimation target setting unit 152. Specifically, in addition to the initial value of aircraft, a tag that is easily mistaken for an aircraft (in this embodiment, a bird and kite tag whose actual size is smaller than that of an aircraft) is set, and the process returns to step S800a. For tags that are easily mistaken, a tracking target size smaller than that of an aircraft, here 10% of the screen size, is preregistered. As a result, from the input data 400 based on the image (image from the next frame onward) newly received from the camera platform device 200, the estimation unit 155 outputs output data 401 including bird and kite tags in addition to aircraft in step S803. Therefore, even if the estimation unit 155 outputs a tag other than an aircraft in step S804, that is, the estimation result is a result of misidentifying an aircraft as a bird or kite, it is possible to continue automatic tracking and shooting of the aircraft.

In this embodiment, when the estimation target managed by the estimation target setting unit 152 is reset, the CPU 103 (notification means) may notify the camera head device 200 of this via the serial communication unit 106. In this case, when the camera head device 200 receives such a notification, it changes the drive parameters stored internally. Specifically, when the estimation target is reset to a bird or a kite, its actual size will be smaller than the initial value of the estimation target, which is an airplane, and the amount of movement within the screen will also be smaller. Therefore, the camera head device 200 changes the drive parameters of pan, tilt, and zoom managed by the setting management unit 252, and achieves finer control than before the estimation target was reset.

In this embodiment, the determinations of steps S806, S807, and S808 are performed in this order, but it is also possible to determine only one of these, or to perform them in a different order.

Note that in this embodiment, an example has been described in which automatic tracking and photography is performed on a scene in which an aircraft is taking off, but this is not limiting, and automatic tracking and photography may also be performed on a scene in which an aircraft is landing.

In this embodiment, the information processing device 100 and the camera platform device 200 are connected by serial communication and a wired video signal line, but the connection is not limited to this and may be made by a public telephone line or a communication line such as the Internet. For example, the information processing device 100 may be a cloud server.

In addition, in this embodiment, the learning unit 150 is also present within the information processing device 100, but the learning unit 150 may be present in an external device connected to the information processing device 100. In this case, the information processing device 100 acquires the internal parameters of the trained model 403 from the external device.

Note that, among the above-mentioned processing units, the estimation unit 155 executes the estimation process using the learned model 403 that has been machine-learned, but rule-based estimation process such as a look-up table (LUT) may also be performed. In that case, for example, the relationship between the input data 400 and the output data 401 is created in advance as an LUT. Then, it is advisable to store this created LUT in the storage unit 105 of the information processing device 100. When the estimation unit 155 executes the estimation process, the output data can be obtained by referring to this stored LUT. In other words, the LUT operates in cooperation with a CPU or GPU as a program for performing the same process as the learned model 403, thereby executing the estimation process of the estimation unit 155.

The learning unit 150 may also learn by using as learning data object images linked to tags of objects other than the tracking target and objects that are easily mistaken for the tracking target (e.g., clouds; hereinafter, referred to as "incorrect tags"). In this case, when the estimation target is redefined in step S809, incorrect tags are also added. After that, if in step S804 the only tags included in the output data 401 are incorrect tags, the process proceeds to step S890, where the information processing device 100 determines that the tracking target has disappeared from the captureable range and ends the automatic tracking mode.

In this embodiment, the learning model that is learned by the learning unit 150 and the trained model 403 that is used in the estimation process of the estimation unit use a neural network algorithm, but other algorithms may be used. For example, a decision tree or an SVM (Support Vector Machine) algorithm may be used.

The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having a computer in the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). The processor or circuitry may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

100 Information processing device 103, 204 CPU
106, 203 Serial communication unit 150 Learning unit 152 Estimation target setting unit 155 Estimation unit 157 Head control unit 200 Head device 201 Camera 202 Driving unit 300 Operation device NET Network

Claims (12)

An information processing device that tracks a target subject by controlling pan, tilt, and zoom of an imaging device that captures a moving image, comprising:
an acquisition means for acquiring frames of a moving image captured by the imaging device;
An estimation means for estimating an object present in the acquired frame;
a control unit that controls the imaging device so as to track an object present in the acquired frame when the estimated object is a preset tracking object;
The information processing device is characterized in that the control means resets the tracking object to an object whose actual size is smaller than the predetermined tracking object when the size of an object present in the acquired frame is equal to or smaller than a first threshold . The information processing apparatus according to claim 1 , wherein the control means resets the tracking object when the estimated object likelihood is equal to or smaller than a second threshold value. 3 . The information processing apparatus according to claim 1 , wherein the control means resets the tracking object when it is determined that there is a factor causing deterioration in image quality in the imaging device based on meta information of the frame. 4 . the imaging device performs the zooming using at least one of an optical zoom and a digital zoom;
4. The information processing device according to claim 3, wherein the control means determines that the factor of image quality degradation exists when the meta information includes information indicating that the digital zoom was used when the frame was captured by the imaging device. The information processing device according to claim 3 or 4, characterized in that the control means determines that there is a factor of image quality degradation based on at least one of the brightness of the frame, the gain of the imaging device when the frame was photographed, and the time when the frame was photographed, which are included in the meta information. 6. The information processing apparatus according to claim 1, further comprising a notification unit that, when the control unit resets the tracking object, notifies the imaging device of that effect. 7. The information processing apparatus according to claim 6 , wherein the imaging device changes driving parameters of pan, tilt and zoom when the imaging device receives the notification from the notification means. The information processing apparatus according to claim 1 , further comprising: a learning unit that generates internal parameters of a trained model used for estimating the object in the estimation unit. 9. The information processing apparatus according to claim 8 , wherein the learning means uses learning data in which an image and teacher data indicating an object contained in the image are linked to each other. The information processing device according to claim 8 or 9, characterized in that the estimation means, each time it receives a frame of the moving image from the imaging device, resizes the received frame as input data to the trained model , and outputs information about the object from the trained model as output data. A method for controlling an information processing device to track a target subject by controlling pan, tilt, and zoom of an imaging device that captures a moving image, comprising:
An acquisition step of acquiring frames of a moving image captured by the imaging device;
an estimation step of estimating an object present in the acquired frame;
and a control step of controlling the imaging device so as to track an object present in the acquired frame when the estimated object is a preset tracking object,
A control method characterized in that the control step resets the tracking object to an object whose actual size is smaller than the predetermined tracking object when the size of an object present in the acquired frame is equal to or smaller than a first threshold . A computer-executable program that causes a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 10 .

Images