JP7449715B2 - Framing area learning device, framing area estimating device, and programs thereof - Google Patents

Description

The present invention relates to a framing area learning device and a framing area estimating device that learn camera framing parameters, and programs thereof.

Performing automatic photography by cropping from a camera attached to a drive mechanism such as a pan head or a fixed camera corresponds to sequentially determining the framing that is the composition of the camera. For example, Patent Document 1 proposes a method of determining a composition including a gaze target from image data from a camera and controlling a drive mechanism. Further, Patent Document 2 proposes a method of learning composition information and updating rules for cutting out from the original image.

Patent No. 4464902 Patent No. 5016540

In the conventional techniques described in Patent Documents 1 and 2, a final composition is determined for a certain photographing intention, and a driving mechanism is controlled and an image is cut out. In these conventional techniques, consider a case where an attempt is made to imitate camera work by an actual photographer. Photographers continuously determine the optimal composition depending on the subject and the surrounding situation. In this case, with the prior art described above, it is difficult for a photographer to automatically determine a composition depending on the situation.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a framing area learning device and a framing area estimating device that can automatically determine framing areas similar to those framed by an actual photographer from various camera positions, as well as programs thereof.

In order to solve the above problems, the framing area learning device according to the present invention converts camera parameters indicating the attitude and angle of view of the camera into framing parameters indicating the framing area to be framed by the camera, and converts the framing parameters and the subject's angle. A framing area learning device that learns in association with object information indicating at least one of a position or a speed, and is configured to include a conversion section and a learning section.

According to this configuration, the conversion unit converts the camera parameters acquired in synchronization with the subject information into framing parameters based on a preset conversion rule.
Then, the learning unit generates a learned inference model of the framing parameter by learning the subject information and the framing parameter in association with each other.

In this way, the framing region learning device learns framing parameters in a world coordinate system that does not depend on camera position, rather than camera parameters that depend on camera position. Therefore, the framing region learning device can generate a trained inference model that does not depend on the camera position.

Furthermore, in order to solve the above-mentioned problems, the framing region estimation device according to the present invention provides a framing method for estimating a framing parameter indicating a framing region of a camera using a trained inference model generated by a framing region learning device according to the present invention. The area estimating device is configured to include an estimating section and an inverse transform section.

According to this configuration, the estimating unit receives subject information indicating at least one of the position and velocity of the subject, and outputs the estimated value of the framing parameter according to the subject information using the learned inference model.
Then, the inverse transformer inversely transforms the estimated value of the framing parameter outputted by the estimator into the estimated value of the camera parameter indicating the attitude and angle of view of the camera, based on a preset inverse transform rule.

In this way, the framing area estimating device uses the trained inference model to estimate framing parameters in a world coordinate system that does not depend on the camera position, and converts the estimated framing parameters into camera parameters. Therefore, the framing area estimating device can estimate unique camera parameters for each camera using a trained inference model that does not depend on camera position.

Note that the present invention can also be realized by a program for causing a computer to function as the above-described framing area learning device or framing area estimating device.

Since the present invention uses an inference model that does not depend on camera positions, it is possible to automatically determine framing areas similar to those framed by an actual cameraman from various camera positions.

FIG. 2 is an explanatory diagram illustrating an example of a conventional camera, in which (a) is an external view of the camera, and (b) is a block diagram of a sensor. FIG. 1 is a block diagram showing the configuration of a framing region learning device according to a first embodiment. FIG. 1 is a block diagram showing the configuration of a framing area estimating device according to a first embodiment. FIG. 2 is an explanatory diagram illustrating a world coordinate system and a camera coordinate system in the first embodiment. FIG. 6 is an explanatory diagram illustrating conversion of the angle of view and scale before and after moving the camera position. FIG. 2 is an explanatory diagram illustrating a soccer match video as an example of situation data. FIG. 2 is an explanatory diagram illustrating position maps, in which (a) is a position map showing the positions of players and the ball, (b) is a position map showing the positions of players of one team, and (c) is a position map showing the positions of players of one team. This is a position map showing the positions of players on the team, and (d) is a position map showing the position of the ball. 2 is an explanatory diagram illustrating speed maps, (a) is a diagram showing the positions and velocity components of players of one team, and (b) is a diagram showing the velocity components of players of one team with dots. (c) is a speed map showing the speed components of players of one team in a circular area. 3 is a flowchart showing the operation of the framing region learning device according to the first embodiment. 3 is a flowchart showing the operation of the framing area estimating device according to the first embodiment. FIG. 2 is a block diagram showing the configuration of a learning section according to a second embodiment. It is a block diagram showing the composition of the learning part concerning a 3rd embodiment.

(First embodiment)
Hereinafter, each embodiment of the present invention will be described with reference to the drawings.
First, an example of the camera 3 related to the first embodiment will be described, and then the configurations of the framing region learning device 1 and the framing region estimating device 2 according to the first embodiment will be described in order.

[camera]
The camera 3 will be explained with reference to FIG.
The camera 3 is a camera configured to be remotely controllable. As shown in FIG. 1A, the camera 3 is mounted on a drive mechanism 30 such as a pan head, and pan, tilt, zoom, and focus are controlled by this drive mechanism 30. Note that zoom and focus control is performed by an internal mechanism of the camera 3, and will be described as being encompassed by this drive mechanism 30.

Further, as shown in FIG. 1(b), the camera 3 includes a sensor 31 that detects the camera operation. This sensor 31 is for measuring the attitude of the camera 3, the angle of view of a lens (not shown), and the focal position. This sensor 31 includes a pan angle sensor 31a that detects the pan angle of the pan head to which the camera 3 is fixed, a tilt angle sensor 31b that detects the tilt angle of the camera 3, and a zoom amount (angle of view) of the camera 3. The camera 3 includes a zoom sensor 31c that detects a zoom sensor 31c, and a focus sensor 31d that detects a focal position (focus) of a lens built into the camera 3.

For example, the pan angle sensor 31a and the tilt angle sensor 31b can be configured by a rotary encoder, a potentiometer, or a gyro sensor attached to a pan head. Further, the zoom sensor 31c and the focus sensor 31d can be configured by a method of reading the rotation angles of the zoom ring and the focus ring using a rotary encoder and a potentiometer. In addition, as the zoom sensor 31c and the focus sensor 31d, a linear sensor installed on the lens sliding part of the camera 3 can be used.

[Configuration of framing region learning device]
The configuration of the framing area learning device 1 will be described with reference to FIG. 2.
The framing area learning device 1 converts camera parameters of the camera 3 into framing parameters of the camera 3, and associates the converted framing parameters with situation data (subject information) j indicating at least one of the position or speed of the subject. It's something to learn.

Here, the framing parameter is a parameter indicating a framing area (composition) framed by the camera 3, and is expressed in a world coordinate system.
Further, the world coordinate system is a three-dimensional coordinate system corresponding to the space in which the subject exists, and is a coordinate system common to each camera 3.
Further, the camera parameters are parameters indicating the attitude and angle of view of the camera 3, and are expressed in a camera coordinate system.
Further, the camera coordinate system is a three-dimensional coordinate system based on the camera 3, and is a unique coordinate system for each camera 3.
Note that details of the framing parameters, world coordinate system, camera parameters, and camera coordinate system will be described later.

As shown in FIG. 2, the framing area learning device 1 includes a situation data storage section (subject information storage section) 10A, a camera parameter storage section 10B, a data selection section 11, a time adjustment section 12, and a situation data reading section. It includes a (subject information reading section) 13, a camera parameter reading section 14, a converting section 15, a quantizing section 16, and a learning section 17.

The situation data storage unit 10A is a storage device such as a hard disk drive (HDD), solid state drive (SSD), or memory that stores situation data j to be described later in advance. Here, the situation data storage unit 10A stores situation data j(ν) at each time ν ({j(ν)} _{νε(1, 2, . . . , N)} ).

The camera parameter storage unit 10B is a storage device such as an HDD, SSD, or memory that stores camera parameters, which will be described later, in advance. Here, the camera parameter storage unit 10B stores camera parameters θ(ν) at each time ν ({θ(ν)} _{νε(1, 2, . . . , N)} ).

Note that it is assumed that the situation data j(v) and the camera parameter θ(v) are acquired synchronously at the same time.
Further, although the situation data storage section 10A and the camera parameter storage section 10B are shown separately in FIG. 2, they may be integrated as a storage section.

The data selection unit 11 sequentially selects times ν=n associated with situation data j(ν) and camera parameters θ(ν) (n=1, 2, . . . , N). Then, the data selection section 11 outputs the selected time n to the time adjustment section 12 and the situation data reading section 13.

The time adjustment unit 12 has an offset time Δn set in advance, and adds the offset time Δ to the time n inputted from the data selection unit 11 to calculate a future time (n+Δn). This offset time Δn can be set to any value. Then, the time adjustment unit 12 outputs the generated future time (n+Δn) to the camera parameter reading unit 14.

The situation data reading section 13 reads out the situation data j(n) at a predetermined time n input from the data selection section 11 from the situation data storage section 10A. Then, the situation data reading section 13 outputs the read situation data j(n) to the learning section 17.

The camera parameter reading unit 14 reads camera parameters θ(n+Δn) at a future time (n+Δn) offset by a preset offset time Δn with respect to the read time n of the situation data j(n). The camera parameter reading unit 14 reads camera parameters θ(n+Δn) at a future time (n+Δn) input from the time adjustment unit 12. Then, the camera parameter reading unit 14 outputs the read camera parameter θ(n+Δn) to the converting unit 15 and the learning unit 17.

Here, the reason why the read time of the camera parameter θ(n+Δn) is offset to the future time by the offset time Δn will be explained.
If the cameraman's camera work is delayed, the delay will also be reflected in the camera parameter θ. Therefore, by offsetting the readout time of the camera parameter θ, it is possible to predict in advance the delay in camera work by the cameraman, and the influence of the delay in the camera parameter θ can be reduced.
Furthermore, by offsetting the readout time of the camera parameter θ, the influence of processing delays occurring in the framing area estimating device 2 can be reduced.

The conversion unit 15 inputs the camera parameter θ(n+Δn) synchronized with the situation data j(n) from the camera parameter reading unit 14. Then, the conversion unit 15 converts the camera parameter θ(n+Δn) of the input camera coordinate system to the continuous value g(n+Δn)=T(θ(n+Δn) of the framing parameter of the world coordinate system based on the preset conversion rule T. )). Further, the converter 15 outputs the converted continuous value g(n+Δn) of the framing parameter to the quantizer 16.
Note that details of the conversion process by the conversion unit 15 will be described later.

The quantization unit 16 receives the framing parameter as a continuous value g(n+Δn) on the world coordinate system from the conversion unit 15. Then, the quantization unit 16 quantizes the input framing parameters into discrete values q(n+Δn)=Q(g(n+Δn)) on the world coordinate system based on a preset quantization rule Q. be. Further, the quantization unit 16 outputs the quantized discrete value q(n+Δn) of the framing parameter to the learning unit 17.
Note that details of the quantization process by the quantization unit 16 will be described later.

Here, the discrete value refers to a value obtained by converting the framing parameter into a discrete value (spatial discrete value) on the world coordinate system by the inference model. For example, the discrete values are in mesh order, such as 0.25 meter increments, 0.5 meter increments, or 1.0 meter increments.
Further, the continuous value refers to a value obtained by converting the framing parameter into a continuous value (spatial continuous value) on the world coordinate system by the inference model.

The learning unit 17 receives the situation data j(n) from the situation data reading unit 13 and receives the discrete value q(n+Δn) of the framing parameter from the quantization unit 16. The learning unit 17 generates a learned inference model of the framing parameter by learning the situation data j(n) and the discrete value q(n+Δn) of the framing parameter in association with each other. That is, the learning unit 17 takes in the data pair of the situation data j(n) and the discrete value q(n+Δn) of the framing parameter, and outputs the learned parameter P.
Note that the configuration of the learning section 17 will be explained in the second embodiment and the third embodiment.

An inference model can be selected arbitrarily. For example, machine learning such as a support vector machine (SVM) or a k-nearest neighbor method (k-NN) may be used as the inference model. Further, a neural network may be used as the inference model. When the situation data j is image data, it is preferable to use a convolutional neural network (CNN) as the inference model.

Here, it is possible to arbitrarily select which of the discrete value q(n+Δn) or the continuous value g(n+Δn) of the framing parameter is to be used for learning, depending on the type of the inference model.
Note that when the learning section 17 directly learns the continuous value g(n+Δn) of the framing parameter, the framing region learning device 1 does not need to include the quantization section 16. In this case, the conversion unit 15 outputs the continuous value g(n+Δn) of the framing parameter to the learning unit 17.

As described above, the framing area learning device 1 learns the discrete value q or continuous value g of the framing parameter in the world coordinate system that does not depend on the camera position, instead of the camera parameter θ that depends on the camera position. It is possible to generate a trained inference model that is independent of dependencies. Therefore, the framing area learning device 1 can automatically determine a framing area similar to that framed by an actual cameraman from various camera positions.

The conventional inference model learns the camera parameter θ as it is, and outputs the camera parameter θ as it is. This inference model cannot be used. In reality, it is difficult to use conventional inference models because the camera positions differ during learning and estimation. On the other hand, since the trained inference model of the framing region learning device 1 has learned the framing parameters in the world coordinate system, it can be used even when the camera position differs between learning and estimation.

Furthermore, since the framing area learning device 1 offsets the readout time of the camera parameter θ(n+Δn) into the future, the influence of the delay in the camera parameter θ and the influence of the processing delay of the framing area estimation device 2 can be reduced.

[Configuration of framing area estimation device]
The configuration of the framing area estimating device 2 will be described with reference to FIG. 3.
The framing region estimating device 2 estimates a framing parameter using the learned inference model generated by the framing region learning device 1, and converts the estimated framing parameter into a camera parameter. As shown in FIG. 3, the framing region estimation device 2 includes an estimation section 20, an inverse quantization section 21, and an inverse transformation section 22.

The estimation unit 20 receives situation data j, which will be described later, and uses a learned inference model to output an estimated value of a framing parameter according to the input situation data j in a world coordinate system. That is, the estimation unit 20 is a learned inference model configured by the learned parameters P, and upon receiving the situation data j, outputs the discrete estimated value q _est of the framing parameter to the inverse quantization unit 21.

Hereinafter, a framing parameter whose estimated value is a discrete value may be referred to as a "discrete estimated value," and a framed parameter whose estimated value is a continuous value may be referred to as a "continuous estimated value."

The dequantization unit 21 converts the discrete estimated value q _est of the framing parameter input from the estimation unit 20 into a continuous estimated value g _est =Q ⁻¹ (q _est ) based on a preset dequantization rule Q ⁻¹ . This is inverse quantization. Then, the dequantization unit 21 outputs the dequantized continuous estimated value _gest of the framing parameter to the inverse transformation unit 22.

Note that details of the inverse quantization by the inverse quantization unit 21 will be described later.
Further, when the estimation unit 20 directly outputs the continuous estimated value _gest of the framing parameter, the framing area estimation device 2 does not need to include the inverse quantization unit 21. In this case, the estimator 20 outputs the continuous estimated value _gest of the framing parameter to the inverse transformer 22.

The inverse transformer 22 estimates the camera parameters in the camera coordinate system based on the inverse transform rule T ^-1 set in advance, using the continuous estimated value g _est of the framing parameters in the world coordinate system inputted from the inverse quantizer 21. The value is inversely converted to the value θ _est =T ⁻¹ (g _est ). Then, the inverse transformer 22 outputs the inversely transformed estimated value θ _est of the camera parameters to the outside.
Note that details of the inverse transformation process by the inverse transformation unit 22 will be described later.

As described above, since the framing area estimating device 2 uses an inference model that does not depend on the camera position, it is possible to automatically determine a framing area similar to that framed by an actual cameraman from various camera positions. That is, the framing region estimating device 2 estimates the discrete estimated value q _est of the framing parameter in a world coordinate system that does not depend on the camera position using the trained inference model, and uses the estimated discrete estimated value q _est of the framing parameter as the camera position. The estimated value θ _est of the parameter is converted. If a control signal for the camera 3 is generated from the estimated value θ _est of the camera parameter, the attitude of the camera 3 can be controlled at various camera positions.

With reference to FIGS. 4 and 5, framing parameters in the world coordinate system, camera parameters in the camera coordinate system, transformation processing and inverse transformation processing, and quantization processing and inverse quantization processing will be described in order.

<Framing parameters of world coordinate system, camera parameters of camera coordinate system>
FIG. 4 illustrates the relationship between the world coordinate system in which the subject exists and the camera coordinate system in which the position and orientation of the camera 3 are reflected, using a soccer field as an example.

As shown in FIG. 4, the world coordinate system has the center 91 of the soccer field 90 as its origin, the axis passing through the origin and parallel to the sideline 92 as the X axis, and the axis parallel to the center line 93 as the Y axis. The axis perpendicular to the plane of the field 90 and pointing upward is the +Z axis. Therefore, the world coordinate system becomes a right-handed system with the X-axis, Y-axis, and Z-axis in this order.

In the camera coordinate system, the optical axis direction of the camera 3 is the +z axis. Consider a case where the origin of the camera coordinate system coincides with the origin of the world coordinate system, and there is no rotation on each axis. In this case, the camera coordinate system +x axis and the world coordinate system +X axis, the camera coordinate system +y and the world coordinate system -Z axis, and the camera coordinate system +z axis and the world coordinate system +Y axis coincide. Therefore, the camera coordinate system becomes a right-handed system with the x-axis, y-axis, and z-axis in this order.

In the camera coordinate system, the rotation angle of the y-axis is a pan angle α [rad], the rotation angle of the x-axis is a tilt angle δ [rad], and the rotation angle of the z-axis is a roll angle φ [rad]. In this case, the attitude R of the world coordinate system in the camera coordinate system is expressed by the following equation (1). In other words, there is a relationship expressed by equation (1) between the pan angle α, tilt angle δ, and roll angle φ in the camera coordinate system and the attitude R in the world coordinate system.

As shown in FIG. 5, the origin of the camera coordinate system (the position of the camera 3) in the world coordinate system is assumed to be _tw = [ _twx , _twy , _twz ]. In this case, the origin t _c =[t _cx , t _cy , t _cz ] of the world coordinate system in the camera coordinate system can be expressed as t _c =-Rt _w .

The framing area of the camera 3 is determined by the position _tw , attitude R, and angle of view (more precisely, the horizontal angle of view of the lens attached to the camera 3) β of the camera 3.
The position _tw of the camera 3 may be actually measured at the shooting site. Further, the position _tw of the camera 3 can be estimated from the correspondence between the object in the image photographed at that position and the ground truth, which is the actual ground data of this object.

The attitude R of the camera 3 can be acquired from the sensors 31 (the pan angle sensor 31a and the tilt angle sensor 31b) connected to the camera 3. Since commonly used pan heads do not have a rotation mechanism for controlling the roll angle φ, the roll angle φ may be set to 0. Therefore, if the pan angle α and tilt angle δ of the camera coordinate system are known, the attitude R of the camera 3 can be determined in the world coordinate system.

The angle of view β of the camera 3 changes depending on the amount of rotation of the zoom ring when a zoom lens is used. When the output value of the zoom sensor 31c is input and the angle of view β of the camera 3 is output, the input and output values change depending on the sensor size of the camera 3 and the combination of zoom lenses. Therefore, the angle of view β of the camera 3 can be determined by measuring the input and output values in advance and creating a lookup table of these input and output values.
In addition, the angle of view β of the camera 3 may be a focal length f that is uniquely determined from the sensor size of the camera 3.

From the above, the minimum camera parameters required and obtainable for determining the framing area are the pan angle α, the tilt angle δ, and the angle of view β. That is, the pan angle α, tilt angle δ, and angle of view β of the camera 3 correspond to the camera parameter θ.

Under the condition that the position of the camera 3 is fixed, a framing parameter according to the situation data j can be estimated by an inference model learned using the camera parameter θ as teacher data. In order to estimate the framing parameters without depending on the camera position, the framing parameters and scale s of camera 3 are transformed into the world coordinate system, then these estimated values are transformed back into the camera coordinate system, and the camera parameters are It is necessary to find the estimated value θ _est .

When the tilt angle δ<0 of the camera 3, as shown in FIGS. 4 and 5, the intersection position where the optical axis (z axis) of the camera 3 intersects with the XY plane (Z=0) in the world coordinate system is c _w = [c _wx , c _wy , 0]. In this case, the intersection position c _w can be calculated using the following equation (2).

The scale s indicates the size of the framing area in the world coordinate system. As shown in FIG. 5, the scale s can be calculated using the following equation (3) from the optical axis length l from the position _tw of the camera 3 to the intersection position _cw and the angle of view β of the camera 3. In other words, the intersection positions c _wx , c _wy and the scale s correspond to the continuous value g of the framing parameter.

Note that in FIG. 5, the position of the camera 3 after the movement is _{tw_new} , and the angle of view estimated after the camera 3 is moved is β _est . Further, the scale estimated after the movement of the camera 3 is set _as sest, the optical axis length is set _as lest, and the intersection position is set as _{cw_est} .

<Conversion processing and inverse conversion processing>
From the above, the above equations (2) and (3) correspond to the conversion rule T set in advance in the conversion section 15. That is, the conversion unit 15 converts the camera parameters θ=[α, δ, β] in the camera coordinate system into the continuous value g=[ of the framing parameter in the world coordinate system, using equations (2) and (3) described above. c _wx , c _wy , s].

We will also consider inverse transformation from the continuous estimated value _gest of the framing parameter in the world coordinate system to the estimated value θ _est of the camera parameter. Here, the estimated value α _est of the pan angle can be calculated using equation (4), and the estimated value δ _est of the tilt angle can be calculated using equation (5).

As shown in FIG. 5, the estimated value β _est of the angle of view of the camera 3 can be calculated using the following equation (6), where the optical axis length l _est from the position _{tw_new} of the camera 3 to the intersection position c _{w_est} is set.

Equations (4) to (6) described above correspond to the inverse transformation rule T ⁻¹ set in advance in the inverse transformation unit 22. That is, the inverse transformation unit 22 converts the continuous estimated value g _est = [c _{wx_est} , c _{wy_est} , s _est ] of the framing parameter into the camera parameter of the camera coordinate system using Equations (4) to (6) described above. The estimated value θ _est = [α _est , δ _est , β _est ].

<Quantization processing and inverse quantization processing>
When an inference model deals with discrete values rather than continuous values, quantization from continuous values to discrete values and inverse quantization from discrete values to continuous values are required. As described above, the continuous estimated value g _est of the framing parameter is composed of three parameters: c _wx , c _wy , and s. In this case, since the number of vector dimensions is k=3, the continuous value of the framing parameter g=[g ₁ , g ₂ , g ₃ ]. At this time, the discrete values q=[q ₁ (g ₁ ), q ₂ (g ₂ ), q ₃ (g ₃ )] of the framing parameters can be calculated using the following equation (7). Note that A _k and B _k are parameters that can be arbitrarily set in each dimension k.

The above equation (7) corresponds to the quantization rule Q set in advance in the quantization section 16. That is, the quantization unit 16 quantizes the framing parameter from a continuous value g to a discrete value q using Equation (7) described above.

The dequantization unit 21 performs dequantization as expressed by the following equation (8). This equation (8) corresponds to the dequantization rule Q ⁻¹ set in advance in the dequantization section 21. That is, the dequantization unit 21 dequantizes the discrete estimated value q _{k_est} of the framing parameter into the continuous estimated value g _{k_est} using equation (8).

<Status data>
With reference to FIGS. 6 to 8, the situation data j will be explained using a soccer match video as an example.
The situation data j is data indicating at least one of the position and speed of the subject as the situation of the subject. For example, the situation data j is a soccer match video _j1 , as shown in FIG. The match video _j1 in FIG. 6 shows the positions of the player _9A and the ball _9B , which are the subjects. In addition, in FIG. 6, in order to make the drawing easier to read, only a part of the players _9A is labeled.

Furthermore, the situation data j may be a map that is an image of at least one of the position and speed of the subject. FIG. 7 shows a position map showing the positions of players and balls as an example of the situation data j.

The position map _j2 in FIG. 7(a) shows the positions of all players and the ball. In this map, 〇 indicates the position of a player on one team, × indicates the position of a player on the other team, and ● indicates the position of the ball.
The position map _j3 in FIG. 7(b) shows the positions of players of one team. In this position map _j3 , the position of each player is shown as a circular area having shading from the center side to the outer circumference side. According to this position map _j3 , since the position of each player is roughly indicated by a blurred circular area, the accuracy of the inference model can be improved.
The position map j ₄ in FIG. 7(c) shows the positions of the players of the other team using circular areas similar to the position map j ₃ . According to this position map j ₄ , the accuracy of the inference model can be improved like the position map j ₃ .
The position map _j5 in FIG. 7(d) shows the position of the ball in a circular area similar to the position map _j3 .

FIG. 8 shows a speed map showing the speed of the player as an example of the situation data j.
FIG. 8(a) illustrates the positions and velocity components (X-axis component, Y-axis component) of players of one team. This velocity component indicates a value normalized according to a predetermined rule.
The velocity map _j6 in FIG. 8(a) ignores the position of each player in FIG. 8(a) and shows the velocity component of each player as a point.
The velocity map _j7 in FIG. 8(c) shows the velocity components of each player in FIG. 8(a) as blurred circular regions. According to this speed map _j7 , since the speed of each player is roughly indicated by a blurred circular area, the accuracy of the inference model can be improved.
Note that in FIGS. 8(b) and 8(c), the vertical axis and the horizontal axis are shown for explanation, and there is no need to include the axis lines in the speed maps j ₆ and j ₇ .

Here, as the situation data j, any of the match video j ₁ , position maps j ₂ to j ₅ or speed maps j ₆ and j ₇ may be used. For example, as the situation data j, only one of the match video j ₁ , the position maps j ₂ to j ₅ , or the speed maps j ₆ and j ₇ may be used. Further, as the situation data j, two or more of the match video j ₁ , the position maps j ₂ to j ₅ or the speed maps j ₆ and j ₇ may be arbitrarily combined. Furthermore, by combining the position maps j ₃ and j ₄ of the players of both teams as the situation data j, the accuracy of the inference model can be improved.

[Operation of framing area learning device]
The operation of the framing area learning device 1 will be described with reference to FIG. 9.
In step S10, the data selection unit 11 selects time ν=n that is linked to the situation data j(ν) and the camera parameter θ(ν).
In step S11, the offset time Δn is preset in the time adjustment unit 12. Then, the time adjustment unit 12 adds the offset time Δ to the time n to calculate a future time (n+Δn).

In step S12, the situation data reading unit 13 reads the situation data j(n) at time n from the situation data storage unit 10A.
In step S13, the camera parameter reading unit 14 reads the camera parameter θ(n+Δn) at the future time (n+Δn) from the camera parameter storage unit 10B.

In step S14, the conversion unit 15 converts the camera parameter θ(n+Δn) into a continuous value g(n+Δn) of the framing parameter in the world coordinate system.
In step S15, the quantization unit 16 quantizes the continuous value g(n+Δn) of the framing parameter into a discrete value q(n+Δn).

In step S16, the learning unit 17 learns the situation data j(n) read out in step S12 and the discrete value q(n+Δn) of the framing parameter quantized in step S15, thereby learning the framing parameter. Generate a trained inference model.
Note that when the learning unit 17 directly learns the continuous value g(n+Δn) of the framing parameter in step S16, it is not necessary to execute the process of step S15.

[Operation of framing area estimation device]
The operation of the framing area estimating device 2 will be described with reference to FIG. 10.
In step S20, the estimation unit 20 inputs the situation data j.
In step S21, the estimation unit 20 outputs a discrete estimated value q _est of the framing parameter according to the situation data j using the learned inference model.

In step S22, the dequantization unit 21 dequantizes the discrete estimated value q _est of the framing parameter into a continuous estimated value g _est .
In step S23, the inverse transformer 22 inversely transforms the continuous estimated value _gest of the framing parameter into the estimated value θ _est of the camera parameter.
Note that in the case where the estimation unit 20 directly outputs the continuous estimated value _gest of the framing parameter in step S21, it is not necessary to execute the process of step S22.

(Second embodiment)
An example of the configuration of the learning section 17 according to the second embodiment will be described with reference to FIG. 11.
The learning unit 17 updates learning parameters during the learning process and outputs the final learning parameters as learned parameters P. As shown in FIG. 11, the learning unit 17 includes an estimation unit 170, an inverse quantization unit 171, an inverse quantization unit 172, an error evaluation unit 173, a parameter update unit 174, and a parameter storage unit 175. .

The estimation unit 170 is an inference model composed of learning parameters stored in the parameter storage unit 175. Similar to the estimation unit ₂₀ of FIG. 171.

Similar to the dequantization unit 21 in FIG. 3, the dequantization unit 171 dequantizes the discrete estimated value q _est of the framing parameter inputted from the estimation unit 170 into a continuous estimated value g _est . Then, the dequantization unit 171 outputs the dequantized continuous estimated value _gest of the framing parameter to the error evaluation unit 173.

Similar to the dequantization unit 21 in FIG. 3, the dequantization unit 172 dequantizes the discrete value q of the framing parameter inputted from the quantization unit 16 in FIG. 2 into a continuous value g. Then, the dequantization unit 172 outputs the dequantized continuous value g of the framing parameter to the error evaluation unit 173.

Note that when the estimation section 170 directly outputs the continuous estimated value _gest of the framing parameter, the learning section 17 does not need to include the inverse quantization section 171 and the inverse quantization section 172. In this case, the estimation unit 170 may output the continuous estimated value g _est of the framing parameter to the error evaluation unit 173 and input the continuous value g of the framing parameter from the conversion unit 15 in FIG. 2 to the error evaluation unit 173.

The error evaluation unit 173 calculates the error between the continuous estimated value g _est of the framing parameter input from the inverse quantization unit 171 and the continuous value g of the framing parameter input from the inverse quantization unit 172 . For example, the error evaluation unit 173 calculates a sum of squares error and a cross entropy error. The error evaluation unit 173 then outputs the calculated error to the parameter update unit 174.

The parameter update unit 174 generates a learning signal from the error between the continuous estimated value g _est of the framing parameter calculated by the error evaluation unit 173 and the continuous value g of the framing parameter, and updates the learning parameter in the parameter storage unit 175. be. That is, the parameter update unit 174 updates the learning parameters in the parameter storage unit 175 by back propagation so that the error calculated by the error evaluation unit 173 is minimized.

The parameter storage unit 175 is a storage device that stores learning parameters updated by the parameter update unit 174 as teacher data. The learning parameters in the parameter storage unit 175 are referenced by the estimation unit 170.

In this way, since the learning unit 17 does not need to convert the estimated value q _est of the framing parameter outputted by the estimating unit 170 into the camera coordinate system, the amount of calculation can be suppressed.

(Third embodiment)
An example of the configuration of the learning section 17B according to the third embodiment will be described with reference to FIG. 12.
The learning unit 17 according to the second embodiment evaluates the error between the continuous estimated value g _est of the framing parameter and the continuous value g of the framing parameter in the world coordinate system. On the other hand, the learning unit 17B according to the third embodiment differs from the second embodiment in that it evaluates the error between the estimated value θ _est of the camera parameter and the camera parameter θ in the camera coordinate system.

As shown in FIG. 12, the learning unit 17B includes an estimation unit 170, an inverse quantization unit 171, an inverse quantization unit 172, a parameter update unit 174, a parameter storage unit 175, an inverse transformation unit 176, and an error and an evaluation section 177.
Note that the configurations other than the inverse transform unit 176 and the error evaluation unit 177 are the same as those in the second embodiment, and therefore their description will be omitted.

The inverse transformer 176, like the inverse transformer 22 in FIG. 3, inversely transforms the continuous estimation value _gest of the framing parameter inputted from the inverse quantizer 171 into the camera parameter θ _est . Then, the inverse transformer 176 outputs the inversely transformed estimated value θ _est of the camera parameters to the error evaluation unit 177 .

The error evaluation unit 177 calculates the error between the camera parameter estimate θ _est input from the inverse transformation unit 176 and the camera parameter θ input from the camera parameter reading unit 14 in FIG. For example, the error evaluation unit 177 calculates a sum of squares error and a cross entropy error. Then, the error evaluation section 177 outputs the calculated error to the parameter updating section 174.

In this way, the learning unit 17B evaluates the error using the camera coordinate system in which the cameraman determines the composition, so that the estimation accuracy can be improved.

(Modified example)
Although each embodiment of the present invention has been described in detail above, the present invention is not limited to each of the above-described embodiments, and includes design changes within a range that does not depart from the gist of the present invention.
In the second embodiment and the third embodiment, the specific configuration of the learning section has been described, but the present invention is not limited to this.

In each of the embodiments described above, a soccer game video was explained as an example, but the present invention is not limited to this. In particular, the present invention is preferable because highly accurate camera parameters can be estimated if the video has a correlation between the camera work of the cameraman and the position of the subject. For example, the present invention is most suitable for video of games such as basketball and rugby, and video of concerts and performances.
Furthermore, in the present invention, when the world coordinate system is common, a trained inference model can be used as is. For example, if it is a soccer field that complies with international standards, the same world coordinate system can be set, so a trained inference model can be used as is.

In the first embodiment described above, the reading time of camera parameters is explained as being offset, but the present invention is not limited to this. That is, the offset time may be set to 0. In this case, the learning unit learns by associating situation data and framing parameters at the same time.
In the first embodiment described above, an example of a conversion rule and an inverse conversion rule, and an example of a quantization rule and an inverse quantization rule were explained, but the present invention is not limited thereto.

In each of the embodiments described above, the framing region learning device and the framing region estimating device are described as independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program that causes hardware resources such as a CPU, memory, and hard disk included in a computer to operate as each of the above-described devices. These programs may be distributed via communication lines, or may be written and distributed on recording media such as CD-ROMs and flash memories.

1 Framing area learning device 10A Situation data storage unit (subject information storage unit)
10B Camera parameter storage section 11 Data selection section 12 Time adjustment section 13 Situation data reading section (subject information reading section)
14 Camera parameter readout section 15 Transformation section 16 Quantization section 17 Learning section 2 Framing region estimation device 20 Estimation section 21 Inverse quantization section 22 Inverse transformation section 170 Estimation section 171 Inverse quantization section 172 Transformation section 173, 177 Error evaluation section 174 Parameter update unit 175 Parameter storage unit 176 Inverse conversion unit

Claims (10)

Converting camera parameters indicating the attitude and angle of view of the camera into framing parameters indicating a framing area to be framed by the camera, and associating the framing parameters with subject information indicating at least one of the position or speed of the subject. A framing area learning device for learning,
a conversion unit that converts the camera parameters acquired in synchronization with the subject information into the framing parameters based on a preset conversion rule;
a learning unit that generates a learned inference model of the framing parameter by learning the subject information and the framing parameter in association with each other;
A framing area learning device comprising: a subject information storage unit that stores the subject information in advance;
a subject information reading unit that reads out the subject information at a predetermined time from the subject information storage unit;
a camera parameter storage section that stores the camera parameters in advance;
further comprising a camera parameter readout unit that reads out the camera parameters at a future time offset by a preset offset time with respect to the readout time of the subject information;
The conversion unit converts the camera parameters read by the camera parameter reading unit into framing parameters at the future time,
2. The framing area learning device according to claim 1, wherein the learning unit learns the object information at the read time and the framing parameter at the future time in association with each other. 2. The object information includes at least one of a position map that visualizes the position of the object in a world coordinate system, and a speed map that images the speed of the object in the world coordinate system. Or the framing area learning device according to claim 2. 4. The framing region learning device according to claim 3, wherein the learning unit uses a convolutional neural network as an inference model. a quantization unit that inputs the framing parameter as a continuous value on the world coordinate system from the conversion unit and quantizes the framing parameter into discrete values on the world coordinate system based on a preset quantization rule; Be even more prepared,
The framing area learning device according to any one of claims 1 to 4, wherein the learning unit learns by associating the discrete value of the framing parameter with the subject information. A framing area estimating device that estimates a framing parameter indicating a framing area of a camera using a learned inference model generated by the framing area learning device according to any one of claims 1 to 5,
an estimation unit that receives subject information indicating at least one of the position or speed of the subject and outputs an estimated value of the framing parameter according to the subject information using the learned inference model;
an inverse transformation unit that inversely transforms the estimated value of the framing parameter outputted by the estimation unit into the estimated value of the camera parameter indicating the attitude and angle of view of the camera, based on a preset inverse transformation rule;
A framing area estimation device comprising: 6. The object information includes at least one of a position map that represents the position of the object in a world coordinate system, and a speed map that represents the speed of the object in the world coordinate system. The framing area estimation device described in . The estimation unit outputs the estimated value of the framing parameter as a discrete value on a world coordinate system,
A claim further comprising: an inverse quantization unit that inversely quantizes the estimated value of the framing parameter input from the estimation unit into continuous values on the world coordinate system based on a preset inverse quantization rule. The framing area estimating device according to claim 6 or claim 7. A program for causing a computer to function as the framing area learning device according to any one of claims 1 to 5. A program for causing a computer to function as the framing area estimating device according to any one of claims 6 to 8.

Images