JP7614853B2 - Framing evaluation device and program - Google Patents

Description

The present invention relates to a framing evaluation device, and in particular to a framing evaluation device and program for evaluating the framing of a broadcast camera.

Conventionally, in television broadcasts of live sports such as soccer, the target scene is shot by multiple cameras. For example, Patent Document 1 describes a system that generates free viewpoint video of a sports stadium such as soccer and distributes the generated free viewpoint video. In this system, multiple imaging devices generate multi-viewpoint video by shooting the same scene from different viewpoints, and image recognition is used to generate information such as whether the current scene is a shooting scene, and a camerawork video is generated by setting the camera gaze point so that the goal is included in the video.

International Publication No. 2018/030206

With conventional technology, when a scene is captured by multiple cameras, it is not possible to evaluate the degree to which the framing area of the main camera matches the framing area of the sub-camera.

The present invention has been made in consideration of the above-mentioned situation, and aims to provide a framing evaluation device and program that can evaluate the degree to which the framing area of a camera to be evaluated matches the framing area of a reference camera.

In order to achieve the above object, a framing evaluation device according to the present invention includes a framing area determination unit that calculates world coordinate values of each vertex for determining a boundary of a framing area of a first camera, using ROI information indicating position information of a predetermined region of interest expressed in a world coordinate system and camera parameters of a first camera that is a reference for photographing the region of interest, and a framing area determination unit that calculates world coordinate values of each vertex for determining a boundary of a framing area of the first camera, using the camera parameters of a second camera to be evaluated, by projecting a predetermined position in the world coordinate system onto an image coordinate system of the second camera. the framing area vertex calculation unit converts image coordinate values of each vertex in the image coordinate system of the second camera into image coordinate values; a framing area vertex selection unit forms a projection area by selecting a vertex constituting the outermost corner from among the vertices projected onto the image coordinate system of the second camera and connecting the vertices constituting the outermost corners; a projection area calculation unit calculates a projection area, which is the area of the projection area; and a framing efficiency calculation unit determines the area of an overlap area where the image frame of the second camera and the projection area overlap, and calculates a framing efficiency that represents the ratio of the area of the overlap area to the projection area and the area of the image frame of the second camera.
The present invention can also be realized by a program for causing a computer to function as the framing evaluation device described above.

The present invention can evaluate the degree to which the framing area of the second camera being evaluated matches the framing area of the first camera that serves as a reference.

1 is a block diagram illustrating a configuration of a framing evaluation device according to an embodiment of the present invention. 1A is a block diagram showing a schematic configuration of a camera, in which FIG. 1A shows a first camera serving as a reference, and FIG. 1B shows a second camera to be evaluated. 3A and 3B are schematic diagrams of camera coordinate systems, in which FIG. 3A shows the camera coordinate system of a first camera, and FIG. 3B shows the camera coordinate system of a second camera. 1A and 1B are schematic diagrams of a soccer field on which an ROI is set, in which FIG. FIG. 5 is a perspective view showing an example of an ROI set on the soccer field of FIG. 4. FIG. 6 is a projection of the ROI of FIG. 5 onto the field ground. An explanatory diagram of an example of projection, where (a) shows the image coordinate system on the image frame of the second camera, (b) shows the projection of the intersection of the line of sight vector and height 0, and (c) shows the projection of the intersection of the line of sight vector and height 0 and height h. FIG. 7B is a diagram showing a polygon obtained by connecting the vertices of the outermost corners of FIG. FIG. 9 is a diagram showing an overlapping area between the polygon in FIG. 8 and the image frame. FIG. 13 is an explanatory diagram of another example of projection. FIG. 11 is a diagram showing a polygon obtained by connecting the vertices of the outermost corners of FIG. 10 . 13 is a diagram showing another example of an overlapping area between a polygon obtained by connecting the vertices of the outermost corners and an image frame. FIG. 4 is a flowchart showing a flow of processing of the framing evaluation device according to the embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First, an example of a camera related to a framing evaluation device 1 (FIG. 1) according to this embodiment will be described, and then the configuration of the framing evaluation device 1 will be described.

[camera]
2, a reference camera and a camera to be evaluated will be described. The first camera 2 is a reference camera when evaluation is performed by the framing evaluation device 1. The first camera 2 is, for example, a camera configured to be remotely controlled.
As shown in Fig. 2(a), the first camera 2 is equipped with a drive mechanism 21, a pan/tilt data detection sensor 22, and a lens data detection sensor 23. The drive mechanism 21 is composed of a camera platform on which the main body of the first camera 2, such as the camera lens, is mounted. Pan, tilt, zoom, and focus are controlled by this drive mechanism 21. Note that the zoom and focus controls are performed by an internal mechanism of the first camera 2, and will be described as being included in the drive mechanism 21.

The pan/tilt data detection sensor 22 is a sensor for measuring the attitude of the first camera 2, and includes a pan angle sensor for detecting the pan angle of the camera head to which the first camera 2 is fixed, and a tilt angle sensor for detecting the tilt angle of the first camera 2. Here, the pan angle sensor and tilt angle sensor can be configured, for example, by a rotary encoder, potentiometer, or gyro sensor attached to the camera head.

The lens data detection sensor 23 is a sensor for measuring the angle of view and focal position of the lens of the first camera 2 (not shown), and includes a zoom sensor that detects the zoom amount (angle of view) of the first camera 2, and a focus sensor that detects the focal position (focus) of the lens built into the first camera 2. Here, the zoom sensor and focus sensor can be configured by a method that reads the rotation angles of the zoom ring and focus ring with a rotary encoder and a potentiometer. Alternatively, the zoom sensor and focus sensor can be a linear sensor installed on the lens sliding portion of the first camera 2.

The second camera 3 is a camera that is the subject of evaluation when evaluation is performed by the framing evaluation device 1. The second camera 3 is, for example, a camera that is configured to be remotely controlled. As shown in FIG. 2(b), the second camera 3 is equipped with a drive mechanism 31, a pan/tilt data detection sensor 32, and a lens data detection sensor 33. These components are similar to the drive mechanism 21, pan/tilt data detection sensor 22, and lens data detection sensor 23 of the first camera 2, so a description thereof will be omitted.

The camera parameters of the first camera 2 (hereinafter, first camera information) correspond to external parameters and internal parameters, and are information required to determine the composition of the camera. The external parameters are, for example, parameters such as the camera position (position of the camera's viewpoint) and the camera attitude (pan angle and tilt angle), and are used for conversion from world coordinates to camera coordinates. The internal parameters are parameters that do not change even if the camera moves, such as focal length and lens distortion, and are used for conversion from camera coordinates to image coordinates.
Similarly, the camera parameters of the second camera 3 (hereinafter, second camera information) correspond to external parameters and internal parameters, and are information required to determine the camera composition.

The first camera 2 has a planar image sensor 25 as shown in FIG. 3(a) and outputs what is projected onto it as an image. Similarly, the second camera 3 has a planar image sensor 35 as shown in FIG. 3(b) and outputs what is projected onto it as an image. The image sensors 25 and 35 are, for example, CMOS (Complementary Metal Oxide Semiconductor) imaging elements or CCD (Charge Coupled Device) imaging elements.

The visual volume (all framing areas) of the first camera 2 can be seen as a space surrounded by line-of-sight vectors 26, 27, 28, and 29 passing through the four corners of the image sensor 25 from the viewpoint 24 shown in FIG. 3A. Similarly, the visual volume (all framing areas) of the second camera 3 can be seen as a space surrounded by line-of-sight vectors 36, 37, 38, and 39 passing through the four corners of the image sensor 35 from the viewpoint 34 shown in FIG. 3B . Note that the tips of the line-of-sight vectors are omitted in FIG. 3. The camera coordinate system for each camera is a right-handed system in which the optical axis direction indicated by the white arrow in FIG. 3 is the + _zc axis, and xc, _yc , _{and zc} are in that order. The position of the viewpoint 24 of the first camera 2 is the position of the origin of the camera coordinate system of the first camera 2, and the position of the viewpoint 34 of the second camera 3 is the position of the origin of the camera coordinate system of the second camera 3.

[Configuration of the framing evaluation device]
The configuration of a framing evaluation device 1 will be described with reference to FIG.
The framing evaluation device 1 is a device that evaluates how well the framing area of a camera to be evaluated matches the framing area of a reference camera. The framing evaluation device 1 includes a framing area determination unit 10, a framing area vertex calculation unit 11, a framing area vertex selection unit 12, a projected area calculation unit 14, and a framing efficiency calculation unit 15. Here, the framing evaluation device 1 further includes a storage means 9, an intersection determination unit 13, a framing efficiency evaluation unit 16, and a framing efficiency display unit 17. The configuration of each unit will be described in order below.

(Storage means)
The storage means 9 stores information, calculation results, operation programs, etc. used in each part of the framing evaluation device 1, and is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a memory. The information can include, for example, ROI information, first camera information, second camera information, etc., which will be described later. These pieces of information are inputted through an input device 4 such as a keyboard, a mouse, or a touch panel, or a predetermined wireless or wired communication means 6. In addition, these pieces of information and calculation results are outputted to a display device 5 such as a liquid crystal display, etc., as necessary, or outputted to an external device via the communication means 6. The display device 5 may be an external device.

(Framing area determination unit)
The framing region determination unit 10 uses ROI information indicating position information of a predetermined region of interest expressed in a world coordinate system and camera parameters of the first camera 2 serving as a reference for photographing the region of interest to calculate world coordinate values of each vertex for determining the boundary of the framing region of the first camera 2. The processing by the framing region determination unit 10 is a calculation processing related to the first camera 2 serving as a reference. The processing by the framing region determination unit 10 will be described in detail below with reference to the mathematical expressions (1) to (6) described later and the drawings in FIG. 3 to FIG. 6.

As an example, one of multiple cameras used on a soccer field (see FIG. 4) for television broadcasting of a soccer game is set as the reference first camera 2 (see FIG. 2(a)), and another camera also used on the soccer field is set as the evaluation target second camera 3 (see FIG. 2(b)). After explaining the definitions of the world coordinate system and camera coordinate system in this case, the processing of the framing area determination unit 10 will be explained. Note that the framing evaluation device 1 is installed in, for example, a broadcast van. The broadcast van is also equipped with a CCU (camera control unit) and other components that are connected to the camera via an optical cable. The camera's remote control device is connected to the CCU via a cable.

4 to 6 are diagrams for explaining the relationship between the world coordinate system in which the subject exists and the camera coordinate system in which the position and attitude of the camera are reflected, using a soccer field 101 as an example. As shown in FIG. 4(a), the world coordinate system has the center 102 of the soccer field as the origin, the axis passing through the origin and parallel to the sidelines 104 and 105 as the Xw _axis , the axis parallel to the center line 103 as the _Yw axis, and the axis perpendicular to the field plane and pointing upward as the + _Zw axis. The world coordinate system is a right-handed system with the Xw _axis , Yw _axis , and _Zw axis in that order. The field plane is a plane at the same height as the ground surface, and will be referred to as the plane 108 with _Zw =0 hereinafter.

Here, the range between an infinitely wide plane of Z _w = 0 in the world coordinate system and an infinitely wide plane of Z _w = h parallel to it is defined as the ROI (Region of Interest). In other words, the ROI is the range between plane 108 of Z _w = 0 and a plane (plane 109 of Z _w = h) at a height h from plane 108 of Z _w = 0, as shown in Fig. 4B.

The ROI can be arbitrarily defined by the user, and may be defined in a plane or in a solid. The range may also be finite. When the ROI is finite, for example, if the vertices of the ROI are defined so as not to be included in the framing area, the line of sight vector will intersect with the ROI unless it is parallel to the boundary surface of the ROI. The area having the world coordinate value of this intersection and surrounded by all the intersections (area not including the intersections) can be set as the framing area of the camera. As another example, if the vertices of the ROI are included in the framing area, the included vertices can be seen as the intersections of the line of sight vector and the ROI. In this way, the framing area of the camera forms a plane or a solid. The framing area of the camera is the area where the view volume of the camera and the ROI overlap.

Here, the actual cameraman (photographer) who operates the first camera 2 cuts out the frame while recognizing what kind of play it is, and determines the framing area of the first camera 2. The photographer who operates the second camera 3 operates the second camera 3 so that the framing area of the second camera 3 approaches the framing area of the first camera 2, which serves as the reference. The first camera 2 photographs the subject from the sideline 105 side. The second camera 3 photographs the subject from a different location, the goal line 107 side. The relationship between the ROI, the world coordinate system, and the camera coordinate systems of each camera at this time is shown in Figure 5, and the relationship between the camera coordinate system of the first camera 2 and the world coordinate system is shown in Figure 6. Figure 6 is a projection of the ROI of Figure 5 onto the ground of the field.

For example, with reference to FIGS. 3(a) and 4(a), if the origin (viewpoint 24) of the camera coordinate system of the first camera 2 coincides with the origin of the world coordinate system (center 102 of the soccer field) and if there is no rotation about any axis, the + _xc axis of the camera coordinate system coincides with the + _Xw axis of the world coordinate system, the + _yc axis of the camera coordinate system coincides with the _−Zw axis of the world coordinate system, and the + _zc axis of the camera coordinate system coincides with the + _Yw axis of the world coordinate system.

Next, the rotation of each axis is considered for the first camera 2, for example. Specifically, the camera attitude in the camera coordinate system of the first camera 2, in which the rotation angle of the _yc axis is α [rad] (pan angle), the rotation of the _xc axis is δ [rad] (tilt angle), and the rotation of the _zc axis is φ [rad] (roll angle), can be transformed into the camera attitude in the world coordinate system by the rotation matrix R shown in the following formula (1). Note that a commonly used camera platform does not have a rotation mechanism for manipulating the roll angle φ, and in that case, the roll angle φ may be set to 0.

For example, assume that the position of the origin of the camera coordinate system of the first camera 2, i.e., the position of the viewpoint 24 of the first camera ₂ , when described in the world coordinate system, is expressed by the following equation (2). Also assume that the position of the origin of the world coordinate system, i.e., the center 102 of the soccer field, when described in the camera coordinate system, is expressed by the following equation (3). In this case, the relationship between _{the position vector tC of} the origin of the world coordinate system described in the camera coordinate system and the viewpoint position vector _tW of the camera described in the world coordinate system is expressed by the following equation (4) using the rotation matrix R of the above equation (1).

Assuming the above definitions of the world coordinate system and the camera coordinate system, the framing region determination unit 10 performs the following calculation process. Here, as shown in Fig. 5, the ROI (region of interest) is defined as a range between a plane 108 of _Zw = 0 and a plane 109 of _Zw = h in the world coordinate system. Therefore, the framing region determination unit 10 calculates the coordinates of each vertex of the framing region by determining a total of eight intersections between the four line-of-sight vectors 26, 27, 28, and 29 passing through the four corners of the image sensor (image pickup element) 25 of the first camera 2, the plane 108 of _Zw = 0, and the plane 109 of _Zw = h.

Specifically, framing region determination unit 10 obtains the world coordinate values of the intersection point between the line of sight vector and the boundary surface of the ROI as follows: The world coordinate values of the intersection point to be obtained on the boundary surface of the ROI, for example, on plane 108 where _Zw = 0, are set to [ _Xw , _Yw ]. The relational equation for conversion from the image coordinate system to the world coordinate values [ _Xw , _Yw ] on plane 108 where _Zw = 0 is expressed by the following equation (5).

Here, as shown in FIG. 7A, the image coordinate system has the upper left corner of the image as the origin _OI, the horizontal right direction as the + _xI axis, and the vertical down direction as the + _yI axis. In formula (5), r _xx , r _yx , r _zx , r _xy , r _yy , and r _zy are components of the rotation matrix of formula (1) described above. _{t cx} , t _cy , and t _cz are components of the position vector t _C of formula (3) described above. _{x I} and y _I are coordinate values of image grid points, and represent the coordinate values of a large number of image grid points. _{u 0} indicates half the number of horizontal pixels, and v ₀ indicates half the number of vertical pixels. f indicates the focal length in pixels. The focal length in pixels is a parameter obtained by normalizing the focal length in mm by the number of pixels of the camera sensor. For example, when the number of horizontal pixels is 1920 and the horizontal angle of view is 60°, the focal length in pixels is 1662.8 [pixels].

In short, framing region determination unit 10 obtains the world coordinate value [Xw, _Yw , 0] of the intersection between the line of sight vector and plane 108 of _Zw = 0 in the ROI by inputting parameters required for the relational expression of equation (5). In Fig. 5, intersection 41 is the intersection between line of sight vector 29 and plane 108. Intersection 42 is _the intersection between line of sight vector 26 and plane 108. Intersection 43 is the intersection between line of sight vector 27 and plane 108. Intersection 44 is the intersection between line of sight vector 28 and plane 108.

Further, the method by which the framing region determination unit 10 obtains the world coordinate value [ _Xw , _Yw , h] of the intersection point of the line of sight vector and the plane 109 of _Zw =h in the _ROI can be calculated in a manner generally similar to the method of obtaining the world coordinate value [ _Xw , _Yw , 0] of the intersection point of the plane 108 of Zw=0, so detailed description will be omitted. In FIG. 5, intersection 51 is the intersection point of the line of sight vector 29 and the plane 109. Intersection 52 is the intersection point of the line of sight vector 26 and the plane 109. Intersection 53 is the intersection point of the line of sight vector 27 and the plane 109. Intersection 54 is the intersection point of the line of sight vector 28 and the plane 109. In this case, the following formula (6) is used instead of the above formula (2) as the position vector t _w when the position of the viewpoint 24 of the first camera 2 is described in the world coordinate system.

Returning to FIG. 1, the description of the configuration of the framing evaluation device 1 will continue.
(Framing area vertex calculation unit)
The framing area vertex calculation unit 11 converts the world coordinate values of each vertex that determines the boundary of the framing area of the first camera 2 into the image coordinate values of each vertex in the image coordinate system of the second camera 3 by projecting a predetermined position in the world coordinate system onto the image coordinate system of the second camera 3 using the camera parameters of the second camera 3 to be evaluated. Here, the framing area vertex refers to the vertex that forms the framing area projected onto the image coordinate system related to the second camera 3 to be evaluated. The framing area vertex calculation unit 11 calculates the coordinate values of the framing area vertices in the image coordinate system by the following calculation.

The framing area vertex calculation unit 11 receives the world coordinate values of the intersections of the line of sight vector and planes 108, 109 in the ROI, determined by the framing area determination unit 10, as vertices of the framing area of the reference first camera 2. In addition, the framing area vertex calculation unit 11 receives the internal and external parameters of the second camera 3 to be evaluated as second camera information.

In the calculations of the framing area vertex calculation unit 11, the conversion formula from coordinate values ( _Xw , _Yw , _Zw ) in the world coordinate system to coordinate values ( _xI , _yI ) in the image coordinate system relating to the position of the viewpoint 34 of the second camera 3 can be, for example, the following formula (7).

In formula (7), r _xx , r _yx , r _zx , r _xy , r _yy , r _zy , r _xz , r _yz , and r _zz are components of the rotation matrix R in formula (1) above. _{t wx} , t _wy , and t _wz are components of the position vector t _w in formula (2) above. Here, formulas (1) and (2) representing the rotation matrix and the viewpoint position, respectively, are used that are obtained for the second camera 3. Similarly, x _I and y _I are the coordinate values of the image grid point of the image sensor 35 of the second camera 3. u ₀ and v ₀ indicate half the value of the number of horizontal and vertical pixels for the image sensor 35. f indicates the focal length in pixels for the image sensor 35.

In this embodiment, the coordinate values ( _Xw , _Yw , _Zw ) in the world coordinate system on the right-hand side of equation (7) represent combinations of coordinate values for a total of eight intersections, intersections 41 to 44 and intersections 51 to 44, shown in Fig. 5. Framing area vertex calculation unit 11 outputs, as the calculation results, image coordinate values indicating the positions of the vertices in the image coordinate system of second camera 3 to framing area vertex selection unit 12.

(Framing area vertex selection section)
Framing area vertex selection unit 12 forms a projection area by selecting vertices that form the outermost corners from among the vertices projected onto the image coordinate system of second camera 3 and connecting the vertices that form the outermost corners. Image coordinate values indicating the positions of the vertices in the image coordinate system of second camera 3 are input to framing area vertex selection unit 12 from framing area vertex calculation unit 11. The reason why framing area vertex selection unit 12 selects the vertices that form the outermost corners to form a projection area is so that the area of the projection area can be calculated in projected area calculation unit 14 at a subsequent stage.

For example, when an ROI is set in the area sandwiched between plane 108 of Z _w = 0 and plane 109 of Z _w = h as shown in Figures 4 to 6, framing area vertex selection unit 12 determines the vertices that constitute the outermost corners by making the following judgment in the image coordinates of the point (vertex) where the intersection (vertex) of the framing area of first camera 2 is projected onto image sensor 35 of second camera 3:

When a vertex of an intersection in either one of the planes _Zw =0 and _Zw =h projected onto the image coordinate system of the second camera 3 is regarded as a vertex to be determined, if the coordinates of the vertex to be determined are outside the area surrounded by the four vertices whose four intersections in the other plane are projected onto the image coordinate system of the second camera 3, the framing area vertex selection unit 12 performs a determination process to determine that the vertex to be determined is the vertex of the outermost corner. The framing area vertex selection unit 12 determines the vertices that constitute the outermost corners by performing this determination process on each intersection in both the planes _Zw =0 and _Zw =h.

The processing of the framing area vertex selection unit 12 will be described with reference to Fig. 7. Fig. 7(a) is a diagram showing an image frame 30 of the second camera 3 by hatching with dots. An image coordinate system is set above the image frame 30. In the image coordinate system of the second camera 3, the upper left corner of the image is the origin _OI , the horizontal right direction is the + _xI axis, and the vertical downward direction is the + _yI axis.

Fig. 7(b) is a schematic diagram showing a rectangular contour line connecting points (vertices 61 to 64) where intersections 41 to 44 between four line-of-sight vectors passing through the four corners of image sensor 35 shown in Fig. 5 and plane 108 of Z _w = 0 are projected onto the image coordinate system of second camera 3. Fig. 7(c) is a schematic diagram showing another rectangular contour line superimposed on the contour line of Fig. 7(b). This other rectangular contour line in Fig. 7(c) is formed by connecting points (vertices 71 to 74) where intersections 51 to 54 between four line-of-sight vectors passing through the four corners of image sensor 35 shown in Fig. 5 and plane 109 of Z _w = h are projected onto the image coordinate system of second camera 3.

For example, when the vertex 61 (FIG. 7C) obtained by projecting the intersection 41 on the plane 108 of Z _w =0 onto the image coordinate system of the second camera 3 is regarded as the vertex to be determined, the framing area vertex selection unit 12 determines whether or not the coordinates of the vertex 61 are outside the area surrounded by the four vertices 71 to 74 obtained by projecting the four intersections 51 to 54 on the plane 109 of Z _w =h onto the image coordinate system of the second camera 3. In the example shown in FIG. 7C, it is clear that the coordinates of the vertex 61 are outside the area surrounded by the vertices 71 to 74. In this case, the framing area vertex selection unit 12 determines that the vertex 61 is the vertex of the outermost corner. On the other hand, in the example shown in FIG. 7C, it is clear that the coordinates of the vertex 63 are within the area surrounded by the vertices 71 to 74. In this case, the framing area vertex selection unit 12 determines that the vertex 63 is not the vertex of the outermost corner.

Similarly, when vertex 71 ( FIG. 7( c )) obtained by projecting intersection 51 on plane 109 of Z _w = h onto the image coordinate system of second camera 3 is regarded as the vertex to be determined, framing region vertex selection unit 12 determines whether or not the coordinates of vertex 71 are outside the area surrounded by four vertices 61 to 64 obtained by projecting four intersections 41 to 44 on plane 108 of Z _w = 0 onto the image coordinate system of second camera 3. In the example shown in FIG. 7( c ), it is clear that the coordinates of vertex 71 are within the area surrounded by vertices 61 to 64. In this case, framing region vertex selection unit 12 determines that vertex 71 is not the vertex of the outermost corner.

In the example shown in FIG. 7(c), the framing area vertex selection unit 12 forms the projection area 80 shown in FIG. 8 by connecting, in this order, vertices 61, 62, 72, 73, 74, and 64 that have been determined as the outermost vertices as a result of the determination process for the eight vertices to be determined. Information on the coordinates of each vertex that forms the contour of this projection area 80 is output to the intersection determination unit 13, the projected area calculation unit 14, and the framing efficiency calculation unit 15. Note that in this example, the total number of vertices that form the contour of the projection area 80 will be anywhere between 4 and 8.

(Intersection determination unit)
The intersection determination unit 13 determines whether there are two or more intersections between the contour of the projection area 80 and the contour of the image frame 30 of the second camera 3. If there are two or more intersections, the intersection determination unit 13 outputs a calculation instruction to the framing efficiency calculation unit 15. In this case, the framing efficiency calculation unit 15 calculates the framing efficiency, which will be described later.

Coordinate information of each vertex forming the contour of the projection area 80 is input to the intersection determination unit 13 from the framing area vertex selection unit 12. Using this input coordinate information of each vertex, the intersection determination unit 13 uses a predetermined algorithm to find intersections between the contour of the projection area 80 and the contour of the image frame 30. For example, for all vertices, linear functions of line segments connecting two adjacent vertices are found, and the intersections with the four sides forming the contour of the image frame 30 are found. The coordinate information of these intersections is output to the framing efficiency calculation unit 15.

With regard to the coordinate information of the contour lines of the image frame 30, for example, among the contour lines of the image frame 30, since the upper side in Fig. 7(a) is on the _xI axis, the _yI coordinate of the contour line corresponding to the upper side is 0. Since the left side in Fig. 7(a) is on the _yI axis, the _xI coordinate of the contour line corresponding to the left side is 0. Also, the _yI coordinate of the contour line corresponding to the lower side and the _xI coordinate of the contour line corresponding to the right side in Fig. 7(a) can be defined from the number of vertical pixels and the number of horizontal pixels on the image sensor 35 of the second camera 3.

Now, refer to FIG. 9. In FIG. 9, the overlapping area 90 between the projection area 80 and the image frame 30 of the second camera 3 is hatched downward to the left. In the example shown in FIG. 9, there are a total of four intersections between the contour of the projection area 80 and the contour of the image frame 30 of the second camera 3: intersection 92, intersection 93, intersection 94, and intersection 95. Therefore, in this case, the intersection determination unit 13 determines that there are two or more intersections, and outputs a calculation instruction to the framing efficiency calculation unit 15.

If there is absolutely no overlapping area between the projection area 80 and the image frame 30 of the second camera 3, the intersection determination unit 13 determines that there are not two or more intersections.
Furthermore, if one vertex (e.g., the lower right corner) of the projection area 80 coincides with one vertex (e.g., the upper left corner) of the image frame 30 of the second camera 3, the intersection determination unit 13 determines that there are not two or more intersections.
If, as a result of the calculation by the previous stage framing area vertex calculation unit 11, all of the vertices projected onto the image coordinate system of the second camera 3 are included inside the contour line of the image frame 30 of the second camera 3 as shown in Fig. 10, the entire projected area 80 will be included inside the contour line of the image frame 30 of the second camera 3 as shown in Fig. 11. In this case, the intersection determination unit 13 determines that the number of intersections is two or more.
12, there are a total of two intersections between the contour of the projection region 80 and the contour of the image frame 30 of the second camera 3, namely, intersection 91 and intersection 92. In this case as well, the intersection determination unit 13 naturally determines that the number of intersections is two or more.

(Projection area calculation unit)
The projected area calculation unit 14 calculates the projected area, which is the area of the projection area 80. The projected area calculation unit 14 receives coordinate information of each vertex forming the outline of the projection area 80 from the framing area vertex selection unit 12. The projected area calculation unit 14 calculates the projected area using a known formula for the area of a polygon based on the input coordinate information of each vertex. Hereinafter, the projected area calculated by the projected area calculation unit 14 is referred to as the projected area S _A. The coordinate projected area S _A can be calculated, for example, by the following formula (8), where the coordinate values of each vertex are x _k , y _k and the number of vertices is n.

However, _xn+1 = _x1 , _yn+1 = _y1 , and the vertices are arranged in a clockwise direction when the image coordinate system is set as shown in Fig. 8. In the example shown in Fig. 8, the number of vertices n is 6, and if the first vertex is vertex 61, the order is vertex 61, vertex 62, vertex 72, vertex 73, vertex 74, vertex 64, and vertex 61 again.

(Framing efficiency calculation unit)
The framing efficiency calculation unit 15 determines the area of the overlap area 90 where the image frame 30 of the second camera 3 overlaps with the projection area 80, and calculates the framing efficiency, which represents the ratio of the area of the overlap area 90 to the projection area S _A and the area of the image frame 30 of the second camera 3.

In this embodiment, the framing efficiency calculation section 15 includes an overlap area calculation section 15a, a frame area calculation section 15b, and a framing efficiency calculation section 15c.
<Overlap area calculation section>
As shown in FIG. 9, the overlap area calculation unit 15a calculates an overlap area, which is an area of an overlap area 90 existing within the image frame 30 of the second camera 3, of the projection area 80. The overlap area calculation unit 15a receives information on the coordinates of each vertex forming the contour of the projection area 80 from the framing area vertex selection unit 12, and receives information on the coordinates of the intersections from the intersection determination unit 13. The overlap area calculation unit 15a calculates the overlap area using a predetermined algorithm based on the input information on the coordinates of the vertices and intersections. For example, the overlap area can also be calculated using a calculation method similar to that of the projection area calculation unit 14. Hereinafter, the overlap area calculated by the overlap area calculation unit 15a is referred to as an overlap area S _C. Note that, when the entire projection area 80 is included within the contour of the image frame 30 as shown in FIG. 11, the overlap area S _C is equal to the projection area S _A.

<Frame area calculation section>
The frame area calculation unit 15b calculates the frame area, which is the area of the image frame 30 of the second camera 3. The frame area calculation unit 15b calculates the frame area by multiplying the number of vertical pixels by the number of horizontal pixels on the image sensor 35 of the second camera 3. Hereinafter, the frame area calculated by the frame area calculation unit 15b will be referred to as frame area S _B.

<Framing efficiency calculation section>
The framing efficiency calculation unit 15c calculates an area coverage ratio R _A expressed by the following formula (9) and an area occupancy ratio R _B expressed by the following formula (10). The framing efficiency calculation unit 15c also calculates a framing efficiency R _AB defined by the following formula (11).

The area coverage ratio _RA indicates the ratio of the overlap area _SC to the projection area _SA . The area occupancy ratio _RB indicates the ratio of the overlap area _SC to the frame area _SB . The framing efficiency _RAB indicates the product of the area coverage ratio _RA and the area occupancy ratio _RB . When the framing efficiency _RAB is 0, it means that the second camera 3 to be evaluated is not able to follow the framing area of the first camera 2, which is the reference, at all. The closer the framing efficiency _RAB is to 1, the more efficiently the second camera 3 can follow the framing area of the first camera 2.
The framing efficiency calculation section 15 outputs the framing efficiency R _AB as a final calculation result to the framing efficiency evaluation section 16 .

(Framing Efficiency Evaluation Unit)
The framing efficiency evaluation unit 16 calculates an evaluation value including a determination result of whether or not the value calculated by the framing efficiency calculation unit 15 is greater than a predetermined threshold. The threshold can be set arbitrarily. The framing efficiency evaluation unit 16 receives the framing efficiency R _AB from the framing area vertex selection unit 12. In addition, the framing efficiency evaluation unit 16 receives a threshold stored in advance in the storage means 9 or a threshold input by an operator from the input device 4 at a predetermined timing.

The framing efficiency evaluation unit 16 judges whether the input framing efficiency R _AB is equal to or greater than the input threshold, and outputs a signal indicating that the framing efficiency R AB is equal to or greater than the threshold, or a signal indicating that the framing efficiency R _{AB is less than the threshold. The framing efficiency evaluation unit 16 may be configured to calculate the difference between the framing efficiency R AB} and the threshold, and output the difference as an evaluation value. For example, if the threshold is 0.97 and the framing efficiency R _AB is 0.78, the framing efficiency evaluation unit 16 outputs a signal indicating that the framing efficiency R AB is less than the threshold, and outputs the difference value -0.19 as a return value to the framing efficiency display unit 17. If a plus or minus sign is added to the difference value, it is clear that the framing efficiency R AB is equal to or greater than the threshold, so the evaluation value may be the framing efficiency R _AB and the signed difference value. The framing efficiency R _AB and the difference value may be output as a percentage, not as a decimal.

(Framing efficiency display)
The framing efficiency display section 17 displays the evaluation value calculated by the framing efficiency evaluation section 16 on the display device 5. The framing efficiency display section 17 receives the evaluation value from the framing efficiency evaluation section 16.

For example, if the first camera 2 or the second camera 3 is a broadcast camera with a viewfinder operated by the photographer, the viewfinder may be the display device 5. In this case, the framing efficiency display unit 17 outputs the framing efficiency R _AB and the difference value (return value) as evaluation values to the display device 5. That is, the framing efficiency R _AB and the difference value, for example, are superimposed and displayed in real time on the video frame of the camera currently displayed by the viewfinder (display device 5) of the camera. For example, if the framing efficiency R _AB and the difference value are superimposed and displayed on the viewfinder of the second camera 3, it is possible to teach the photographer of the second camera 3 how close the current framing area is to the framing area of the first camera 2, which is the reference.

[Operation of the Framing Evaluation Device]
Next, the operation of the framing evaluation device according to the embodiment of the present invention will be described with reference to Fig. 13 (and other drawings as appropriate). First, the framing evaluation device 1 determines the framing area of the first camera 2, which is the reference, using the first camera information and the ROI information by the framing area determination unit 10 (step S1). Then, the framing evaluation device 1 projects the framing area of the first camera 2 by the framing area vertex calculation unit 11 using the second camera information, and calculates the coordinate values of the framing area vertices in the image coordinate system of the second camera 3 by calculation (step S2).

Then, the framing evaluation device 1 selects framing area vertices using the framing area vertex selection unit 12 to form the projection area 80 (step S3). Then, the framing evaluation device 1 determines whether or not there are two or more intersections between the contour of the projection area 80 and the contour of the image frame 30 using the intersection determination unit 13 (step S4). If there are not two or more intersections (step S4: No), the framing evaluation device 1 ends the process. On the other hand, if there are two or more intersections (step S4: Yes), the framing evaluation device 1 calculates the projection area S _A using the projection area calculation unit 14 (step S5).

Then, the framing evaluation device 1 calculates an overlap area S _C of the overlap area 90 that exists within the image frame 30 in the projection area 80 by the overlap area calculation section 15a of the framing efficiency calculation section 15 (step S6). Furthermore, the frame area calculation section 15b of the framing efficiency calculation section 15 calculates a frame area S _B (step S7). The processing of this step S7 may be performed before the processing of step S6. Then, the framing evaluation device 1 calculates a framing efficiency R AB by the framing efficiency calculation section 15c of the framing efficiency calculation section 15 using the projection area S _A , the frame area S _B , and the overlap area S _C according to the above-mentioned formulas (9) to ( ₁₁ ) (step S8).

The framing evaluation device 1 then evaluates the framing efficiency R _AB by determining whether the framing efficiency R _AB is equal to or greater than a threshold value using the framing efficiency evaluation unit 16 (step S9).The framing evaluation device 1 then causes the framing efficiency display unit 17 to display the framing efficiency R _AB and the difference value (return value) as evaluation values on the display device 5 (step S10).

As described above, the framing evaluation device 1 according to this embodiment can quantitatively evaluate how well the framing area of the second camera 3 being evaluated matches the framing area of the first camera 2 being the reference. In addition, since the framing evaluation device 1 calculates the framing efficiency based on the projected area of the framing area of the first camera 2, it is possible to perform the evaluation with less calculations than evaluation using voxels in world coordinates. Furthermore, the framing evaluation device 1 can inform the photographer of the second camera 3 being evaluated of the framing efficiency in real time, thereby instructing the photographer of the second camera 3 how close the current framing area is to the first camera 2 being the reference.

(Modification)
Although the framing evaluation device according to the embodiment of the present invention has been described above, the gist of the present invention is not limited to these descriptions and should be broadly interpreted based on the claims. Furthermore, it goes without saying that various changes and modifications based on these descriptions are also included in the gist of the present invention.

For example, in this embodiment, the framing evaluation device 1 is provided with the intersection determination unit 13, but this is not a required configuration. If the intersection determination unit 13 is not provided, the framing area vertex selection unit 12 outputs information on the coordinates of each vertex forming the contour of the projection area 80 to the projection area calculation unit 14 and the framing efficiency calculation unit 15, respectively. In addition, in order to calculate the overlap area S _C , the framing efficiency calculation unit 15 uses the information on the coordinates of each vertex inputted to obtain the intersection between the contour of the projection area 80 and the contour of the image frame 30 of the second camera 3 using a predetermined algorithm. Note that, for example, when there is no overlapping area between the projection area 80 and the image frame 30 of the second camera 3, it is meaningless to calculate the framing efficiency, and when the intersection determination unit 13 is provided, such meaningless calculations can be prevented, thereby reducing the processing load related to the calculation of the framing efficiency.

In addition, in this embodiment, the framing evaluation device 1 is equipped with a framing efficiency evaluation unit 16 and a framing efficiency display unit 17, but these components are not essential and may be external devices. In this case, the calculation result of the framing efficiency calculation unit 15 may be treated as the evaluation value, or the calculation result of the framing efficiency calculation unit 15 may be output to a framing efficiency evaluation unit or the like as an external device to obtain an evaluation value.

Also, the framing efficiency calculation unit 15 may output the area coverage ratio R _A to the framing efficiency evaluation unit 16 as the broad framing efficiency instead of the framing efficiency R _AB . In this case, a threshold value of the area coverage ratio R _A is input to the framing efficiency evaluation unit 16. Furthermore, the area coverage ratio R _A and the difference value may be superimposed on the video frame displayed on the display device 5. Similarly, the framing efficiency calculation unit 15 may output the area occupancy ratio R _B to the framing efficiency evaluation unit 16 as the broad framing efficiency, and a threshold value of the area occupancy ratio R _B is input to the framing efficiency evaluation unit 16, and the area occupancy ratio R _B and the difference value may be superimposed on the video frame displayed on the display device 5. Furthermore, the framing efficiency calculation unit 15 may be configured to output both the framing efficiency R _AB and the broad framing efficiency to the framing efficiency evaluation unit 16.

Although the first camera 2 is described as photographing the subject from the sideline 105 side, it may be photographed from the sideline 104 side. Although the second camera 3 is described as photographing the subject from the goal line 107 side, it may be photographed from the goal line 106 side. Furthermore, the same effect can be achieved by providing multiple second cameras 3 to be evaluated and installing the second cameras 3 in different locations, such as near the goal line 106, near the goal line 107, and near the sideline 104.

Although an actual cameraman is assigned to both the first camera 2 and the second camera 3, one or both of the cameras may be replaced with a robot camera if the purpose is to reduce the number of people required for actual shooting. For example, if the first camera 2 is operated by an actual cameraman and the second camera 3 is a robot camera, the second camera 3 is unmanned and not operated by a cameraman. In this case, the robot camera (second camera 3) can determine the framing area of the second camera 3 by referring to the framing area of the first camera 2 determined by the actual cameraman. Also, when there are enough people required for actual shooting but some cameramen are not very experienced, it is possible to leave the robot camera to operate the first camera 2 and have the less experienced cameraman operate the second camera 3.

The first camera 2 and the second camera 3 may be fixedly installed or may be configured to be movable. Examples of movable cameras include a camera mounted on a wheeled head, aerial special photography equipment that moves back and forth on a set trajectory along a wire, and a camera mounted on a drone. When a drone camera is used as the first camera 2, the framing evaluation device 1 can be used to determine the framing area (drone shooting area) of the first camera 2 and frame the other camera (second camera 3) to follow the drone shooting area.

Moreover, the framing evaluation device 1 is described as being installed in, for example, a broadcast van, but this is not limiting and it may be installed, for example, in a stadium.
Furthermore, the framing evaluation device 1 may be configured integrally with the second camera 3 to be evaluated in the same housing.
The framing evaluation device 1 may also be installed at a location (broadcast station) away from the stadium. In this case, for example, camera information is sent from a broadcast van via communication means to a location (broadcast station) different from the soccer stadium, and the framing evaluation device 1 installed at the broadcast station calculates the framing efficiency using the camera information. The framing evaluation device 1 then feeds back the evaluation value to the second camera 3 to be evaluated via communication means 6, and can display the evaluation value (framing efficiency R _AB and the difference value) on the viewfinder of the second camera 3.

The viewfinder screen display of the second camera 3 to be evaluated may also be displayed in a multi-screen format, showing the framing area of the first camera 2, which serves as the reference. Depending on the situation, for example, any one of Figures 8 to 12 may be displayed on this multi-screen along with the camera images. In this way, when the second camera 3 is operated by a photographer, the photographer of the second camera 3 can intuitively recognize where the framing area of the first camera 2, which is the reference, is located relative to the frame of his or her own camera. This has the effect of making it easier to correct the framing of his or her own camera.

In the above embodiment, framing of a soccer broadcast was described as an example, but the present invention is not limited to this. For example, the present invention is suitable for live broadcasts of sports other than soccer, such as rugby, baseball, and basketball, as well as footage of concerts and performances.

In addition, in the above embodiment, the framing evaluation device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to operate as each of the above-mentioned devices. These programs may be distributed via a communication line, or may be written onto a recording medium such as a CD-ROM or flash memory and distributed.

REFERENCE SIGNS LIST 1 Framing evaluation device 2 First camera 3 Second camera 4 Input device 5 Display device 6 Communication means 9 Storage means 10 Framing area determination section 11 Framing area vertex calculation section 12 Framing area vertex selection section 13 Intersection determination section 14 Projection area calculation section 15 Framing efficiency calculation section 15a Overlap area calculation section 15b Frame area calculation section 15c Framing efficiency calculation section 16 Framing efficiency evaluation section 17 Framing efficiency display section 21, 31 Drive mechanism 22, 32 Pan/tilt data detection sensor 23, 33 Lens data detection sensor 24, 34 Viewpoint 25, 35 Image sensor 30 Image frame 80 Projection area 90 Overlap area

Claims (7)

a framing area determination unit that calculates world coordinate values of each vertex for determining a boundary of a framing area of the first camera, using ROI information indicating position information of a predetermined region of interest expressed in a world coordinate system and camera parameters of a first camera that is a reference for photographing the region of interest;
a framing area vertex calculation unit that converts world coordinate values of each vertex for determining a boundary of a framing area of the first camera into image coordinate values of each vertex in the image coordinate system of the second camera by projecting a predetermined position of a world coordinate system onto an image coordinate system of the second camera using camera parameters of the second camera to be evaluated;
a framing area vertex selection unit that selects vertices that form outermost corners from among the vertices projected onto the image coordinate system of the second camera, and connects the vertices that form the outermost corners to form a projection area;
a projection area calculation unit that calculates a projection area, which is the area of the projection region;
a framing efficiency calculation unit that calculates an area of an overlap area where the image frame of the second camera and the projection area overlap, and calculates a framing efficiency that represents the ratio of the area of the overlap area to the projection area and to the area of the image frame of the second camera. an intersection determination unit that determines whether there are two or more intersections between a contour line of the projection area and a contour line of the image frame of the second camera;
2. The framing evaluation device according to claim 1, wherein the framing efficiency calculation section calculates the framing efficiency when two or more of the intersections exist. The framing efficiency calculation unit
3. The framing evaluation device according to claim 1 or claim 2, characterized in that it calculates an area coverage ratio indicating the ratio of the area of the overlapping area to the projected area, calculates an area occupancy ratio indicating the ratio of the area of the overlapping area to the area of the image frame of the second camera, and calculates the product of the area coverage ratio and the area occupancy ratio as the framing efficiency. The region of interest is defined as a range between a plane of Z=0 and a plane of Z=h in a world coordinate system,
The framing evaluation device according to any one of claims 1 to 3, characterized in that the framing area determination unit calculates the coordinates of each of the eight intersections of four line-of-sight vectors passing through the four corners of the image sensor of the first camera, the Z = 0 plane, and the Z = h plane, as the vertices of the framing area. The framing region vertex selector:
When the vertex of the intersection point in either plane Z=0 or Z=h projected onto the image coordinate system of the second camera is the vertex to be determined,
When the coordinates of the vertex to be determined are outside the area surrounded by the four vertices of the four intersections in the other plane projected onto the image coordinate system of the second camera,
A determination process for determining that the vertex to be determined is the vertex of the outermost corner,
5. The framing evaluation device according to claim 4, wherein the vertices constituting the outermost angles are determined by performing the above-mentioned process for each intersection point of the planes at both Z=0 and Z=h. a framing efficiency evaluation unit that calculates an evaluation value including a determination result of whether or not the value calculated by the framing efficiency calculation unit is greater than a predetermined threshold value;
6. The framing evaluation device according to claim 1, further comprising: a framing efficiency display unit that displays the calculated evaluation value on a display device. A program that causes a computer to function as a framing evaluation device according to any one of claims 1 to 6.

Images