JP7548728B2 - Camera control device and program - Google Patents

Description

The present invention relates to a camera control device and a program.

Conventionally, there is known technology that reconstructs a three-dimensional model of a subject from images captured by multiple cameras and obtains a virtual viewpoint image from any position where no cameras are located.

For example, by using a camera equipped with an electric pan head (hereinafter referred to as a "robot camera") as the camera to be positioned, it is possible to capture images of the subject without it ever moving outside the angle of view of the robot camera. Even when a limited number of robot cameras are used, it is possible to capture images with both precision and freedom (see, for example, Patent Document 1).

A method that uses subject position information has also been proposed as a technology for controlling the position of a robot camera so that the subject does not move out of the angle of view (see, for example, Patent Document 2). This method acquires subject position information from an external source, as well as timing information indicating the passage of time, and controls the camerawork of the robot camera that captures the subject based on the subject position information and timing information.

Furthermore, a method has been proposed in which a subject is detected from an image captured by a sensor camera, and the subject is photographed while being automatically tracked by a robot camera (see, for example, Patent Document 3). In this method, multiple colors that make up the surface of the subject are preset, and from the captured image, areas that have the multiple preset colors are extracted as image portions, the subject is detected based on that image information, and the subject is automatically tracked.

JP 2019-67419 A International Publication No. 2017/154953 Patent No. 3704201

The method described in the aforementioned Patent Document 2 obtains subject position information from GNSS (Global Navigation Satellite System) satellites, and controls the robot camera by reading camera work corresponding to the position information from a camera work information table set.

However, this method cannot be used if a device for acquiring location information is not attached to the subject, and cannot be used efficiently in environments where location information cannot be received from satellites.

In addition, with the method described in Patent Document 3, if a control error causes the robot camera's posture to become unstable, the subject may move out of the camera's field of view, resulting in an angle of view in which part of the subject is missing, and it is not always possible to obtain useful images.

For this reason, it was desirable to be able to detect the position of the subject without using position information obtained from an external source and to control the robot camera quickly and accurately.

The present invention has been made to solve the above problems, and its purpose is to provide a camera control device and program that can detect the position of a subject without using position information obtained from an external source, and keep the subject within the camera's field of view at all times.

In order to solve the above problem, the camera control device of claim 1 is a camera control device that controls the attitude of a camera, and is characterized in that it includes an image capture area division unit that inputs an image capture video including a subject within an image capture area captured by an overhead camera, and divides the image capture area into a plurality of areas in accordance with a predetermined rule, thereby extracting divided images corresponding to each of the plurality of areas from an image of the image capture video ; a subject position detection unit that detects one or more areas in which the subject is present based on the divided images corresponding to each of the plurality of areas extracted by the image capture area division unit, and generates position information including an area pattern indicating the one or more areas; a camera control unit that holds a control table in which parameters indicating the attitude of the camera are stored corresponding to each of one area and all area patterns indicating a plurality of areas in which the subject may be present, reads out from the control table the parameters corresponding to the area pattern included in the position information generated by the subject position detection unit, and generates attitude information including the parameters; and a camera communication unit that transmits a control signal including the attitude information generated by the camera control unit to the camera.

a camera control device for controlling the attitude of a camera, the camera control device including: an image capture area division unit that inputs an image capture video including a subject within an image capture area captured by an overhead camera, and divides the image capture area into a plurality of areas in accordance with a predetermined rule, and extracts divided images corresponding to each of the plurality of areas from an image of the image capture video; a subject position detection unit that detects a silhouette area of the subject for each of the plurality of areas based on the divided images corresponding to each of the plurality of areas extracted by the image capture area division unit, and generates position information including the area and the silhouette area of the area; a camera control unit that holds a control table in which parameters indicating the attitude of the camera are stored corresponding to each of the plurality of areas, reads out from the control table the parameters corresponding to the areas included in the position information generated by the subject position detection unit, calculates a weighted average of the silhouette area of the area with respect to the entire silhouette area of the subject for the parameters, and generates attitude information including the parameter of the weighted average; and a camera communication unit that transmits a control signal including the attitude information generated by the camera control unit to the camera .

The camera control device of claim 3 is characterized in that, in the camera control device of claim 1 or 2 , the center of the imaging area is defined as the origin, a line from the origin passing through the installation position of the camera is defined as the starting line, and a point on the plane of the imaging area is represented by polar coordinates (r, δ) of a distance r from the origin and a deviation angle δ from the starting line, and the multiple areas divided by the imaging area dividing unit are defined as regions into which the imaging area is divided using the polar coordinates or in a horizontal or vertical direction to the optical axis of the camera.

Furthermore, the camera control device of claim 4 is a camera control device described in any one of claims 1 to 3 , characterized in that the camera control device controls a plurality of cameras, the camera control unit holds the control table for each of the plurality of cameras, generates the posture information for each of the plurality of cameras using the corresponding control table, and the camera communication unit transmits a control signal including the posture information generated by the camera control unit to each of the corresponding plurality of cameras.

Furthermore, the camera control device of claim 5 is a camera control device described in any one of claims 1 to 3 , wherein the camera control device controls a plurality of cameras, the camera control unit holds the control table for any one of the plurality of cameras, generates the posture information for the one camera using the control table, and generates the posture information for the other camera based on the posture information and the position and posture relationship between the one camera and the other camera of the plurality of cameras, and the camera communication unit transmits a control signal including the posture information generated by the camera control unit to each of the corresponding cameras of the plurality of cameras.

Further, a program according to a sixth aspect of the present invention is characterized in that it causes a computer to function as the camera control device according to any one of the first to fifth aspects of the present invention.

As described above, according to the present invention, the position of the subject can be detected without using position information obtained from an external source, and the subject can always be kept within the field of view of the camera.

1 is a block diagram showing an example of the overall configuration of an imaging system including a camera control device according to an embodiment of the present invention. 1A and 1B are diagrams illustrating examples of imaging environments and area divisions. 4 is a flowchart illustrating a processing example of the first embodiment. 4A to 4C are diagrams illustrating an example of the position of a subject within an imaging area in the first embodiment. FIG. 4 is a diagram illustrating an example of a configuration of a control table in the first embodiment. FIG. 13 is a diagram illustrating an example of the configuration of a control table in a modified example of the first embodiment. 13 is a flowchart illustrating a processing example of the second embodiment. 13A and 13B are diagrams illustrating an example of the position of a subject within an imaging area in the second embodiment. 13 is a flowchart illustrating a processing example of a modified example of the second embodiment. FIG. 13 is a diagram illustrating an example of the configuration of a control table in a modified example of the second embodiment. 13 is a flowchart illustrating a processing example of a third embodiment. 13A and 13B are diagrams illustrating examples of the positions of the robot cameras and area divisions in the fourth embodiment. 13A to 13C are diagrams illustrating an example of an imaging environment, a position of a robot camera, and area divisions in a fourth embodiment. 13A and 13B are diagrams illustrating examples of the positions of the robot cameras and area divisions in the fifth embodiment. 13 is a diagram illustrating a process of generating posture information for two robot cameras using one control table when the two robot cameras are installed in a space. FIG.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[Imaging system]
Fig. 1 is a block diagram showing an example of the overall configuration of an imaging system including a camera control device according to an embodiment of the present invention, and Fig. 2 is a diagram for explaining an example of an imaging environment and area divisions.

This imaging system 1 is comprised of an overhead camera 2, a camera control device 3, and one or more robot cameras 4. The imaging system 1 is a system that detects a subject 120 present in an imaging area 110 from an image of an image captured by the overhead camera 2, and controls the robot camera 4 so that the subject 120 is always within the field of view of the robot camera 4.

The subject 120 is present on a stage 115 within the imaging area 110, and the overhead camera 2 and robot camera 4 are attached to, for example, a hemispherical mounting member 116, which is set up so as to cover the subject 120. In FIG. 2, the overhead camera 2 is set up at the apex of the hemispherical mounting member 116 so as to overlook the subject 120.

The overhead camera 2 is a camera that captures the subject 120 and outputs the captured image to the camera control device 3. The overhead camera 2 is a camera or robot camera that is attached at an overhead position that can capture at least 80% of the imaging area 110, and preferably 100%, for example, at an elevation angle of 45° or more, and preferably 80°, from the imaging area 110. The overhead camera 2 may also be a camera that can be mounted on a drone and placed in the air.

The camera control device 3 inputs the captured image from the overhead camera 2, detects the position of the subject 120 within the imaging area 110 based on the captured image, generates a control signal for the posture etc. according to the position, and transmits the control signal to the robot camera 4.

The robot camera 4 receives the control signal and performs control such as panning and tilting based on the control signal, changing the posture, etc. This allows the robot camera 4 to keep the subject 120 within its field of view.

Here, the imaging area 110 is divided into a number of areas according to a preset rule. For example, in FIG. 2, the imaging area 110 is the area photographed by the overhead camera 2 when the stage 115 is viewed from the overhead camera 2, and its shape is a circle. The imaging area 110 is divided into four areas 111 to 114 in a clockwise direction from the upper right by two straight lines in a cross direction that intersect at a right angle and pass through the center of the circle of the imaging area 110. In other words, the areas 111 to 114 are regions in which the imaging area 110 is divided by two straight lines in a cross direction that pass through the center of the circle.

Referring to FIG. 1, the camera control device 3 includes an imaging area division unit 10, a subject position detection unit 11, a camera control unit 12, and a camera communication unit 13. The camera control unit 12 holds a preset control table 20.

The imaging area dividing unit 10 inputs the captured image from the overhead camera 2. Then, according to preset rules, as shown in FIG. 2, the imaging area dividing unit 10 divides the image of imaging area 110 in the captured image into images of four areas 111 to 114, thereby extracting divided images of areas 111 to 114. The imaging area dividing unit 10 outputs the divided images of areas 111 to 114 to the subject position detection unit 11.

The subject position detection unit 11 receives the divided images of areas 111 to 114 from the imaging area division unit 10, detects the subject 120 from these divided images, and generates position information indicating the position of the subject 120. The subject position detection unit 11 then outputs the position information of the subject 120 to the camera control unit 12.

Specifically, the time when the subject 120 is not within the field of view of the overhead camera 2 (within the imaging area 110) is defined as T0, and the time when the subject 120 is within the field of view of the overhead camera 2 is defined as T1. The subject position detection unit 11 detects the position of the subject 120 by known processing such as background subtraction using the divided images of areas 111 to 114 at times T0 and T1, chromakey that transmits a specific color, or luminance keying that transmits a specific brightness.

The camera control unit 12 receives position information of the subject 120 from the subject position detection unit 11, and generates posture information of the robot camera 4 based on the position information using a preset control table 20. The control table 20 stores parameters indicating the position information of the subject 120 and the posture of the robot camera 4 corresponding to the position information. The camera control unit 12 then outputs the posture information to the camera communication unit 13.

The camera communication unit 13 inputs the posture information from the camera control unit 12, generates a control signal including the posture information, and transmits the control signal to the robot camera 4. This enables the robot camera 4 to change its posture according to the posture information included in the control signal, and to bring the subject 120 into its field of view.

The processing of the camera control device 3 is explained in detail below with examples.

Example 1
First, a description will be given of Example 1. Example 1 is an example in which a subject 120 is present within any one of areas 111 to 114 within an imaging area 110.

Figure 3 is a flowchart showing an example of processing in the first embodiment, and shows the processing of the camera control device 3 shown in Figure 1. Figure 4 is a diagram explaining an example of the position of the subject 120 in the imaging area 110 in the first embodiment.

As shown in FIG. 4, subject 120 is present in area 111 of areas 111 to 114 within imaging area 110, but is not present in areas 112 to 114.

The imaging area division unit 10 of the camera control device 3 inputs the captured image from the overhead camera 2 (step S301), divides the imaging area 110 of the captured image into areas 111 to 114, and extracts the divided images (step S302).

The subject position detection unit 11 detects the position of the subject 120 from the divided images of areas 111 to 114 by processing such as the background difference described above (step S303). In this example, the subject position detection unit 11 detects that the subject 120 is present in area 111. Then, the subject position detection unit 11 generates position information including area x=111 (step S304).

Figure 5 is a diagram showing an example of the configuration of the control table 20-1 in the first embodiment. This control table 20-1 is composed of an area x and a pan-tilt value LUT(x) which is a parameter (posture parameter) indicating the posture corresponding to the area x. The pan-tilt value LUT(x) is composed of a pan value Pan and a tilt value Tilt in the area x.

In the control table 20-1, a pan-tilt value LUT (111) = ( _φ111 , _θ111 ) is stored corresponding to the area x = 111. In addition, a pan-tilt value LUT (112) = ( _φ112 , _θ112 ) is stored corresponding to the area x = 112, and a pan-tilt value LUT (113) = ( _φ113 , _θ113 ) is stored corresponding to the area x = 113. In addition, a pan-tilt value LUT (114) = ( _φ114 , _θ114 ) is stored corresponding to the area x = 114.

3, the camera control unit 12 reads out the pan/tilt value LUT (111)=( _φ111 , _θ111 ) corresponding to the area x=111 included in the position information from the control table 20-1 (step S305).Then, the camera control unit 12 generates orientation information including the pan value _φ111 and the tilt value _θ111 (step S306).

The camera communication unit 13 generates a control signal including the pan value φ ₁₁₁ and tilt value θ ₁₁₁ of the attitude information, and transmits the control signal to the robot camera 4 (step S307).

As a result, the robot camera 4 can accommodate the subject 120 within its field of view by changing its posture according to the pan value φ ₁₁₁ and tilt value θ ₁₁₁ included in the control signal. The posture of the robot camera 4 can be controlled by the camera control device 3 of the first embodiment.

[Modification of the first embodiment]
Next, a modified example of the embodiment 1 will be described. The modified example of the embodiment 1 is an example in which, in addition to the attitude of the robot camera 4 in the embodiment 1, the zoom rotation amount and the focus rotation amount, which are lens parameters of the robot camera 4, are also controlled. This is also applied to the embodiment 2, the modified example of the embodiment 2, the embodiment 3, the embodiment 4, and the embodiment 5 described later.

Figure 6 is a diagram showing an example of the configuration of a control table 20-1' in a modified example of the first embodiment. This control table 20-1' is composed of an area x, and the attitude parameters and lens parameters pan/tilt values and zoom/focus rotation amount LUT(x) corresponding to the area x. The lens parameters are the zoom rotation amount and focus rotation amount of the lens provided in the robot camera 4. The pan/tilt values and zoom/focus rotation amount LUT(x) are composed of the pan value Pan, tilt value Tilt, zoom rotation amount Zoom, and focus rotation amount Focus in the area x.

In the control table 20-1', a pan/tilt value and zoom/focus rotation amount LUT (111) = ( _φ111 , _θ111 , _Z111 , _F111 ) is stored corresponding to the area x = 111. In addition, a pan/tilt value and zoom/focus rotation amount LUT (112) = ( _φ112 , _θ112 , _Z112 , _F112 ) is stored corresponding to the area x = 112. In addition, a pan/tilt value and zoom/focus rotation amount LUT (113) = ( _φ113 , _θ113 , _Z113 , _F113 ) is stored corresponding to the area x = 113. Also, corresponding to area x=114, pan/tilt values and zoom/focus rotation amount LUT (114)=(φ ₁₁₄ , θ ₁₁₄ , Z ₁₁₄ , F ₁₁₄ ) are stored.

The camera control unit 12 reads out the pan/tilt values and zoom/focus rotation amount LUT(111)=( _φ111 , _θ111 , _Z111 , _F111 ) corresponding to the area x=111 included in the position information from the control table 20-1'. Then, the camera control unit 12 generates attitude lens information including the pan value _φ111 , tilt value _θ111 , zoom rotation amount _Z111 , and focus rotation amount _F111 .

The camera communication unit 13 generates a control signal including the pan value φ ₁₁₁ and tilt value θ ₁₁₁ of the attitude information, and the zoom rotation amount Z ₁₁₁ and focus rotation amount F ₁₁₁ of the lens information, and transmits the control signal to the robot camera 4 .

As a result, the robot camera 4 can place the subject 120 within its field of view by changing its posture according to the pan value φ ₁₁₁ and tilt value θ ₁₁₁ included in the control signal. Also, by changing the degree of image enlargement and focal length according to the zoom rotation amount Z ₁₁₁ and focus rotation amount F ₁₁₁ included in the control signal, the subject 120 can be photographed at a desired size and in focus. In other words, the posture and lens of the robot camera 4 can be controlled by the camera control device 3 of the modified example of the first embodiment.

Example 2
Next, a description will be given of Example 2. The above-described Example 1 is an example of a case where the subject 120 is contained within the area 111, but Example 2 is an example of a case where the subject 120 exists across all of the areas 111 to 114 within the imaging area 110.

Example 2 is an example of silhouette area control when subject 120 is present not only in area 111 but also across areas 112 to 114. Silhouette area control is a control that identifies one area in which the silhouette area of subject 120 is maximum, and uses posture information corresponding to the identified area.

Figure 7 is a flowchart showing a processing example of the second embodiment, and shows silhouette area control by the camera control device 3 shown in Figure 1. Figure 8 is a diagram explaining an example of the position of the subject 120 within the imaging area 110 in the second embodiment.

8, subject 120 exists across all of areas 111 to 114 within imaging area 110. The silhouette areas of subject 120 in areas 111 to 114 are defined as S _subject (111), S _subject (112), S _subject (113), and S _subject (114), respectively. Here, it is assumed that S _subject (111) > S _subject (114) > S _subject (112) = S _subject (113).

The imaging area division unit 10 of the camera control device 3 inputs the captured image from the overhead camera 2 (step S701), divides the imaging area 110 of the captured image into areas 111 to 114, and extracts the divided images (step S702).

The subject position detection unit 11 obtains the silhouette area S _subject (x) of the subject 120 from the divided images for each area (each of the areas 111 to 114) by the above-mentioned background subtraction processing or the like (step S703).

The subject position detection unit 11 compares the silhouette areas S _subject (x) between the areas 111 to 114. Then, the subject position detection unit 11 identifies the area x=111 having the largest silhouette area S _subject (x), and generates position information including the area x=111 (step S704).

The camera control unit 12 reads out the pan/tilt value LUT (111)=( _φ111 , _θ111 ) corresponding to the area x=111 included in the position information from the control table 20-1 shown in Fig. 5 (step S705).Then, the camera control unit 12 generates orientation information including the pan value _φ111 and the tilt value _θ111 (step S706).

The camera communication unit 13 generates a control signal including the pan value φ ₁₁₁ and tilt value θ ₁₁₁ of the posture information, and transmits the control signal to the robot camera 4 (step S707).

This allows the robot camera 4 to place the subject 120 within its field of view by changing its posture in accordance with the pan value φ ₁₁₁ and tilt value θ ₁₁₁ contained in the control signal.

[Modification of the second embodiment]
Next, a modified example of the second embodiment will be described. The modified example of the second embodiment is an example in which list control is performed when the subject 120 is present not only in the area 111 but also across the areas 112 to 114. The list control is a control that identifies one or more areas in which the silhouette of the subject 120 is present, and uses posture information corresponding to the identified one or more areas.

Figure 9 is a flowchart showing a processing example of a modified example of the second embodiment, showing list control by the camera control device 3 shown in Figure 1.

The imaging area division unit 10 of the camera control device 3 inputs the captured image from the overhead camera 2 (step S901), divides the imaging area 110 of the captured image into areas 111 to 114, and extracts the divided images (step S902).

The subject position detection unit 11 detects areas x=111, 112, 113, 114 in which the subject 120 exists as shown in the example of FIG. 8 from the divided images of areas 111 to 114 by processing such as the background difference described above (step S903).The subject position detection unit 11 then generates position information including area pattern x'=111, 112, 113, 114 (step S904).

Figure 10 is a diagram showing an example of the configuration of control table 20-2 in a modified example of Example 2. This control table 20-2 is composed of area pattern x' and pan-tilt value LUT(x'), which is the posture parameter corresponding to area pattern x'. Area pattern x' in control table 20-2 is all patterns in which subject 120 can exist in one or more of areas 111, 112, 113, and 114.

In the control table 20-2, a pan-tilt value LUT (111) = ( _φ111 , _θ111 ) is stored corresponding to the area pattern x' = 111. Also, a pan-tilt value LUT (112) = ( _φ112 , _θ112 ) is stored corresponding to the area pattern x' = 112. Similarly, a pan-tilt value LUT (141) = ( _φ141 , θ141) is stored corresponding to the area patterns x' = 111, 112, 113, ₁₁₄ .

9, the camera control unit 12 reads out from the control table 20-2 the pan/tilt value LUT (111, 112, 113, 114) = ( _φ141 , _θ141 ) corresponding to the area pattern x' = 111, 112, 113, 114 included in the position information (step S905).Then, the camera control unit 12 generates attitude information including the pan value _φ141 and the tilt value _θ141 (step S906).

The camera communication unit 13 generates a control signal including the pan value φ ₁₄₁ and tilt value θ ₁₄₁ of the posture information, and transmits the control signal to the robot camera 4 (step S907).

This allows the robot camera 4 to place the subject 120 within its field of view by changing its posture in accordance with the pan value φ ₁₄₁ and tilt value θ ₁₄₁ contained in the control signal.

Example 3
Next, a third embodiment will be described. The third embodiment is an example of performing weighted average control when the subject 120 exists across multiple areas 111 to 114 in the imaging area 110. The weighted average control is a control that identifies one or more areas in which the silhouette of the subject 120 exists, and uses weighted average posture information obtained from the area of the silhouette of the subject 120 for the identified one or more areas.

In the first and second embodiments, an example has been described in which the imaging area 110 is divided into four areas 111 to 114. Here, when the imaging area 110 is divided into n areas (n is an integer equal to or greater than 2), the number of pan-tilt value LUTs (x) stored in advance in the control table 20 becomes _nC1 + _{nC2 ... nCn -1} + _nCn . As the number of divisions increases, the number of _pan -tilt value LUTs (x) stored in the control table 20 also increases _, making it complicated.

Therefore, by performing the weighted average control of the third embodiment, the number of pan-tilt value LUT(x) pre-stored in the control table 20 is reduced, and the subject 120 is placed within the field of view of the robot camera 4 with high accuracy.

Figure 11 is a flowchart showing a processing example of the third embodiment, illustrating weighted average control by the camera control device 3 shown in Figure 1.

The imaging area division unit 10 of the camera control device 3 inputs the captured image from the overhead camera 2 (step S1101), divides the imaging area 110 of the captured image into areas 111 to 114, and extracts the divided images (step S1102).

The subject position detection unit 11 obtains the silhouette area S _subject (x) of the subject 120 from the divided images for each area (each of the areas 111 to 114) by the above-mentioned background subtraction process or the like (step S1103).

The subject position detection unit 11 generates position information including the area x and the silhouette area S _subject (x) (step S1104).

Here, the silhouette areas of subject 120 in areas 111 to 114 obtained by subject position detection section 11 are S _subject (111), S _subject (112), S _subject (113), and S _subject (114), respectively.

The camera control unit 12 reads out the pan/tilt value LUT(x) of the area x included in the position information from the control table 20-1 shown in Fig. 5 (step S1105). Then, the camera control unit 12 calculates a weighted average of the pan/tilt value LUT(x) by the silhouette area _Ssubject (x) of each area included in the position information to obtain the pan value φ and the tilt value θ (step S1106). The weighted average by the silhouette area _Ssubject (x) is a process of obtaining a weighted average of the pan/tilt value LUT(x) by using the ratio of the silhouette area _Ssubject (x) of the area x to the entire silhouette area of the subject 120 as a weight.

The camera control unit 12 generates orientation information including the pan value φ and the tilt value θ (step S1107).

Specifically, the camera control unit 12 reads out pan/tilt value LUT(111)=( _φ111 , _θ111 ) etc. corresponding to area x=111 etc. included in the position information. Similarly, the camera control unit 12 reads out pan/tilt value LUT(112)=( _φ112 , _θ112 ), LUT(113)=( _φ113 , _θ113 ), and LUT(114)=( _φ114 , _θ114 ).

ｆは、例えば増加関数である。 The camera control unit 12 calculates a weighted average of the pan/tilt value LUT(x) based on the silhouette area S _subject (x) included in the position information based on the following formula (1), and obtains the pan value φ and the tilt value θ. The camera control unit 12 may also obtain the pan value φ and the tilt value θ using the following formula (2).

f is, for example, an increasing function.

The camera communication unit 13 generates a control signal including the pan value φ and tilt value θ of the posture information, and transmits the control signal to the robot camera 4 (step S1108).

As a result, the robot camera 4 can place the subject 120 within its field of view by changing its posture according to the pan value φ and tilt value θ included in the control signal. In this case, the camera control unit 12 calculates a weighted average based on the silhouette area S _subject (x) to determine the pan value φ and tilt value θ, so there is no need to increase the number of pan-tilt value LUT(x) in the control table 20-1. In other words, highly accurate posture control can be achieved using a small number of pan-tilt value LUT(x).

Although the camera control unit 12 calculates a weighted average based on the silhouette area S _subject (x), it may also calculate a weighted average based on the area S _area (x) excluding the subject 120 within the area.

Specifically, the camera control unit 12 subtracts the silhouette area S _subject (x) of the area x included in the position information from the area of the area x to obtain the area S _area (x) of the area x other than the subject 120. Then, the camera control unit 12 reads out the pan/tilt value LUT(x) of the area x included in the position information from the control table 20-1.

ｇは、例えば減少関数である。 The camera control unit 12 calculates a weighted average of the pan/tilt value LUT(x) based on the area S _area (x) of the area x other than the subject 120 based on the following formula (3), and obtains the pan value φ and the tilt value θ. The camera control unit 12 may also obtain the pan value φ and the tilt value θ using the following formula (4).

g is, for example, a decreasing function.

Example 4
Next, a fourth embodiment will be described. In order to control the robot camera 4 with high precision, the imaging area 110 may be divided into smaller areas, and a control table 20 may be used in which the pan/tilt value LUT(x) corresponding to each area is stored.

Example 4 is an example in which the robot camera 4 is controlled with high precision by using a plurality of areas that are further divided into smaller areas within the imaging area 110 in Example 1, the modified example of Example 1, Example 2, the modified example of Example 2, and Example 3.

As described above, the imaging area dividing unit 10 of the camera control device 3 divides the image of the imaging area 110 into images of four areas 111 to 114 as shown in FIG. 2 according to preset rules. In the fourth embodiment, the imaging area dividing unit 10 further divides the image of the imaging area 110 into smaller areas according to preset rules.

Figure 12 is a diagram for explaining an example of the position of the robot camera and area division in Example 4. In (1) to (3), the center of the imaging area 110 is the origin, and the line that passes from the origin through the installation position of the robot camera 4 is the starting line, and points on the plane of the imaging area 110 are expressed as polar coordinates (r, δ) with the distance r from the origin and the deviation angle δ from the starting line.

Depending on the distance r and the deflection angle δ, which are parameters of the polar coordinates (r, δ), rules are defined for dividing the image of the imaging area 110 into areas shown in (1) to (3). The imaging area dividing unit 10 divides the image of the imaging area 110 into multiple areas divided by polar coordinates, as shown in (1) to (3), in accordance with preset rules. The multiple areas divided by the imaging area dividing unit 10 are regions into which the imaging area 110 is divided by polar coordinates.

Rule (1) divides the imaging area 110 into areas with a constant interval of distance r and deflection angle δ, which are parameters of the polar coordinates (r, δ), and the areas are densely and coarsely divided according to the distance r and deflection angle δ. Rule (1) allows for finer area division, and although the number of pan-tilt value LUTs (x) stored in the control table 20 increases, the robot camera 4 can be controlled with high precision.

Rule (2) divides the imaging area 110 so that the area near the center is a sparse area and the areas other than the center are dense areas. Rule (2) is suitable for use when using a robot camera 4 installed at a bird's-eye view position higher than the subject 120. Because the robot camera 4 views the subject 120 from a high position, there is less change in the pan and tilt values in the area near the center, making it possible to make that area sparse.

The rule (3) divides the imaging area 110 so that areas closer to the robot camera 4 are sparse areas and areas farther from the robot camera 4 are dense areas, and different divisions can be used in front of and behind the robot camera 4. Assuming that the size of the subject 120 is approximately the same in the front and back, the angle of view is wide in the front, so fewer areas are required, whereas the angle of view is narrow in the back, so the number of areas needs to be increased and their area narrowed. In other words, in front of the robot camera 4, posture control is performed with a small number of areas, and in the back, posture control with a large number of areas and high accuracy is performed. As a result, control of the robot camera 4 can be suitably achieved according to the area in front and the area in back.

Figure 13 is a diagram for explaining an example of the imaging environment, the position of the robot camera, and the area division in Example 4, and shows an example of the area division different from that shown in Figure 12. The robot camera 4 is installed in a bird's-eye view position higher than the subject 120.

As shown on the right side of Figure 13, this rule divides the imaging area 110 into areas perpendicular to the optical axis of the robot camera 4. The imaging area dividing unit 10 divides the image of the imaging area 110 in a direction perpendicular to the optical axis of the robot camera 4 according to a preset rule.

In this rule, the robot camera 4 looks down on the subject 120 from a high position, so movement in the tilt direction is important. Therefore, by dividing the area so that it is perpendicular to the optical axis of the robot camera 4, movement in the tilt direction can be achieved with high precision.

When the robot camera 4 is installed at the same height as the subject 120, it is preferable that the image capturing area 110 is divided into areas parallel to the optical axis of the robot camera 4. In this case, the image capturing area dividing unit 10 divides the image of the image capturing area 110 horizontally with respect to the optical axis of the robot camera 4 according to a preset rule.

For example, if the subject 120 is a human, vertical movement is limited to jumping, bending, stretching, etc., while horizontal movement is from one end of the stage 115 to the other. Vertical movement can be accommodated by widening the angle of view of the robot camera 4, but if horizontal movement is also accommodated by widening the angle of view, the subject 120 will become smaller.

For this reason, under this rule, the movement of the robot camera 4 in the panning direction is important. Therefore, by dividing the area so that it is parallel to the optical axis of the robot camera 4, the movement in the panning direction can be realized with high precision.

Similarly, when the imaging area 110 has a horizontally long shape, it is desirable as a rule to divide the imaging area 110 into areas that are parallel to the optical axis of the robot camera 4.

Example 5
Next, a description will be given of Example 5. Example 5 is an example in which a plurality of robot cameras 4 are used in Example 1, the modified example of Example 1, Example 2, the modified example of Example 2, Example 3, and Example 4.

(Rules for division)
When multiple robot cameras 4 are installed in a space so as to surround the subject 120, the imaging area 110 is divided into a predetermined number of areas of a predetermined shape according to a preset rule corresponding to each of the multiple robot cameras 4. Note that the imaging area 110 may be divided into common areas of the same number and shape common to the multiple robot cameras 4.

Specifically, the imaging area division unit 10 of the camera control device 3 divides the image of the imaging area 110 in the captured video according to preset rules corresponding to each of the multiple robot cameras 4. As a result, a predetermined number of divided images are extracted corresponding to each of the multiple robot cameras 4.

Figure 14 is a diagram explaining an example of the position of the robot camera 4 and area division in Example 5. In (1) and (2), two robot cameras 4-1 and 4-2 are shown installed at positions where their optical axes intersect at right angles.

Referring to (1), the rule corresponding to the robot camera 4-1 is to divide the image capturing area 110 in a direction perpendicular to the optical axis of the robot camera 4-1 (see the thick line in the image capturing area 110). The image capturing area dividing unit 10 divides the image of the image capturing area 110 in a direction perpendicular to the optical axis of the robot camera 4-1 according to a preset rule corresponding to the robot camera 4-1.

The rule corresponding to the robot camera 4-2 is to divide the image capturing area 110 in a direction perpendicular to the optical axis of the robot camera 4-2 (see the thin lines in the image capturing area 110). The image capturing area dividing unit 10 divides the image of the image capturing area 110 in a direction perpendicular to the optical axis of the robot camera 4-2 according to the preset rule corresponding to the robot camera 4-2.

Referring to (2), the rule corresponding to the robot camera 4-1 is to divide the image capturing area 110 horizontally with respect to the optical axis of the robot camera 4-1 (see the thick line in the image capturing area 110). The image capturing area dividing unit 10 divides the image of the image capturing area 110 horizontally with respect to the optical axis of the robot camera 4-1 according to a preset rule corresponding to the robot camera 4-1.

The rule corresponding to the robot camera 4-2 is to divide the imaging area 110 into areas with a certain interval between the parameters of the polar coordinates (r, δ), that is, distance r and deflection angle δ, and with a density that corresponds to the distance r and deflection angle δ (see the thin lines in the imaging area 110). The imaging area division unit 10 divides the image of the imaging area 110 in the polar coordinate (r, δ) direction according to the preset rule corresponding to the robot camera 4-2.

In addition, the imaging area 110 may be divided into common areas of the same number and shape according to pre-set rules common to multiple robot cameras 4.

(Control Table 20)
Furthermore, when multiple robot cameras 4 are installed in a space, different control tables 20 may be used in correspondence with each of the multiple robot cameras 4 .

In this case, the camera control unit 12 holds multiple control tables 20 (for the number of robot cameras 4) corresponding to each of the multiple robot cameras 4. The camera control unit 12 inputs position information of the subject 120 from the subject position detection unit 11 for each of the multiple robot cameras 4, and uses the corresponding control table 20 to determine posture information of the robot camera 4 based on the position information.

The camera communication unit 13 inputs posture information from the camera control unit 12 for each of the multiple robot cameras 4, generates a control signal including the posture information, and transmits the control signal to the corresponding robot camera 4.

In addition, it is also possible to use only one control table 20 corresponding to one of the multiple robot cameras 4.

Specifically, the camera control unit 12 holds a control table 20 corresponding to a given robot camera 4. The camera control unit 12 calculates pan/tilt value LUT(x) corresponding to the other robot cameras 4 based on the pan/tilt value LUT(x) stored in the control table 20 and the relationship between the positions and postures of the multiple robot cameras 4. The camera control unit 12 then generates a control table 20 corresponding to the other robot cameras 4. The camera control unit 12 generates posture information for each of the multiple robot cameras 4 using the corresponding control table 20.

In this case, the camera control unit 12 may generate posture information for a given robot camera 4 using the control table 20, and convert the posture information of the given robot camera 4 into posture information of other robot cameras 4. Specifically, the camera control unit 12 generates posture information of other robot cameras 4 based on the posture information of one robot camera 4 and the relationship between the positions and postures of multiple robot cameras 4.

Figure 15 is a diagram explaining the process of generating posture information for two robot cameras 4-1, 4-2 using one control table 20 when the two robot cameras 4-1, 4-2 are installed in space. Below, we explain the process in which the camera control unit 12 generates posture information for robot camera 4-1 using one control table 20, and generates posture information for robot camera 4-2 from the posture information for robot camera 4-1 using a predetermined formula.

For any directional vector, let R _a ^(b) be the rotation matrix that transforms each component in the coordinate system Σ ^(a) into each component in the coordinate system Σ ^(b) . Vectors in the coordinate system Σ ^(a) are superscripted with (a). A coordinate system fixed to the imaging area 110 is the world coordinate system Σ ^(W) . Here, the first axis X and second axis Y are set on the floor surface of the imaging area 110, and the third axis Z is set vertically upward. Unless otherwise specified, all coordinate systems are right-handed.

Moreover, the coordinate system fixed to the robot camera 4-1 is defined as the first camera coordinate system Σ ^(c1) (solid line), and the first camera coordinate system Σ ^(c1) when the pan angle _φ1 and tilt angle _θ1 of the robot camera 4-1 are both 0 is defined as the first reference camera coordinate system Σ ⁽ⁿ¹⁾ (dashed line). Note that hereinafter, it is assumed that the origins of the first camera coordinate system Σ ^(c1) and the first reference camera coordinate system Σ ⁽ⁿ¹⁾ are the same.

ＸＹＺ－オイラー角（α₁，β₁，γ₁）は、ロボットカメラ４－１を設置したときに決定される定数である。 For example, when the orientation of the first reference camera coordinate system Σ ⁽ⁿ¹⁾ relative to the world coordinate system Σ ^(W) is expressed by XYZ-Euler angles (α ₁ , β ₁ , γ ₁ ), the rotation matrix R _n1 ^(w) is expressed by the following equation.

The XYZ-Euler angles (α ₁ , β ₁ , γ ₁ ) are constants that are determined when the robot camera 4-1 is installed.

Furthermore, when the pan axis of the robot camera 4-1 is the second axis of the first camera coordinate system Σ ^(c1) and the tilt axis is the first axis of the first camera coordinate system Σ ^(c1) , the rotation matrix R _c1 ⁽ⁿ¹⁾ is expressed by the following equation.

Furthermore, the coordinate system fixed to the robot camera 4-2 is the second camera coordinate system Σ ^(c2) (solid line), and the second camera coordinate system Σ ^(c2) when the pan value _φ2 and tilt value _θ2 of the robot camera 4-2 are both 0 is the second reference camera coordinate system Σ ⁽ⁿ²⁾ (dashed line). Note that the pan axis of the robot camera 4-2 is the second axis of the second camera coordinate system Σ ^(c2) , and the tilt axis is the first axis of the second camera coordinate system Σ ^(c2) . In the following, it is assumed that the origins of the second camera coordinate system Σ ^(c2) and the second reference camera coordinate system Σ ⁽ⁿ²⁾ are the same.

For example, when the orientation of the second reference camera coordinate system Σ ⁽ⁿ²⁾ relative to the world coordinate system Σ ^(W) is expressed by XYZ-Euler angles (α ₂ , β ₂ , γ ₂ ), the rotation matrix R _w ⁽ⁿ²⁾ is expressed by the following equation.

Let T ₁ ^(W) be the vector that runs from the origin O of the world coordinate system Σ ^(W) to the position of robot camera 4-1 (i.e., the origin of the first camera coordinate system Σ ^(c1) ), and let T ₂ ^(W) be the vector that runs from the origin O of the world coordinate system Σ ^(W) to the position of robot camera 4-2 (i.e., the origin of the second camera coordinate system Σ ^(c2) ).

Assume that the pan value φ ₁₁₁ and tilt value θ ₁₁₁ , which are the posture information of the robot camera 4-1, are obtained by the camera control unit 12 using the control table 20 corresponding to the robot camera 4-1. When the robot camera 4-1 is controlled at the pan value φ ₁₁₁ and tilt value θ ₁₁₁ , it is determined which part of the imaging area 110 the robot camera 4-1 is imaging.

とする。例えば、光軸が第一カメラ座標系Σ^(c1)の第３軸と平行である場合には、以下の式にて表される。

For example, it is sufficient to obtain a point P where the optical axis of the robot camera 4-1 intersects with the floor surface of the imaging area 110 when the robot camera 4-1 has a pan value of φ ₁₁₁ and a tilt value of θ _111. The unit direction vector of the optical axis of the robot camera 4-1 is

For example, when the optical axis is parallel to the third axis of the first camera coordinate system Σ ^(c1) , it is expressed by the following equation.

係数λは、正の実数である。

The position of the robot camera 4-1 is represented as point R _1. In this case, it is expressed by the following equation.

The coefficient λ is a positive real number.

の第３成分は零である。つまり、以下の式にて表される。

Here, since the point P is on the floor of the imaging area 110, the vector

The third component of is zero. That is, it is expressed by the following equation.

Therefore, the coefficient λ is determined by the following formula:

と重なるべきであり、以下の式が成立する。

Next, we determine the direction in which the robot camera 4-2 should be pointed, that is, the direction of the optical axis. If the position of the robot camera 4-2 is the point _R2 , the optical axis is the vector

and the following equation holds:

を定めることができる。

From the above formulas (9), (11), and (12), the line of sight vector is calculated by the following formula:

can be determined.

When the above equation (13) is coordinate-transformed using the above equation (7), it is expressed by the following equation.

の各成分を、以下の式で表す。

The vector thus obtained

Each component is represented by the following formula:

尚、atan２（ξ，ζ）は、２次元直交座標系における座標［ｐ，ｑ］^Tを極座標に変換した際の偏角を求める関数である。 At this time, the pan value φ' ₁₁₁ and tilt value θ' ₁₁₁ to which the robot camera 4-2 should be directed are expressed by the following equations.

Incidentally, atan2(ξ, ζ) is a function for finding the argument when the coordinates [p, q] ^T in a two-dimensional orthogonal coordinate system are converted into polar coordinates.

In this way, the camera control unit 12 generates posture information for a given robot camera 4-1 using the control table 20, and calculates the posture information of the other robot camera 4-2 from the posture information using the above formula (16).

Here, instead of the formula (16), a table consisting of the posture information of one robot camera 4-1 and the posture information of another robot camera 4-2 may be used. Specifically, the camera control unit 12 generates posture information for a given robot camera 4-1 using the control table 20, and reads out the posture information of the other robot camera 4-2 that corresponds to the posture information of the one robot camera 4-1 from the table. This makes it possible to generate posture information of the other robot camera 4-2 from the posture information of one robot camera 4-1.

In addition, instead of using the formula (16), a control table 20' corresponding to the other robot camera 4-2 may be set in advance and the control table 20' may be used. Specifically, the camera control unit 12 calculates the pan/tilt value LUT(x) of the other robot camera 4-2 from the pan/tilt value LUT(x) stored in the control table 20 corresponding to a specific robot camera 4-1 in advance using the formula (16). Then, the camera control unit 12 sets the control table 20' corresponding to the other robot camera 4-2 using the pan/tilt value LUT(x) of the other robot camera 4-2. The camera control unit 12 generates posture information for each of the multiple robot cameras 4-1 and 4-2 using the corresponding control tables 20 and 20'. Note that the same process can be used for three or more robot cameras 4.

As described above, according to the camera control device 3 of Example 1, the modified example of Example 1, Example 2, the modified example of Example 2, Example 3, Example 4, and Example 5, the imaging area division unit 10 divides the image of the imaging area 110 in the captured video into a predetermined number of area images (divided images) of a predetermined shape in accordance with preset rules.

The subject position detection unit 11 detects the subject 120 from multiple divided images and generates position information including the position of the subject 120.

The camera control unit 12 uses the control table 20 to determine posture information, etc. of the robot camera 4 based on the position information.

The camera communication unit 13 generates a control signal including posture information, etc., and transmits the control signal to the robot camera 4.

As a result, the robot camera 4 operates according to the posture information, etc., contained in the control signal. Therefore, the position of the subject 120 can be detected without using position information obtained from an external source, and the subject 120 can always be kept within the field of view of the robot camera 4.

In addition, because posture information is generated using a preset control table 20, the amount of calculation required in automatic control of the robot camera 4 can be reduced, enabling high-speed control to be achieved.

The present invention has been described above using Example 1, etc., but the present invention is not limited to Example 1, etc., and various modifications are possible without departing from the technical concept thereof.

In addition, a normal computer can be used as the hardware configuration of the camera control device 3 according to the first embodiment of the present invention. The camera control device 3 is configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface, etc.

The functions of the imaging area division unit 10, the subject position detection unit 11, the camera control unit 12, and the camera communication unit 13 provided in the camera control device 3 are each realized by having the CPU execute a program that describes these functions.

These programs are stored in the storage medium and are read and executed by the CPU. In addition, these programs can be distributed by storing them on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network.

Reference Signs List 1: Imaging system 2: Overhead camera 3: Camera control device 4: Robot camera 10: Imaging area division unit 11: Subject position detection unit 12: Camera control unit 13: Camera communication unit 20: Control table 110: Imaging areas 111, 112, 113, 114: Area 115: Stage 116: Mounting member 120: Subject

Claims (6)

In a camera control device that controls the attitude of a camera,
an image capture area division unit that receives an image capture video including a subject within an image capture area captured by an overhead camera, divides the image capture area into a plurality of areas in accordance with a preset rule, and extracts divided images corresponding to each of the plurality of areas from an image of the image capture video;
a subject position detection unit that detects one or more areas in which the subject exists based on the divided images corresponding to the plurality of areas extracted by the imaging area division unit, and generates position information including an area pattern indicating the one or more areas ;
a camera control unit that holds a control table in which parameters indicating an attitude of the camera are stored corresponding to each of all area patterns indicating one area and multiple areas in which the subject may be present , reads out from the control table the parameters corresponding to the area pattern included in the position information generated by the subject position detection unit, and generates attitude information including the parameters;
a camera communication unit that transmits a control signal including the attitude information generated by the camera control unit to the camera;
A camera control device comprising: In a camera control device that controls the attitude of a camera,
an image capture area division unit that receives an image capture video including a subject within an image capture area captured by an overhead camera, divides the image capture area into a plurality of areas in accordance with a preset rule, and extracts divided images corresponding to each of the plurality of areas from an image of the image capture video;
a subject position detection unit that detects a silhouette area of the subject for each of the plurality of areas based on the divided images corresponding to each of the plurality of areas extracted by the imaging area division unit, and generates position information including the area and the silhouette area of the area ;
a camera control unit that holds a control table in which parameters indicating an attitude of the camera are stored corresponding to each of the plurality of areas, reads out from the control table the parameters corresponding to the areas included in the position information generated by the subject position detection unit, calculates a weighted average of a silhouette area of the area relative to an overall silhouette area of the subject for the parameters, and generates attitude information including the parameters of the weighted average;
a camera communication unit that transmits a control signal including the attitude information generated by the camera control unit to the camera;
A camera control device comprising: 3. The camera control device according to claim 1 ,
The center of the imaging area is defined as the origin, a line passing through the installation position of the camera from the origin is defined as the starting line, and a point on the plane of the imaging area is expressed as polar coordinates (r, δ) of a distance r from the origin and a deviation angle δ from the starting line,
A camera control device characterized in that the multiple areas divided by the imaging area dividing unit are regions into which the imaging area is divided using polar coordinates or horizontally or vertically to the optical axis of the camera. 4. The camera control device according to claim 1 ,
The camera control device controls a plurality of cameras,
The camera control unit is
storing the control table for each of the plurality of cameras, and generating the posture information for each of the plurality of cameras using the corresponding control table;
The camera communication unit includes:
a camera control device that transmits a control signal including the attitude information generated by the camera control unit to each of the corresponding cameras; 4. The camera control device according to claim 1 ,
The camera control device controls a plurality of cameras,
The camera control unit is
holding the control table for any one of the plurality of cameras, generating the attitude information for the one camera using the control table, and generating the attitude information for the other camera based on the attitude information and a position and attitude relationship between the one camera and the other camera of the plurality of cameras;
The camera communication unit includes:
a control signal including the attitude information generated by the camera control unit, transmitted to each of the corresponding cameras; A program for causing a computer to function as the camera control device according to any one of claims 1 to 5 .

Images