JP7336346B2 - Framing information generator and its program - Google Patents

Description

The present invention relates to a framing information generating apparatus and program for generating framing information for specifying a partial area of a subject of interest.

2. Description of the Related Art Conventionally, there has been proposed an automatic shooting technique for specifying a region of interest in a video shot using a television camera.
For example, Japanese Patent Application Laid-Open No. 2002-200003 proposes a technique of drawing camera work with a light pen on a monitor television and performing pan and tilt control according to the trajectory.
Further, Japanese Patent Application Laid-Open No. 2002-200002 proposes a method of recognizing a subject within a field of view captured by a wide-angle camera and controlling the orientation of the photographic camera in accordance with the recognized movement of the subject.
Further, in Japanese Patent Laid-Open No. 2006-100003, when interpreting a medical image for medical diagnosis, as a technique for assigning a region of interest (ROI) to a medical image, for each of a plurality of pieces of diagnostic information, when they are assigned, A method has been proposed for generating a plurality of pieces of diagnostic information without obstructing the field of view for each piece of diagnostic information using parameters and priorities for displaying medical images.

JP-A-6-133189 JP-A-9-322051 WO2012/049741

Viewing forms of video content are diversifying, such as two-dimensional video, three-dimensional video, AR (Augmented Reality), and VR (Virtual Reality). In order to efficiently produce these video contents, for example, 2D video shooting with a live-action camera, 2D image virtual shooting with a 3D CG virtual camera, and 3D partial area display on a 3D display. Various production methods such as determination are used. In response to these production methods, there is a strong demand for an integrated method for automatically determining a partial area of a subject of interest in a scene.

The technique disclosed in Patent Document 1 described above implements the pan/tilt operation of the live-action camera by the drawing operation on the monitor television. However, since such a conventional technique operates on a two-dimensional screen, it is difficult to specify a region of interest in the depth direction.

Further, the technique of Patent Document 2 automatically sets an attention area based on automatic extraction of a subject and motion information, and can reflect the attention area in the three-dimensional real space in the control of the photographing camera. However, this prior art does not have a function of generating information regarding rendering such as which subject should be focused on.

In addition, the technique of Patent Literature 3 makes it possible to annotate three-dimensional information with parameters suitable for a plurality of pieces of diagnostic information with priority. However, since this prior art can only use parameters that have already been set, it does not have the function of determining a partial area of interest in a three-dimensional space, and is therefore applicable to the production of two-dimensional and three-dimensional images. It is difficult to
As described above, with the techniques described in Patent Literatures 1 to 3, it is difficult to specify a partial area of a subject to be focused on in accordance with various production methods.

SUMMARY OF THE INVENTION An object of the present invention is to provide a framing information generating apparatus and a program thereof that can generate framing information for specifying a partial area of a subject of interest corresponding to various production methods.

In view of the above-described problems, the framing information generating apparatus according to the present invention generates a set of the world coordinates of a subject, an identifier that uniquely identifies the subject, and a first importance level indicating the importance of the whole or part of the subject. Framing for generating framing information specifying a framing target, which is a partial area of a subject, based on the described framing auxiliary data and an effect profile describing a framing target, which specifies a partial area that is a part of the subject of interest. The information generation device is configured to include a selection unit, a collation unit, a weighted statistics unit, and a framing information generation unit.

According to such a configuration, the selector receives the framing auxiliary data and sequentially selects the framing auxiliary data one by one.
The collation unit receives an effect profile, compares the effect profile with the framing auxiliary data selected by the selection unit, and calculates a weighted value according to the degree of collation.
The weighted statistic unit weights the world coordinates of the subject with a weight value and calculates a predetermined statistic of the weighted world coordinates.
The framing information generator generates framing information based on the statistics.
In this way, the framing information generation device refers to the presentation profile describing the framing target, so that it is possible to generate framing information for specifying the partial area of the subject of interest corresponding to various production methods.

The present invention can also be implemented by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate as the framing information generation device described above.

According to the present invention, it is possible to generate framing information for specifying a partial area of a subject of interest corresponding to various production methods.

1 is a block diagram showing the configuration of an automatic framing device according to an embodiment; FIG. (a) is an explanatory diagram showing three-dimensional shape/texture information in the embodiment, (b) is an explanatory diagram showing identifiers, and (c) is an explanatory diagram showing importance. FIG. 4 is an explanatory diagram illustrating a first example of framing auxiliary data in the embodiment; FIG. 10 is an explanatory diagram illustrating a second example of framing auxiliary data in the embodiment; FIG. 5 is an explanatory diagram illustrating an example of identification assistance data in the embodiment; FIG. 4 is an explanatory diagram illustrating conversion from a world coordinate system to a local coordinate system in an embodiment; 4 is a flow chart showing the operation of the automatic framing device according to the embodiment; 10 is a block diagram showing the configuration of an automatic framing device according to Modification 1; FIG.

(embodiment)
[Configuration of automatic framing device]
Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. In addition, the same reference numerals are given to the same means, and the description thereof is omitted.
A configuration of an automatic framing device (framing information generating device) 1 according to an embodiment will be described in detail with reference to FIG. In this embodiment, it is assumed that the automatic framing device 1 is arranged on the side of a broadcasting station that produces video content.

The automatic framing device 1 generates framing information for specifying a partial area of a subject of interest based on framing auxiliary data F and an effect profile, which will be described later. Then, the automatic framing device 1 outputs the generated framing information to an external device (for example, a photographing camera or a 3D CG production device).
As a result, these external devices can use the framing information input from the automatic framing device 1 to frame the subject of interest and produce desired video content.

As shown in FIG. 1, the automatic framing apparatus 1 includes a scanning unit 10, a selection unit 11, a filter unit (collation unit) 12, a weighted statistics unit 13, a coordinate conversion unit 14, a framing determination unit (framing information generator) 15.

The scanning unit 10 sequentially generates an index designating which set to select from N sets of framing auxiliary data F input to the automatic framing device 1 (N is an integer equal to or greater than 1). . For example, the scanning unit 10 generates integers from 1 to N in a predetermined order (counting up from 1 to N in increments of 1), and outputs the generated integers to the selection unit 11 and the weighted statistics unit 13 as indexes. .

The selection unit 11 receives the framing auxiliary data F and sequentially selects the framing auxiliary data F one by one based on the index input from the scanning unit 10 . The selection unit 11 then outputs the selected set of framing auxiliary data F to the filter unit 12 and outputs the world coordinate values included in the set of framing auxiliary data F to the weighted statistics unit 13 .

<Framing auxiliary data>
The framing auxiliary data F will be described in detail with reference to FIGS. 2 and 3. FIG.
The framing auxiliary data F is information describing a set of the world coordinates of the subject, the identifier of the subject, and/or the degree of importance of the subject (first degree of importance). That is, the framing auxiliary data F includes at least one of the subject identifier and the subject importance in addition to the subject world coordinates as data items.

The world coordinates are coordinates common to each object in a virtual or real three-dimensional space.
An identifier is information that uniquely identifies a subject.
Also, the degree of importance is information indicating the importance of the whole or part of the subject.

Video content is often produced using a three-dimensional model (three-dimensional shape and texture information) of a subject. Therefore, information such as the identifier and the degree of importance may be expressed along with the three-dimensional shape/texture information of the object. For example, when expressing 3D shape and texture information in a point cloud model (point cloud) format, color information is added to each constituent point of the point cloud (3D coordinates of each point that constitutes the subject). . Therefore, an identifier and a degree of importance may be added to each constituent point of the object.

As shown in FIG. 2(a), a three-dimensional model in which a male dancer _9M and a female dancer _9W are dancing as a pair will be described as an example. Here, the world coordinate values (X1, Y1, Z1) of one point of the head of the female dancer 9 _W , the world coordinate values (X2, Y2, Z2) of one point of the upper body excluding the head, and the world coordinates of one point of the lower body Let the value be (X3, Y3, Z3). In addition, the world coordinate values (X4, Y4, Z4) of the male dancer _9M 's head, the world coordinate values (X5, Y5, Z5) of the upper body, and the world coordinate values of the lower body (X6, Y6, Z6). Also, as shown in FIG. 2(b), the identifier of the female dancer _9W is set to '1' (ID=1), and the identifier of the male dancer _9M is set to '2' (ID=2). Furthermore, as shown in FIG. 2(c), for each of the male dancer 9 _M and the female dancer 9 _W , the importance of the head is 'H', the importance of the upper body is 'M', and the importance of the lower body is be 'L'. In this case, the framing auxiliary data F are six sets of data as shown in FIG. 'H' indicates high importance, 'M' indicates medium importance, and 'L' indicates low importance.

Also, as shown in FIG. 4, the framing auxiliary data F may include an identifier written in characters. For example, identifiers written in characters include proper nouns such as the subject's name and name, and character information indicating classification such as occupation and role. In the framing auxiliary data F of FIG. 4, the identifier of the female dancer _9W is 'female dancer', and the identifier of the male dancer _9M is 'male dancer'.

In addition, in FIGS. 3 and 4, for the sake of simplification of _explanation , it _is assumed that there are six sets of framing auxiliary data F. In this case, the framing auxiliary data F also have N sets.
Also, although the framing auxiliary data F in FIGS. 3 and 4 contain both the identifier and the degree of importance, it suffices if only one of the identifier or the degree of importance is contained.
Hereafter, the world coordinate value, the identifier, and the degree of importance included in the n-th framing auxiliary data _Fn are assumed to be _Xn , _In , and _Wn, respectively (where n is an integer from 1 to N).

Returning to FIG. 1, the description of the configuration of the automatic framing apparatus 1 is continued.
The filter unit 12 receives an effect profile, compares the effect profile with the framing auxiliary data _Fn input from the selection unit 11, and calculates a weighting value according to the degree of matching. The filter unit 12 then outputs the calculated weighted values to the weighted statistics unit 13 .

A presentation profile is information describing a framing target that specifies a partial area that is part of a subject of interest. For example, the presentation profile describes conditions for filtering framing targets, such as notable subjects and notable places.
Here, the rendering profile can be described by data items included in the framing auxiliary data _Fn . For example, if the framing auxiliary data _Fn includes the world coordinate value _Xn , the identifier _In , and the importance level _Wn , the presentation profile can be described by any one or more of the world coordinate value, the identifier, and the importance level.
On the other hand, the rendering profile may be described with subject attributes that are not included in the framing auxiliary data _Fn . In this case, it is necessary to input identification auxiliary data H, which will be described later, to the filter unit 12 (broken line in FIG. 1). Note that an attribute is a proper noun for identifying a subject, and information representing the properties and features of the subject.

<First Example of Matching: Matching by Identifier>
First to fourth examples of collation by the filter unit 12 will be sequentially described below.
In this first example, the filter unit 12 compares the rendering profile and the framing auxiliary data _Fn using the identifier, and calculates the weight value _Vn according to the degree of matching. That is, the filter unit 12 checks the identifier (for example, ID=2) described in the production profile with the identifier _In of the framing auxiliary data _Fn . Then, when the identifier of the rendering profile and the identifier I _n of the framing auxiliary data F _n match, the filter unit 12 calculates the weight V _n =α. On the other hand, when the identifier of the effect profile and the identifier I _n of the framing auxiliary data F _n do not match, the filter unit 12 calculates the weight V _n =β. Note that α and β are preset real numbers, and preferably satisfy 0≦β<α (for example, α=1, β=0).

Further, when the framing auxiliary data _Fn includes the importance level _Wn , the filter unit 12 may calculate the weight value _Vn in consideration of the importance level _Wn . For example, the filter unit 12 calculates the weight value V _n =W _n· α when the identifiers match, and calculates the weight value V _n =W _n· β when the identifiers do not match. Here, as shown in FIG. 2(c), when the importance level _Wn is not quantified, numerical values corresponding to the importance level _Wn such as H=1, M=0.5, L=0 It is set in advance and the numerical value is multiplied.

Here, as shown in FIG. 4, let us consider a case where identifiers are described in characters such as 'female dancer' and 'male dancer' in the framing auxiliary data F. FIG. At this time, characters such as 'female dancer', 'soccer player', and 'man' are described in the presentation profile. In this case, the filter unit 12 collates whether the identifier of the effect profile and the identifier _In of the framing auxiliary data _Fn correspond to complete match, partial match, or complete mismatch. For example, when a 'female dancer' is described in the performance profile, the filter unit 12 perfectly matches the identifier I _n of the framing auxiliary data F _n , and therefore calculates the weight value V _n =α. Further, when the effect profile describes 'man', the filtering unit 12 partially matches the identifier I _n of the framing auxiliary data F _n , and therefore calculates the weight value V _n =γ. γ is a preset real number, and preferably satisfies β<γ<α (eg, γ=0.5). Further, when the presentation profile describes a 'soccer player', the filtering unit 12 does not completely match the identifier I _n of the framing auxiliary data F _n , so the filter unit 12 calculates the weight value V _n =β. In this manner, the filter unit 12 calculates a larger weighted value _Vn as the degree of collation increases.

<Second Example of Matching: Matching by Importance>
In this second example, the filter unit 12 compares the presentation profile and the framing auxiliary data _Fn according to the degree of importance, and calculates the weight value _Vn according to the degree of matching. That is, when the importance of a subject to be noticed (for example, 'H') is described in the production profile, the filter unit 12 compares the importance with the importance _Wn of the framing auxiliary data _Fn . . Then, when the degree of importance of the effect profile and the degree of importance _Wn of the framing auxiliary data _Fn match, the filter unit 12 calculates the weighted value _Vn =α. On the other hand, the filter unit 12 calculates the weight value V _n =β when the identifier of the effect profile and the importance level W _n of the framing auxiliary data F _n do not match.

<Third Example of Matching: Matching by World Coordinate Values>
In this third example, the filter unit 12 compares the rendering profile and the framing auxiliary data _Fn based on the world coordinate values, and calculates the weight value _Vn according to the degree of matching.
For example, consider a case where one point on world coordinates such as [5, 10, 0] ^T is described in the presentation profile. In this case, the filter unit 12 calculates the weight value _Vn with a larger value as the world coordinate value _Xn of the framing auxiliary data _Fn is closer to one point on the world coordinates described in the effect profile.

Also, when a partial area on the world coordinates such as {[x, y, z] ^T |-1 ≤ x ≤ +1, -2 ≤ y ≤ +2, 0 ≤ z ≤ +0.2} is described in the production profile think of. In this case, the filter unit 12 calculates the weight value _Vn based on whether or not the world coordinate value _Xn of the framing auxiliary data _Fn is included in the partial area on the world coordinate system described in the presentation profile. For example, when the world coordinate value X _n is included in a partial area on the world coordinate system, the filter unit 12 calculates the weight value V _n =α. On the other hand, when the world coordinate value X _n is not included in the partial region on the world coordinate system, the filter unit 12 calculates the weight value V _n =β.

<Fourth example of matching: Matching by attribute>
In this fourth example, the filter unit 12 identifies the identifier by matching the identification auxiliary data H and the production profile according to the attribute. Then, the filter unit 12 compares the production profile with the framing auxiliary data _Fn using the specified identifier, and calculates the weight value _Vn according to the degree of matching.

First, the identification assistance data H input to the filter unit 12 will be described. As shown in FIG. 5, the identification auxiliary data H describes the subject's identifier and attribute. The example of FIG. 5 represents identification auxiliary data H for male dancers _9M and female dancers _9W . That is, the identification auxiliary data H associates the identifier '1' of the female dancer _9W with the name 'Hanako', the gender 'female', and the place of birth 'Shizuoka' as attributes of the female dancer _9W . Further, the identification auxiliary data H associates the identifier '2' of the male dancer _9M with the name 'Taro', the gender 'male', and the birthplace 'Toyama' as attributes of the male dancer _9M . Attributes are not limited to proper nouns such as names, gender, and hometowns, and may be, for example, occupations or roles.

Consider the case where the name 'Taro' is described as an attribute in the production profile. In this case, the filter unit 12 refers to the identification assistance data H in FIG. 5 and identifies the identifier (ID=2) of the subject corresponding to the name 'Taro'. Subsequently, as in the first example, the filter unit 12 compares the specified identifier (ID=2) with the identifier _In of the framing auxiliary data _Fn , and sets the weight _Vn according to the degree of matching. calculate.

It should be noted that in the filter unit 12, which of the first to fourth examples is used for collation is set in advance. In addition, in the filter unit 12, matching can be performed by combining two or more of the methods of the first to third examples.

Returning to FIG. 1, the description of the configuration of the automatic framing apparatus 1 is continued.
The weighted statistic unit 13 weights the world coordinate value _Xn input from the selection unit 11 with the weight value _Vn input from the filter unit 12, and calculates a statistic in the weighted world coordinate value _Xn . . For example, the weighted statistics unit 13 uses the following equation (1) to calculate the expected value vector μ (average) and the variance/covariance matrix S (variance) as statistics at the world coordinate value X _n . After that, the weighted statistic unit 13 outputs the calculated statistic to the coordinate transformation unit 14 .

The coordinate transformation unit 14 receives coordinate transformation parameters, and transforms the statistics in the world coordinates into the statistics in the local coordinates using the input coordinates transformation parameters.
Note that the local coordinates are coordinate values in a coordinate system defined with a virtual or real viewpoint position (eg, camera position for framing) as a reference (eg, origin).
Coordinate transformation parameters are parameters for transforming world coordinates into local coordinates (for example, rotation matrix R and translation vector T).

For example, as shown in FIG. 6, the origin of the local coordinate system L is located at the coordinate T in the world coordinate system W (translation vector T), and the rotation matrix representing the attitude change from the world coordinate system W to the local coordinate system L Let be R. In this case, when a point represented by a position vector _Xn in the world coordinate system W is represented by a position vector _Yn in the local coordinate system L, the following equation (2) holds.

Consider the case where the expected value vector μ and the variance/covariance matrix S in the world coordinate system W are coordinate-transformed into the local coordinate system L as shown in equation (1). In this case, the expected value vector ξ and the variance/covariance matrix V in local coordinates are represented by the following equation (3).

That is, the coordinate transformation unit 14 converts, for example, the expected value vector μ and the variance/covariance matrix S in the world coordinate system W to the expected value vector ξ and the variance/covariance matrix V Convert to Then, the coordinate transformation unit 14 outputs the expected value vector ξ and the variance/covariance matrix V in the local coordinate system L to the framing determination unit 15 .

The framing determination unit 15 performs framing for specifying a framing target, which is a partial area of the subject, based on the statistics (expected value vector ξ and variance/covariance matrix V) in the local coordinate system L input from the coordinate transformation unit 14. It generates information.

The framing information is information representing important areas to be cut out from the subject. The important area to be cropped represents, for example, a partial area in the local coordinate system L, or the shooting posture (eg, pan, tilt, roll) and angle of view (eg, focal length) of the camera expressed in the local coordinate system L. This camera includes not only a live-action camera that performs live-action photography, but also a virtual camera that performs virtual photography by CG.

First to fourth examples of the generation of framing information by the framing determination unit 15 will be sequentially described below. It should be noted that the framing determination unit 15 is set in advance as to which of the first to fourth examples the framing information is to be generated.

<First Example: Ellipsoidal Partial Area in Local Coordinate System>
In this first example, the framing determination unit 15 generates framing information representing an ellipsoidal partial area in the local coordinate system L. FIG. For example, this partial area is an ellipsoid whose center is the expected value vector ξ and whose spread corresponds to the variance/covariance matrix V. FIG. Therefore, the framing determination unit 15 generates framing information specifying an ellipsoidal partial region A represented by statistics (expected value vector ξ and variance/covariance matrix V) by the following equation (4). .
Note that the constant r is a preset real number equal to or greater than 0, and is set to 0.1 to 10, for example. Also, the volume of the partial area A increases as the value of the constant r increases. Y represents each point of the local coordinate system L.

Using the framing information of this first example, it is possible to determine the field of view and depth range to be reproduced on the 3D display. Furthermore, by using this framing information, when shooting with a virtual camera, or when shooting by controlling a live-action camera equipped with an automatic platform and a lens with a servo, the shooting posture and shooting angle of view of these cameras can be calculated appropriately. You can also

<Second example: rectangular parallelepiped partial area in local coordinate system>
In this second example, the framing determination unit 15 generates framing information representing a rectangular parallelepiped partial area A that approximates the ellipsoid of the first example.
For example, the rectangular parallelepiped partial area A is the maximum that is included in the ellipsoid of formula (4) or the minimum that includes the ellipsoid of formula (4). The rectangular parallelepiped partial area A has sides parallel to the axes of the local coordinate system L. As shown in FIG.
By using the framing information of the second example, the amount of calculation can be reduced by approximation in addition to the same effect as the first example.

<Third example: Addition of importance>
In this third example, the framing determination unit 15 calculates the importance (second importance) J(Y) for each partial area A or each divided area obtained by further dividing this partial area A. FIG. This divided area is obtained by dividing the partial area A into meshes of a predetermined size. Also, the importance level J(Y) can be used for clipping video content suitable for a display device.

For example, the framing determination unit 15, associated with the partial area A in the local coordinate system L, attaches importance J (Y ) may be calculated.

In addition, the framing determination unit 15 determines that the third component (for example, the depth component) of the local coordinate system L is large, as represented by the following equation (6), associated with each point Y of the local coordinate system L. The degree of importance J(Y) may be calculated so as to take a smaller value. After that, the framing determination unit 15 adds the calculated importance J(Y) to the framing information.

Using the framing information of this third example, it is possible to determine the field of view and depth range to be reproduced on the 3D display. Furthermore, by using this framing information, it becomes possible to compress and display a region of low importance in the depth direction in a 3D display with a limited depth reproduction range such as the integral method or the lenticular method.

<Fourth example: Camera posture and angle of view>
In this fourth example, the framing determination unit 15 generates framing information representing three degrees of freedom in attitude angle and one degree of freedom in angle of view of the camera. This camera is installed so that the origin of the local coordinate system L is aligned with the first optical principal point. In addition, the first axis of the local coordinate system L is horizontally rightward of the image plane photographed by the camera, the second axis is vertically downward of the image plane photographed by the camera, and the third axis is the depth of the optical axis of the camera.

For example, when shooting with a camera (including not only a real camera but also a virtual camera), the framing determination unit 15 sets the pan angle α and the tilt angle δ represented by the following equation (7), and Calculate the focal length f to be used.

For example, the focal length f is long according to the distance from the origin of the local coordinate system L to the center of the ellipsoid (including the inside) represented by Equation (4), and the ellipse represented by Equation (4) Shortened according to the extension of the body (including the inside).
Note that tr in Equation (8) represents a trace. Also, the constant c can be set to an arbitrary value, for example, a value about 0.1 to 10 times the sensor size of the camera (for example, 0.8 times). Also, although it has been described that all of the pan angle α, tilt angle δ, and focal length f are obtained, any one or more of them may be obtained.

By using the framing information of the fourth example, when shooting with a virtual camera, or when shooting by controlling a real camera equipped with an automatic camera platform and a servo-equipped lens, the shooting posture and shooting angle of view of these cameras can be set appropriately. can be calculated to As a result, these cameras can photograph the partial area A with an appropriate posture and angle of view.

[Operation of automatic framing device]
The operation of the automatic framing device 1 will be described with reference to FIG.
As shown in FIG. 7, in step S1, the scanning unit 10 initializes index n (n=1).
In step S2, n-th framing auxiliary data F _n (world coordinate value X _n , identifier I _n , importance W _n ) is selected based on index n.

In step S3, _the filter unit 12 compares the rendering profile with the framing auxiliary data _Fn , and calculates a weight value _Vn according to the degree of matching. is weighted with a weight value _Vn , and a statistic (expected value vector μ, variance/covariance matrix S) at the weighted world coordinate value _Xn is calculated.

In step S5, the coordinate transformation unit 14 transforms the statistics in the world coordinate system W into the statistics in the local coordinate system L (expected value vector ξ and variance/covariance matrix V) using the coordinate transformation parameters. The framing determination unit 15 generates framing information for specifying the partial area A of the subject of interest based on the statistics in the local coordinate system L (the expected value vector ξ and the variance/covariance matrix V).

In step S7, the scanning unit 10 determines whether the index n is N or more.
If the index n is less than N (No in step S7), the process proceeds to step S8.
In step S8, the scanning unit 10 increments the index n (n=n+1) and returns to the process of step S2.
If the index n is greater than or equal to N (Yes in step S7), the framing determination unit 15 outputs the framing information, and the process ends.

[Action/effect]
As described above, according to the automatic framing apparatus 1, framing information for specifying a partial area of a subject of interest corresponding to various production methods can be generated.
In other words, according to the automatic framing device 1, it is possible to adaptively identify the partial area A of interest in the local coordinate system L according to the presentation profile, and output the partial area A of interest as the framing information. can be done. As a result, it becomes possible to automatically determine the photographing area according to the program presentation or the like. In addition, since the region to be noticed is determined in the local coordinate system L defined individually for each camera, framing can be performed appropriately regardless of the installation position of the camera.

(Modification 1)
Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like within the scope of the present invention.

In the above-described embodiment, the world coordinate system is converted into the local coordinate system, but the present invention is not limited to this. For example, when the local coordinate system and the world coordinate system are the same, the automatic framing apparatus 1B may be configured without the coordinate conversion unit 14, as shown in FIG. In this case, the automatic framing device 1B does not transform the statistics in the world coordinate system into the local coordinate system. Therefore, the framing determination unit 15B uses the statistics (expected value vector μ, variance/covariance matrix S) in the world coordinate system instead of the statistics (expected value vector ξ and variance/covariance matrix V) in the local coordinate system. Use Other points of the automatic framing apparatus 1B are the same as those of the first embodiment, so further explanation is omitted.

As described above, according to the automatic framing device 1B, it is possible to adaptively identify the partial area A of interest in the world coordinate system according to the effect profile, and output the partial area A of interest as the framing information. becomes possible. As a result, it becomes possible to automatically determine the photographing area according to the program production.

(Other modifications)
In the above-described embodiment, the expected value vector (average value) and the variance/covariance matrix (variance value) are obtained as the statistics, but the present invention is not limited to this. For example, one or more of the median value, maximum value, or minimum value may be obtained as the statistic.

In the above-described embodiment, the automatic framing device is arranged on the side of the broadcasting station that produces the video content, but it is not limited to this. For example, the autoframer may be built into a receiver for viewing video content. As a result, the receiver can use the framing information to produce various video presentations.

Although the automatic framing device has been described as independent hardware in the above-described embodiments, the present invention is not limited to this. For example, the present invention can be implemented by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate as the automatic framing device described above. These programs may be distributed via a communication line, or may be distributed after being written in a recording medium such as a CD-ROM or flash memory.

1, 1B Automatic framing device (framing information generating device)
10 scanning unit 11 selection unit 12 filter unit (collation unit)
13 weighted statistics unit 14 coordinate transformation unit 15, 15B framing determination unit (framing information generation unit)

Claims (8)

framing auxiliary data describing a set of world coordinates of a subject, an identifier that uniquely identifies the subject, and a first importance level indicating the importance of the whole or a part of the subject; and part of the subject that should be noted. A framing information generating device for generating framing information for specifying the framing target, which is the partial region of the subject, based on an effect profile describing the framing target for specifying the partial region,
a selection unit that inputs the framing auxiliary data and sequentially selects the framing auxiliary data one by one;
a matching unit that inputs the rendering profile, matches the rendering profile with the framing auxiliary data selected by the selecting unit, and calculates a weighted value according to the degree of matching;
a weighted statistic unit that weights the world coordinates of the subject with the weight value and calculates a predetermined statistic of the weighted world coordinates;
a framing information generating unit that generates the framing information based on the statistic;
A framing information generating device comprising: a coordinate transformation unit that transforms the statistic in the world coordinates into the statistic in the local coordinates using preset coordinate transformation parameters;
2. The framing information generating apparatus according to claim 1, wherein said framing information generating unit generates said framing information based on statistics in said local coordinates. 2. The framing information generating unit generates the framing information specifying an ellipsoidal partial area represented by the statistic or a cuboidal partial area approximating the ellipsoid. The framing information generation device according to claim 2. The framing information generation unit calculates a second importance level for each partial area or each divided area obtained by further dividing the partial area based on a center position or a depth direction of the partial area, and calculates the calculated second importance. 4. The framing information generating apparatus according to claim 1, wherein the degree of importance is added to the framing information. The framing information generation unit selects one of three degrees of freedom of attitude angle and one degree of freedom of angle of view of the camera based on the statistics in the local coordinates and the installation position of the camera represented by the local coordinates. 3. The framing information generating apparatus according to claim 2, wherein the above information is generated as the framing information. 6. The weighted statistic unit calculates one or more of an expected value, a variance value, a median value, a maximum value, and a minimum value as the statistic. The framing information generation device according to . The collation unit
identification auxiliary data describing the identifier and attributes of the subject and the production profile describing the framing target with the attributes are input;
identifying the identifier by comparing the identification auxiliary data and the production profile with the attribute;
7. The framing information generating apparatus according to claim 1, wherein the effect profile and the framing auxiliary data selected by the selection unit are compared with the specified identifier. A program for causing a computer to function as the framing information generation device according to any one of claims 1 to 7.

Images