JP7634974B2 - Image generating device, image generating method, and program - Google Patents

Description

The present invention relates to an image generation device, an image generation method, and a program.

In the field of computer graphics, an image of an object placed in a three-dimensional virtual space is captured by a virtual camera (hereafter referred to as a virtual camera) placed in the virtual space, and the captured image is displayed on any display device. In the following explanation, an image captured by a virtual camera is referred to as a virtual viewpoint image.

Patent Document 1 discloses a method of moving a virtual camera when a person placed in a virtual space moves and the moving person needs to be kept captured in a virtual viewpoint image. Here, the method of moving a virtual camera is a method that calculates virtual camera coordinates that include multiple subjects to be captured in the shooting range from the relative positions of the subjects, and moves the virtual camera to the calculated virtual camera coordinates.

JP 2017-545 A

However, depending on the characteristics of the display device that displays the virtual viewpoint image, the desired virtual viewpoint image may not be displayed. For example, naked-eye stereoscopic displays that display stereoscopic images have the characteristic of having a physical imaging range in which an image of a subject can be formed, and the imaging range may be narrower than the shooting range of the virtual camera. Therefore, even if the virtual camera is moved so that the subject is included in the shooting range of the virtual camera, as in the virtual camera movement method described in Patent Document 1, if the subject moves beyond the imaging range, it may not be possible to form an image because the subject does not fall within the imaging range. As a result, the object reflected in the virtual viewpoint image may appear blurred, and the desired virtual viewpoint image may not be displayed.

The present invention was made in consideration of the above problems, and aims to enable the display of a desired virtual viewpoint image according to the characteristics of the display device.

In order to achieve the above object, an image generating device according to one aspect of the present invention includes: a control means for controlling a position of a virtual camera, which is a virtual camera located in a virtual space, such that an image forming range of a display means for a shooting range of the virtual camera includes a position of a subject in the virtual space;
a generating means for generating a virtual viewpoint image based on the position of the virtual camera controlled by the control means;
Equipped with
The control means
a depth calculation means for calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject;
a comparison means for comparing the imaging range with the depth of the subject,
When the comparison by the comparison means shows that the depth of the subject is not included in the imaging range, the position of the virtual camera is controlled so that the depth of the subject is included in the imaging range .

According to the present invention, it is possible to display a desired virtual viewpoint image according to the characteristics of the display device.

FIG. 2 is a block diagram for explaining the functional configuration of the virtual visual point image generating system according to the first embodiment. 1A and 1B are diagrams showing examples of arrangement of a plurality of image capture devices according to the first and second embodiments. FIG. 2 is a block diagram for explaining the hardware configuration of the virtual viewpoint image generating system according to the first and second embodiments. FIG. 2 is a diagram for explaining an imaging range of a display device. 4 is a diagram for explaining the functional configuration of an operation area control unit according to the first embodiment. FIG. 4 is a flowchart for explaining a processing flow according to the first embodiment. FIG. 4 is an explanatory diagram of a control process according to the first embodiment. FIG. 11 is a block diagram for explaining the functional configuration of a virtual visual point image generating system according to a second embodiment. 10A and 10B are diagrams for explaining the functional configuration of an operation area control unit according to a second embodiment. 13A to 13C are diagrams for explaining an example of calculation by a depth calculation unit according to the second embodiment. 10 is a flowchart illustrating a process flow according to a second embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

[Embodiment 1]
In this embodiment, a process of controlling the position of the virtual camera so that an arbitrary subject is included in the imaging range of the display device will be described.

In the following explanation, FIG. 1 will be used to explain the overall virtual viewpoint image generation system according to the first embodiment, and FIG. 2 will be used to explain an example of the arrangement of an imaging device when a subject in a virtual space is generated based on an image captured by an actual imaging device.

Then, the hardware configuration of the virtual viewpoint image generation system is described with reference to FIG. 3, and the imaging range of the display device is described with reference to FIG. 4. Next, the functional configuration relating to the process of controlling the imaging range so that an arbitrary subject is included therein, which is a feature of the first embodiment of the present invention, is described with reference to FIG. 5, and the flow of the control process is described with reference to FIG. 6.

(Explanation of the virtual viewpoint image generation system 100)
1 is a block diagram for explaining the functional configuration of a virtual viewpoint image generation system 100 according to the first embodiment. The virtual viewpoint image generation system 100 is a system that images a subject in a virtual space as a virtual viewpoint image representing a view from a virtual viewpoint specified by a user. The virtual viewpoint image generation system 100 includes an operation information input unit 101 and a virtual viewpoint image generation device 102.

(Operation Information Input Unit 101)
An operation input of the virtual viewpoint is performed via an operation information input unit 101, and a virtual viewpoint image is generated by a virtual viewpoint image generating device 102. Note that the virtual viewpoint image in the first embodiment is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of viewpoint candidates provided in advance is also included in the virtual viewpoint image. Also, in the first embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the present invention is not limited to this example, and the virtual viewpoint image may be a still image.

The operation information input unit 101 has a function of receiving instructions from a user, processing the input information, and transmitting the processed information to the virtual viewpoint image generating device 102. The transmitted information includes operation instruction information for a virtual camera for operating the virtual viewpoint, physical information for identifying the physical imaging range of a display device (not shown), and subject information to be imaged. Here, the user's instructions are input, for example, by an external device such as a mouse or keyboard connected to the operation information input unit 101.

The virtual camera operation instruction information is a parameter value that specifies the amount of movement of the virtual camera in the virtual space. The parameter value is, for example, a vector value indicating the XYZ directions in three-dimensional space. The parameter value is assumed to be specified by the user, and can be specified, for example, by a controller that operates three axes in the pan, tilt, and roll directions, such as a joystick connected to the operation information input unit 101. Note that, other than a joystick, a device that can be operated by the user, such as a mouse or a keyboard, may also be used.

In addition, the virtual camera operation instruction information may be parameter values capable of operating the angle of view, which is the size of the field of view of the virtual viewpoint, in addition to the three axes in the X, Y and Z directions, and may be parameter values capable of operating multiple virtual cameras, not limited to a single virtual camera.

The physical information for identifying the physical imaging range of the display device is, for example, information that identifies an individual device, such as the model number of the display device (not shown).

As a method of acquiring the physical information, for example, the user can manually select the relevant model number from a list of model numbers stored in advance in the operation information input unit 101 to acquire the physical information, but it is also possible for the operation information input unit 101 to automatically recognize an unintended display device connected to the virtual viewpoint image generation system 100 and select the relevant model number from the list. Here, the display device includes, for example, a naked eye stereoscopic display.

The subject information to be imaged is, for example, identification information (ID) assigned to each subject in the virtual space. The operation information input unit 101 has a table in which subjects and identification information (ID) are previously associated with each other, and the identification information (ID) associated with an arbitrary subject is selected by clicking the mouse on an arbitrary subject on the monitor of a display device (not shown) capable of displaying a virtual viewpoint image. In addition to selecting an arbitrary subject on the monitor using a mouse, it is also possible to select the corresponding subject from an image by image processing based on an arbitrary threshold value. For example, a rugby player holding the ball may be extracted from an image and selected as the subject. Also, a subject that is not to be imaged may be selected.

(Virtual viewpoint image generating device 102)
The virtual viewpoint image generating device 102 determines the viewpoint of a virtual camera such that a subject in a virtual space is included in the imaging range of the display device, and generates a virtual viewpoint image, based on each piece of information transmitted from the operation information input unit 101. The information transmitted from the operation information input unit 101 is processed by a display characteristic identification unit 103, a virtual camera coordinate acquisition unit 104, and a subject coordinate acquisition unit 105, respectively.

The display characteristic identification unit 103 identifies the position and size of the imaging range relative to the shooting range of the virtual camera from the physical information identifying each device received from the operation information input unit 101. The position and size of the imaging range are associated with physical information indicating each device, such as a model number, and the virtual viewpoint image generation device 102 has a table (not shown) that associates the position and size of the imaging range relative to the shooting range with the physical information. The display characteristic identification unit 103 can identify the position and size of the imaging range associated with the physical information by referring to the table based on the physical information of the display device received from the operation information input unit 101. The display characteristic identification unit 103 inputs the position and size of the identified imaging range to the operable area control unit 106.

The virtual camera coordinate acquisition unit 104 acquires the position coordinates (position information) of the virtual camera in the virtual space from the operation instruction information received from the operation information input unit 101. Here, the operation instruction information is a parameter value, for example a vector value, that specifies the amount of movement of the virtual camera in the virtual space, and the coordinates obtained by moving the current virtual camera position by the vector value are acquired as the position coordinates (position information) of the next virtual camera that reflects the operation instruction. The virtual camera coordinate acquisition unit 104 inputs the acquired position coordinates to the operable area control unit 106.

The subject coordinate acquisition unit 105 acquires the position coordinates of the subject in the virtual space from the subject information received from the operation information input unit 101. The virtual viewpoint image generation device 102 has a table (not shown) that associates subject information with the position coordinates of the subject in the virtual space, and the subject coordinate acquisition unit 105 can acquire the position coordinates of the subject by referring to the table. The subject coordinate acquisition unit 105 inputs the acquired position coordinates to the operationable area control unit 106.

The operable area control unit 106 controls the viewpoint of the virtual camera (the position of the virtual camera) based on the information received from the display characteristic identification unit 103, the virtual camera coordinate acquisition unit 104, and the subject coordinate acquisition unit 105 so that the subject in the virtual space is included within the imaging range of the display device. The operable area control unit 106 transmits the controlled position of the virtual camera to the virtual camera position setting unit 107, which actually sets the position of the virtual camera based on the information on the position of the virtual camera controlled by the operable area control unit 106. After that, the virtual viewpoint image generation unit 108 generates a virtual viewpoint image based on the viewpoint of the virtual camera.

Note that multiple operation information input units 101 may be connected to one virtual viewpoint image generating device 102, and multiple virtual viewpoint image generating devices 102 may be connected to one operation information input unit 101. When connecting the virtual viewpoint image generating device 102 and the operation information input unit 101 in a one-to-many or many-to-one manner, the connection method may be a wired communication cable, or may be a wireless communication connection, or may be a configuration via a relay device such as a router or a network hub.

(Example of arrangement of multiple imaging devices)
The subject in the virtual space may be a subject generated by computer graphics, or may be generated based on images actually captured by a plurality of imaging devices. Fig. 2 is a diagram showing an example of the arrangement of a plurality of imaging devices when capturing images using the plurality of imaging devices.

For example, multiple imaging devices 210a to 210n are set in different positions surrounding a stadium 211 where sports such as soccer or rugby are played, and multiple captured images are acquired by the multiple imaging devices 210a to 210n by synchronously capturing images. Next, from the multiple captured images, a foreground image is acquired in which a foreground area corresponding to a specific object such as a person or a ball is extracted, and a background image is acquired in which a background area other than the foreground area is extracted.

After that, a foreground model representing the three-dimensional shape of a specific object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of the background, such as a stadium, is generated based on the background image. The texture data is then mapped onto the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a subject in a virtual space. However, the method of generating a subject is not limited to this, and various methods can be used, such as a method of generating a subject by projective transformation of a captured image without using a three-dimensional model.

The multiple imaging devices 210a-210n do not have to be installed around the entire circumference of the imaging area, and may be installed only in one direction of the imaging area depending on restrictions on the installation location, etc. The number of imaging devices 210a-210n is not limited to the example shown in the figure, and for example, if the imaging area is a soccer stadium, about 30 imaging devices 210a-210n may be installed around the stadium 211. Imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may also be installed.

(Hardware configuration of the virtual viewpoint image generation system 100)
FIG. 3 is a block diagram for explaining the hardware configuration of the virtual viewpoint image generating system 100. As shown in FIG.

The CPU (Central Processing Unit) 320 executes the processing according to the embodiment of the present invention using programs and data stored in the RAM (Random Access Memory) 321 and the ROM (Read Only Memory) 322.

The CPU 320 controls the overall operation of the virtual viewpoint image generation system 100. Note that the operation information input unit 101 or the virtual viewpoint image generation device 102 may have one or more pieces of dedicated hardware different from the CPU 320, and at least a part of the processing by the CPU 320 may be executed by the dedicated hardware. Examples of the dedicated hardware include, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

ROM321 holds programs and data. RAM322 has a work area that temporarily stores programs and data read from ROM321. RAM322 also provides a work area that is used when CPU320 executes each process.

The operation input unit 323 is, for example, a mouse, keyboard, or a three-axis controller such as a joystick, and acquires information operated by the user. The acquired information is sent to the operation information input unit 101. Note that the operation input unit 323 is not limited to these.

The external interface 324 may transmit information to an external monitor via an image output port such as a high-definition multimedia interface (HDMI (registered trademark)) or SDI (serial digital interface). It may also transmit image data via Ethernet (registered trademark) or the like.

(Imaging range of the display device 403)
Fig. 4 is a diagram for explaining the imaging range of a display device. Fig. 4(a) is a simulated diagram of a display of a display device 403 with a person as the subject. The subject 405 drawn with a dotted line indicates that the subject is displayed in a blurred state without being imaged, and the subject 404 drawn with a solid line indicates that the subject is displayed in an imaged state. The shooting conditions at this time will be explained with reference to Figs. 4(b) and 4(c).

Figure 4(b) is a diagram of a scene in which the virtual camera 400 is capturing a virtual viewpoint image to be displayed on the display device 403, and Figure 4(c) is a diagram of Figure 4(b) seen from the side. A subject 404 is located in front of the position of the virtual camera 400, and a subject 405 is located behind the position of the virtual camera 400. The display range 401 indicates the capturing range that can be displayed on the display device 403 from the capturing range at the viewpoint of the virtual camera 400, and only the subject within the frame is displayed on the display device 403. The imaging range 402 indicates the range in which the subject is imaged from the display range 401, and only the subject within the frame is imaged on the display device 403. The subject 404 is located within the imaging range 402, so an image is formed on the display device 403. On the other hand, the subject 405 is located outside the imaging range 402, so it is not imaged on the display device 403, and is displayed, for example, in a blurred state.

The shooting range, display range 401, and imaging range 402 have the relationship shown in FIG. 4B due to the characteristics of the naked eye stereoscopic display assumed as the display device in this embodiment. A naked eye stereoscopic display is a display that allows three-dimensional (3D) images to be viewed with the naked eye without using devices such as three-dimensional (3D) glasses. For example, in the case of a naked eye stereoscopic display using a lens array, the optical characteristics of the lens array arranged on the surface of the display are used to provide an imaging surface of the lens array on a surface in the depth direction within the display where the subject is displayed. This is the imaging range 402. Due to this principle, it is difficult to make the entire depth direction of the display an imaging surface, and the imaging range 402 is narrower than the display range 401.

In addition, since the imaging surface is determined by a lens array physically provided on the display surface, the position and size of the imaging range 402 relative to the display range 401 are unique settings for each display (display device). Therefore, it is not possible to enlarge only the imaging range 402 relative to the display range 401. To image a subject 405 located outside the imaging range 402, for example, it is necessary to move the position of the virtual camera 400 away from the subject and increase the relative size of the imaging range 402 relative to the subject.

The operable area control unit 106 shown in FIG. 3 determines whether a specific subject is included within the imaging range 402, and if the subject is not included within the imaging range 402, controls the position of the virtual camera 400 so that the position of the subject in the virtual space is included within the imaging range 402.

(Configuration of the operable area control unit 106)
5 is a block diagram for explaining the functional configuration of the operable area control unit 106 according to the first embodiment of the present invention. The operable area control unit 106 has, as its functional configuration, an imaging range calculation unit 500, a depth calculation unit 501, a comparison unit 502, a coordinate calculation unit 503, and a coordinate transmission unit 504.

The imaging range calculation unit 500 first obtains the shooting range of the virtual camera 400 based on the position coordinates of the virtual camera 400 input from the virtual camera coordinate acquisition unit 104. Next, the imaging range calculation unit 500 calculates the imaging range 402 at the position coordinates of the virtual camera 400 from a table of the position and size of the imaging range relative to the shooting range of the virtual camera 400 identified by the display characteristic identification unit 103. That is, the imaging range calculation unit 500 calculates the imaging range of the display device at the current viewpoint of the virtual camera 400 (the position of the virtual camera 400) based on the instruction information for operating the virtual camera input from the virtual camera coordinate acquisition unit 104 and the physical information for identifying the imaging range of the display device input from the display characteristic identification unit 103.

The depth calculation unit 501 obtains position coordinates between a first position relative to the virtual camera 400 and a second position farther from the virtual camera 400 than the first position among the intersections of the optical axis 700 of the virtual camera 400 and the subject, and calculates the distance based on these position coordinates as the depth of the subject. For example, based on the position information of the virtual camera 400 and the coordinate position information of the subject, the depth calculation unit 501 obtains position coordinates between the nearest part (first position) that is closest to the virtual camera 400 and the farthest part (second position) that is farthest from the virtual camera 400 among the intersections of the optical axis direction of the virtual camera 400 and the subject for the subject selected by the subject coordinate acquisition unit 105, and calculates the distance between the two points as the depth of the subject.

The comparison unit 502 compares the imaging range 402 calculated by the imaging range calculation unit 500 with the depth of the subject calculated by the depth calculation unit 501 to determine whether the subject is included within the imaging range 402.

When the comparison result of the comparison unit 502 determines that the depth of the subject is not included within the imaging range 402 (subject depth > imaging range 402), the coordinate calculation unit 503 calculates the position of the virtual camera 400 by correcting the position coordinates of the virtual camera 400 so that the subject is included within the imaging range 402.

On the other hand, if the comparison result of the comparison unit 502 indicates that the depth of the subject is included within the imaging range 402 (subject depth ≦ imaging range 402), the coordinate transmission unit 504 updates the position coordinates of the virtual camera 400 based on the acquired position coordinates of the virtual camera 400 without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

The coordinate transmission unit 504 sets the position coordinates of the virtual camera 400 in the virtual camera position setting unit 107 based on the information of the comparison unit 502 or the position information of the virtual camera 400 corrected by the coordinate calculation unit 503.

(Processing of the First Embodiment)
6 is a flowchart for explaining the flow of processing according to the first embodiment, and processing for controlling the position of virtual camera 400 so that an arbitrary subject is included in the imaging range, which is a feature of the first embodiment of the present invention, will be explained using the flowchart in FIG. This control processing is executed by CPU 320.

First, in step S601, the physical information of the display device 403 connected to the operation information input unit 101 is referenced, and the operation information input unit 101 transmits the physical information of the display device 403 to the display characteristic identification unit 103.

In the next step S602, the display characteristic identification unit 103 identifies the position and size of the imaging range relative to the shooting range of the virtual camera from the physical information indicating the characteristics of each display device (display device) received from the operation information input unit 101 by referring to a table stored therein.

In step S603, the user selects a subject to be imaged on a monitor (not shown), and the operation information input unit 101 acquires, for example, the ID of the selected subject by referring to a table that holds the ID, and transmits the acquired ID to the subject coordinate acquisition unit 105 as subject information to be imaged.

In step S604, the subject coordinate acquisition unit 105 acquires the position coordinates of the subject in the virtual space associated with the received subject information (ID) by referring to the table stored therein. The subject coordinate acquisition unit 105 then transmits the acquired position coordinates of the subject to the operable area control unit 106.

Note that the present invention is not limited to selecting a subject to be imaged and controlling the position of virtual camera 400 so that the subject is included in the imaging range. It is also possible to select a subject that is not to be imaged. In this case, in the subsequent steps, control is performed to correct the position of virtual camera 400 so that the subject that is not to be imaged is not included in the imaging range.

In step S605, the user operates the position of the virtual camera 400. The operation information input unit 101 transmits operation instruction information for the virtual camera based on the user's operation (e.g., a movement vector that specifies the amount of movement of the virtual camera 400) to the virtual camera coordinate acquisition unit 104.

In step S606, the virtual camera coordinate acquisition unit 104 acquires the position coordinates of the virtual camera 400 in the virtual space based on the operation instruction information (movement vector) received from the operation information input unit 101. After that, the virtual camera coordinate acquisition unit 104 transmits the acquired position coordinates of the virtual camera 400 to the operationable area control unit 106.

In step S607, the imaging range calculation unit 500 calculates the display range 401 (capture range) that can be displayed on the display device 403 based on the position coordinates of the virtual camera 400 received in step S606.

In step S608, the imaging range calculation unit 500 calculates the imaging range 402 at the current position coordinates of the virtual camera 400 from a table of the position and size of the imaging range relative to the shooting range of the virtual camera 400. The imaging range calculation unit 500 then transmits the calculated imaging range 402 to the comparison unit 502.

In step S609, the depth calculation unit 501 calculates the optical axis 700 at the position coordinates of the virtual camera 400 received in step S606.

In step S610, the depth calculation unit 501 calculates the depth 701 of the subject located at the position coordinates calculated in step S604 in the direction of the optical axis 700 of the virtual camera 400. If there are multiple subjects to be imaged, for example, as shown in FIG. 7B, the distance between the subject 404 closest to the virtual camera 400 and the subject 405 farthest from the virtual camera 400 is found, and the distance between the two points is calculated as the depth 701.

In step S611, the comparison unit 502 determines whether the depth 701 calculated by the depth calculation unit 501 in step S610 is included within the imaging range 402 calculated by the imaging range calculation unit 500 in step S608. If the result of the determination in step S611 is that the depth 701 of the subject is included within the imaging range 402 (S611-Yes), the process proceeds to step S612. In step S612, the coordinate transmission unit 504 transmits the position coordinates of the virtual camera 400 acquired in step S606 to the virtual camera position setting unit 107. Here, the coordinate transmission unit 504 maintains the position of the viewpoint of the virtual camera 400 based on the position coordinates of the virtual camera 400 acquired in step S606 (updates the position without correction), without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

On the other hand, if the result of the judgment in S611 is that the depth 701 of the subject is not included within the imaging range 402 (S611-No), the process proceeds to step S613. In step S613, the coordinate calculation unit 503 corrects the position coordinates of the virtual camera 400 so that the depth 701 calculated in step S610 is included within the imaging range 402, and updates the position of the virtual camera using the corrected position coordinates. For example, as shown in FIG. 7C, the position coordinates of the virtual camera 400 can be corrected in a direction away from the subject along the optical axis 700 (in the direction of the arrow 702), thereby increasing the relative size of the imaging range 402 with respect to the subject. This allows the position of the virtual camera to be controlled so that the depth 701 of the subject is included within the expanded imaging range 402. This processing makes it possible to include the subject within the imaging range 402, from a state in which the subject 405 to be imaged cannot be imaged (as shown by the dashed line) as in Fig. 7(a) before the position coordinates are corrected, to a state in which the subject can be imaged (as shown by the solid line) as in Fig. 7(d) after the position coordinates are corrected. Note that the control for correcting the position coordinates of the virtual camera 400 is assumed to be corrected along the optical axis 700 so as not to shift the viewpoint, but the correction may be made in a direction deviating from the optical axis 700.

As described above, by controlling the processing to include an arbitrary subject within the imaging range of the display device, it becomes possible to generate a virtual viewpoint image without manually checking whether or not the subject is included within the imaging range. As a result, when displaying a virtual viewpoint image on a display device such as a naked eye stereoscopic display, it becomes possible to continue imaging even if the subject moves. For example, it is possible to continue imaging an arbitrary subject even in scenes where the subject moves continuously, such as in dancing or sports.

[Embodiment 2]
In this embodiment, a control process will be described in which the extent to which each subject is included in the imaging range of the display device is arbitrarily specified.

In the first embodiment, a method for controlling the position coordinates of the virtual camera 400 so that the subject to be imaged is completely contained within the imaging range 402 has been described. However, video production techniques that intentionally do not image the subject are also possible. For example, in the production of videos such as dramas and movies, a representation method may be used in which the camera's depth of field is narrowed and other subjects are blurred to a certain extent in order to make a target subject stand out. In the second embodiment, it is possible to accommodate such representation methods by making it possible to set a threshold value for the degree to which each subject is imaged.

In the following explanation, Fig. 8 will be used to explain the overall virtual viewpoint image generation system according to the second embodiment, Figs. 9 and 10 will be used to explain the functional configuration related to the process of specifying the range in which the subject is included in the imaging range, which is a feature of the second embodiment of the present invention, and Fig. 11 will be used to explain the flow of that process.

The same components as those in the first embodiment are designated by the same numbers, and their explanations are omitted here.

(Explanation of the virtual viewpoint image generation system 800)
8 is a block diagram for explaining the functional configuration of a virtual visual point image generating system 800 according to embodiment 2. The virtual visual point image generating system 800 includes an operation information input unit 801 and a virtual visual point image generating device 802.

(Operation Information Input Unit 801)
The operation information input unit 801 transmits to the virtual viewpoint image generating device 802 operation instruction information for the virtual camera for operating the virtual viewpoint, physical information for identifying the physical imaging range of the display device, subject information to be imaged, and threshold information for each subject.

The threshold information for each subject is a threshold that arbitrarily specifies the extent (proportion) of the subject's inclusion in the imaging range of the display device, and is additionally set when the user selects the subject as the imaging target. The threshold information is set as a percentage, for example, 50% or 100%, and the depth 701 is corrected based on the set threshold information. Note that the method of setting the threshold information may be a setting that indicates a threshold level, such as "high" priority or "low" priority, in addition to a percentage.

The virtual viewpoint image generating device 802 generates a virtual viewpoint image by controlling the position of the virtual camera so that the subject in the virtual space is included in the imaging range of the display device based on each piece of information transmitted from the operation information input unit 801. The virtual viewpoint image generating device 802 differs from the virtual viewpoint image generating device 102 described in the first embodiment in that it has a threshold setting unit 803 and an operable area control unit 804 as functional components.

The threshold setting unit 803 identifies the threshold information transmitted from the operation information input unit 801, which indicates the percentage of each subject to be included in the imaging range 402. For example, the threshold (percentage) when the entire subject is included in the imaging range 402 is set to 100%. When the threshold information transmitted from the operation information input unit 801 is transmitted with a priority such as "high" or "low", the threshold setting unit 803 modifies the threshold to the threshold set as the high priority (e.g., 70%) or the threshold set as the low priority (e.g., 30%).

The operable area control unit 804 controls the position of the virtual camera so that the subject is included within the imaging range at a ratio equal to the specified threshold, based on the information received from the display characteristic identification unit 103, the virtual camera coordinate acquisition unit 104, the subject coordinate acquisition unit 105, and the threshold setting unit 803. The operable area control unit 804 transmits the controlled position of the virtual camera to the virtual camera position setting unit 107, and the virtual camera position setting unit 107 actually sets the position of the virtual camera based on the information on the position of the virtual camera controlled by the operable area control unit 804.

(Configuration of the operable area control unit 804)
9 is a block diagram for explaining the functional configuration of an operable area control unit 804 according to the second embodiment of the present invention. The operable area control unit 804 has a functional configuration in which a depth calculation unit 900 is changed from the functional configuration of the operable area control unit 106 of the first embodiment.

The depth calculation unit 900 calculates the depth based on the threshold information set for each subject acquired from the threshold setting unit 803, the position information of the virtual camera 400, and the coordinate position information of the subject. The depth calculation unit 900 transmits the calculated depth to the comparison unit 502. The comparison unit 502 compares the imaging range 402 calculated by the imaging range calculation unit 500 with the depth of the subject calculated by the depth calculation unit 900, and determines whether the imaging range 402 includes the depth of the subject based on the threshold information. The result of the comparison by the comparison unit 502 is transmitted to the coordinate calculation unit 503 or the coordinate transmission unit 504, and thereafter, the position coordinates of the virtual camera 400 are transmitted to the virtual camera position setting unit 107 through the same processing as in the first embodiment.

(Explanation of Depth Calculation Unit 900)
10 is a diagram for explaining an example of calculation of depth by the depth calculation unit 900 according to the second embodiment. The subject 404 is located in front of the position of the virtual camera 400, and the subject 405 is located behind the position of the virtual camera 400. In this example, the threshold value set for the subject 404 is 100%, and the threshold value set for the subject 405 is 50%. In this case, a depth 1002 is calculated as the distance from the closest position of the subject 404 to the virtual camera 400 to a position that is half the depth 1001 of the subject 405 itself (corresponding to the set threshold value: 50%).

(Processing of the Second Embodiment)
Fig. 11 is a flowchart for explaining the flow of processing according to the second embodiment, and will be used to explain processing for controlling the position of the virtual camera 400 based on a threshold value for arbitrarily specifying the extent to which each subject is to be included in the imaging range of the display device, which is a feature of the second embodiment of the present invention. This processing is executed by the CPU 320. Processing similar to that in the processing flowchart (Fig. 6) explained in the first embodiment is given the same step number, and explanations will be omitted.

First, in step S1101, the user sets threshold information for each subject to be imaged selected in step S603. The set threshold information is transmitted from the operation information input unit 801 to the threshold setting unit 803.

In step S1102, the depth calculation unit 900 calculates the depth of the subject based on the coordinate position information of the subject (S604), the position information of the virtual camera 400 (S606), and the threshold information set for each subject obtained from the threshold setting unit 803 (S1101). The depth calculation unit 900 transmits the calculated depth 1102 to the comparison unit 502.

The control process thereafter is the same as that according to the first embodiment. If the result of the determination in step S611 is that the depth of the subject based on the threshold information set for each subject is included within the imaging range 402 (S611-Yes), the process proceeds to step S612. In step S612, the coordinate transmission unit 504 transmits the position coordinates of the virtual camera 400 acquired in step S606 to the virtual camera position setting unit 107. Here, the coordinate transmission unit 504 updates the position coordinates of the virtual camera 400 based on the position coordinates of the virtual camera 400 acquired in step S606, without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

On the other hand, if the result of the determination in S611 is that the depth of the subject based on the threshold information set for each subject is not included within the imaging range 402 (S611-No), the process proceeds to step S613. In step S613, the coordinate calculation unit 503 corrects the position coordinates of the virtual camera 400 and controls the position of the virtual camera 400 so that the depth 701 calculated in step S610 is included within the imaging range 402.

In the first embodiment described above, an example was described in which the position coordinates of the virtual camera 400 are corrected in a direction away from the subject along the optical axis 700 (in the direction of the arrow 702) as shown in FIG. 7(c) in order to include the depth 701 of the subject in the imaging range 402, thereby increasing the relative size of the imaging range 402 for the subject.

In the second embodiment, in addition to the processing example described in FIG. 7(c), for example, when it is necessary to reduce the relative size of the imaging range for the subject in order to include the depth of the subject for which a threshold is set in the imaging range, it is also possible to modify the position coordinates in a direction that brings the virtual camera 400 closer to the subject along the optical axis 700 (the opposite direction of the arrow 702).

As described above, when a virtual viewpoint image is displayed on a display device such as a naked-eye stereoscopic display, it is possible to continue to image the subject even if the subject moves. Furthermore, by performing a process that arbitrarily specifies the range to be included in the imaging range of each subject based on threshold information set for each subject, it is possible to generate a virtual viewpoint image in which only a specific subject is imaged and other subjects are blurred to a certain extent.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: Virtual viewpoint image generation system,
101: operation information input unit, 102: virtual viewpoint image generating device,
103: display characteristic identification unit, 104: virtual camera coordinate acquisition unit,
105: Subject coordinate acquisition unit, 106: Operable area control unit,
107: virtual camera position setting unit, 400: virtual camera,
402: Imaging range, 403: Display device

Claims (13)

a control means for controlling a position of the virtual camera, which is a virtual camera located in a virtual space, so that the position of a subject in the virtual space is included in an image formation range of the display means relative to a shooting range of the virtual camera;
a generating means for generating a virtual viewpoint image based on the position of the virtual camera controlled by the control means;
Equipped with
The control means
a depth calculation means for calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject;
a comparison means for comparing the imaging range with the depth of the subject,
An image generating device characterized in that, when a comparison by said comparison means shows that the depth of the subject is not included in the imaging range, the position of the virtual camera is controlled so that the depth of the subject is included in the imaging range. an input means for inputting operation instruction information for operating the virtual camera, physical information for identifying an imaging range of the display means, and subject information for identifying a selected subject;
a camera coordinate acquisition means for acquiring a position of the virtual camera in the virtual space based on the operation instruction information;
an identification means for identifying a position and a size of the imaging range relative to a shooting range of the virtual camera based on the physical information;
a subject coordinate acquisition means for acquiring a position of the subject in the virtual space based on the subject information;
2. The image generating apparatus of claim 1, further comprising: The control means
3. The image generating device according to claim 1, further comprising an imaging range calculation means for calculating a shooting range based on a position of the virtual camera, and calculating the imaging range at the position of the virtual camera from a table that holds the position and size of the imaging range for the shooting range. 4. The image generating device according to claim 1, wherein the control means controls the position of the virtual camera in a direction away from the subject when the depth of the subject is not included in the imaging range. The image generating device according to claim 4, characterized in that the control means controls the position of the virtual camera along the optical axis of the virtual camera in a direction away from the subject. 4. The image generating device according to claim 1 , wherein the control means maintains the position of the virtual camera when the comparison by the comparison means indicates that the depth of the subject is included in the imaging range. The image generating device according to any one of claims 1 to 6, characterized in that the depth calculation means determines position coordinates between a first position with respect to the virtual camera and a second position, which is farther from the virtual camera than the first position, among the intersections of the optical axis of the virtual camera and the subject, and calculates a distance based on the position coordinates as the depth of the subject. a threshold setting unit that sets a ratio of the depth of the subject to the imaging range,
the depth calculation means calculates a depth of the subject based on a position of the subject, a position of the virtual camera, and the ratio;
8. The image generating device according to claim 1 , wherein the control means controls the position of the virtual camera so that the depth of the subject for which the ratio is set is included in the imaging range. The image generating device according to claim 8, characterized in that, when there are multiple subjects, the threshold setting means sets the ratio for each of the multiple subjects. The image generating device according to claim 8 or 9, characterized in that the depth calculation means corrects the depth of the subject based on the ratio set by the threshold setting means. The image generating device according to any one of claims 1 to 10, characterized in that the display means includes a naked eye stereoscopic display. 1. An image generation method for an image generation device that generates a virtual viewpoint image, comprising:
controlling a position of the virtual camera such that an image-forming range of a display means for a shooting range of the virtual camera, which is a virtual camera located in a virtual space, includes a position of a subject in the virtual space;
generating the virtual viewpoint image based on the controlled position of the virtual camera ;
Controlling the position of the virtual camera includes:
calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject;
Comparing the imaging range with the depth of the subject;
An image generating method characterized by including, when the comparison shows that the depth of the subject is not included in the imaging range, controlling the position of the virtual camera so that the depth of the subject is included in the imaging range. A program for causing a computer to function as each of the means of the image generating device according to any one of claims 1 to 11.

Images