JP7683607B2 - Information processing device and image data generating method - Google Patents

Description

The present disclosure relates to an information processing device and a method for generating image data.

Conventionally, a method has been proposed in which 3D objects are generated within a viewing space using information sensed from real 3D space, such as multi-viewpoint video in which a subject is captured from different viewpoints, and an image (volumetric image) in which the object appears to exist within the viewing space.

Furthermore, Patent Document 1 discloses an example of superimposing information related to the subject on the generated volumetric image.

International Publication No. 2019/021375

However, in Patent Document 1, what is superimposed on the subject is information related to the subject, and a different image is not superimposed. Therefore, there was a problem that it could not be applied to generating images that switch between a real image of the subject captured by an actual camera and a virtual image of the subject observed from a virtual viewpoint.

This disclosure proposes an information processing device and a method for generating image data that can switch between real images captured by an actual camera and virtual images.

In order to solve the above problem, one form of information processing device according to the present disclosure is an information processing device that includes an image generation unit that generates a composite image by combining a real image including a first object captured by a first imaging device, a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information, and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, at a combination ratio according to the viewpoint information.

1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment; FIG. 1 illustrates an example of an imaging device. 11 is a flowchart illustrating an example of a flow of processing performed by the information processing system. FIG. 2 is a hardware block diagram showing an example of a hardware configuration of an information processing system. FIG. 2 is a functional block diagram illustrating an example of a functional configuration of a rendering unit. FIG. 2 is a diagram illustrating an example of an arrangement of a real camera and a virtual camera. FIG. 13 is a diagram illustrating a method of superimposing a plurality of images. 13A and 13B are diagrams illustrating an example of setting a blending ratio according to the position of a virtual viewpoint. FIG. 11 is a diagram showing an example of a result of superimposing images. 4 is a flowchart illustrating an example of a flow of a process performed by the information processing system of the first embodiment. 13A and 13B are diagrams illustrating another example of setting the superimposition ratio according to the position of the virtual viewpoint. FIG. 1 is a diagram illustrating an overview of an information processing system according to a second embodiment. FIG. 2 is a functional block diagram showing an example of a functional configuration of a mobile terminal. 13A and 13B are diagrams illustrating an example of setting a blending ratio according to the position of a virtual viewpoint in the second embodiment. 13 is a flowchart illustrating an example of a flow of a process performed by an information processing system according to a second embodiment.

Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are designated by the same reference numerals, and duplicated descriptions will be omitted.

The present disclosure will be described in the following order.
1. First embodiment 1-1. Functional configuration of information processing system 1-2. Processing flow performed by the information processing system 1-3. Hardware configuration of the information processing system 1-4. Detailed configuration of the rendering unit 1-5. Operation of the first embodiment 1-6. Processing flow of the first embodiment 1-7. Modified example of the first embodiment 1-8. Effects of the first embodiment 2. Second embodiment 2-1. Overview of the information processing system 2-2. Functional configuration of the information processing system 2-3. Operation of the second embodiment 2-4. Processing flow of the second embodiment 2-5. Effects of the second embodiment 3. Application examples of the present disclosure 3-1. Content production 3-2. Experience in virtual space 3-3. Application to communication with remote locations 3-4. Others

1. First embodiment
[1-1. Functional configuration of information processing system]
First, an overview of an information processing system 10a to which the present disclosure is applied will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment.

As shown in FIG. 1, the information processing system 10a includes a data acquisition unit 11, a 3D model generation unit 12, an encoding unit 13a, a transmission unit 14, a receiving unit 15, a decoding unit 16a, a rendering unit 17a, and a display unit 18.

The data acquisition unit 11 acquires image data for generating a 3D model 90M of a subject 90, which is an object to be imaged. For example, as shown in FIG. 2, a plurality of viewpoint images captured by a plurality of real cameras 70 (70a, 70b, 70c, 70d, 70e, ...) arranged to surround the subject 90 are acquired as image data. In this case, it is preferable that the plurality of viewpoint images are images captured synchronously by the plurality of real cameras 70. In addition, the data acquisition unit 11 may acquire image data of the subject 90 captured from a plurality of viewpoints by moving one real camera 70, for example. The data acquisition unit 11 may perform calibration based on the image data and acquire internal parameters and external parameters of each real camera 70. In addition, the data acquisition unit 11 may acquire a plurality of depth information indicating the distance from a plurality of viewpoints to the subject 90. In addition, the real cameras 70 (70a, 70b, 70c, 70d, 70e, ...) are an example of a first imaging device in this disclosure. Moreover, subject 90 is an example of a first object in this disclosure.

The 3D model generating unit 12 generates a model having three-dimensional information of the subject based on image data for generating a 3D model 90M of the subject 90. The 3D model generating unit 12 generates a 3D model of the subject by, for example, using a so-called Visual Hull to carve out the three-dimensional shape of the subject using images from multiple viewpoints (for example, silhouette images from multiple viewpoints). In this case, the 3D model generating unit 12 can further deform the 3D model 90M generated using the Visual Hull with high accuracy using multiple depth information indicating the distances from multiple viewpoints to the subject. The 3D model generating unit 12 may also generate the 3D model 90M of the subject 90 from one captured image of the subject 90.

The 3D model 90M generated by the 3D model generating unit 12 can be said to be a 3D model video by generating it in units of frames in a time series. In addition, the 3D model 90M can be said to be a live-action 3D model because it is generated using images captured by an imaging device. The 3D model can be expressed in the form of three-dimensional shape mesh data, which expresses the shape information representing the surface shape of the subject 90, for example, in the form of a polygon mesh, which expresses the connection between vertices. The three-dimensional shape mesh data has, for example, three-dimensional coordinates of the vertices of the mesh and index information indicating which vertices are combined to form a triangular mesh. Note that the method of expressing the 3D model is not limited to these, and it may be described using a so-called point cloud expression method that expresses the position information of points. Color information data is also generated as a texture in a form linked to these 3D shape data. There are cases of View Independent textures, which have a constant color from any direction, and View Dependent textures, which change color depending on the viewing direction.

The encoding unit 13a converts the data of the 3D model 90M generated by the 3D model generating unit 12 into a format suitable for transmission and storage. For example, the 3D model generated by the 3D model generating unit 12 may be converted into multiple two-dimensional images by perspective projection from multiple directions. In this case, the 3D model may be used to generate depth information, which is a two-dimensional depth image from multiple viewpoints. The depth information and color information in this two-dimensional image state are compressed and output to the transmitting unit 14. The depth information and color information may be transmitted side by side as one image, or may be transmitted as two separate images. In this case, since the data is in the form of two-dimensional image data, it can also be compressed using a two-dimensional compression technique such as AVC (Advanced Video Coding). Also, for example, the 3D data may be converted into a point cloud format, or may be output to the transmitting unit 14 as three-dimensional data. In this case, for example, a three-dimensional compression technique of Geometry-based-Approach discussed in MPEG can be used.

The transmitting unit 14 transmits the transmission data formed by the encoding unit 13a to the receiving unit 15. The transmitting unit 14 transmits the transmission data to the receiving unit 15 after performing a series of processes by the data acquiring unit 11, the 3D model generating unit 12, and the encoding unit 13a offline. The transmitting unit 14 may also transmit the transmission data generated from the series of processes described above to the receiving unit in real time.

The 3D model generation unit 12, the encoding unit 13a, and the transmission unit 14 constitute an encoding device 40a. The encoding device 40a is an example of an information processing device in the present disclosure.

The receiving unit 15 receives the transmission data transmitted from the transmitting unit 14.

The decoding unit 16a restores the bit stream received by the receiving unit 15 into a two-dimensional image, and restores mesh and texture information that can be drawn by the rendering unit 17a from the restored two-dimensional image.

The receiving unit 15, the decoding unit 16a, and the rendering unit 17a constitute a decoding device 50a. The decoding device 50a is an example of an information processing device in the present disclosure.

The rendering unit 17a performs texture mapping by projecting the mesh of the 3D model 90M from the viewpoint of the rendering camera and pasting a texture representing a color or pattern. The feature of this embodiment is that the rendering at this time can be viewed from any viewpoint that is set arbitrarily, regardless of the camera position at the time of image capture. Texture mapping includes a method called View Dependent (VD method) that takes into account the user's viewing viewpoint, and a method called View Independent (VI method) that does not take into account the user's viewing viewpoint. The VD method has the advantage of being able to achieve higher quality rendering than the VI method because the texture pasted on the 3D model changes depending on the position of the viewing viewpoint. On the other hand, the VI method has the advantage of requiring less processing than the VD method because it does not take into account the position of the viewing viewpoint. The data on the viewing viewpoint is input to the rendering unit 17a from the display device, which detects the user's viewing location (Region of Interest).

The display unit 18 displays the results rendered by the rendering unit 17a on the display unit of the display device. The display device may be, for example, a head-mounted display, a spatial display, a mobile phone, a television, a PC, or a 2D or 3D monitor.

In addition, the information processing system 10a in FIG. 1 shows a series of flows from the data acquisition unit 11 that acquires the captured images that are the material for generating the content to the display unit 18 that controls the display device on which the user views. However, this does not mean that all functional blocks are necessary to implement this embodiment, but this embodiment can be implemented for each functional block or for a combination of multiple functional blocks. For example, FIG. 1 provides a transmission unit 14 and a reception unit 15 to show a series of flows from the side that creates the content to the side that views the content through the distribution of content data, but if the content is created and viewed on the same information processing device (e.g., a personal computer), it is not necessary to provide the transmission unit 14 or the reception unit 15.

When implementing the information processing system 10a, the same implementer may implement everything, or different implementers may implement each functional block. For example, operator X generates 3D content through the data acquisition unit 11, 3D model generation unit 12, and encoding unit 13a. In some cases, multiple implementers work together to deliver the 3D content through operator Y's transmission unit 14 (platform), and operator Z's display device receives, renders, and controls the display of the 3D content.

In addition, each of the above-mentioned functional blocks can be implemented on the cloud. For example, the rendering unit 17a may be implemented within the display device or on a server. In that case, information is exchanged between the display device and the server.

1 describes the data acquisition unit 11, 3D model generation unit 12, encoding unit 13a, transmission unit 14, reception unit 15, rendering unit 17a, and display unit 18 collectively as information processing system 10a. However, in this specification, information processing system 10a is referred to as an information processing system if two or more functional blocks are involved, and for example, the data acquisition unit 11, 3D model generation unit 12, encoding unit 13a, transmission unit 14, reception unit 15, decoding unit 16a, and rendering unit 17a can be collectively referred to as information processing system 10a without including display unit 18.

[1-2. Flow of processing performed by information processing system]
Next, the flow of processing performed by the information processing system 10a will be described with reference to Fig. 3. Fig. 3 is a flow chart showing an example of the flow of processing performed by the information processing system.

When the process starts, in step S11, the data acquisition unit 11 acquires image data for generating a 3D model of the subject.

Next, in step S12, the 3D model generation unit 12 generates a 3D model having three-dimensional information of the subject based on the image data for generating the 3D model of the subject.

In step S13, the encoding unit 13a encodes the shape and texture data of the 3D model generated by the 3D model generation unit 12 into a format suitable for transmission and storage.

Then, in step S14, the transmitting unit 14 transmits the encoded data, and in step S15, the receiving unit 15 receives the transmitted data.

In step S16, the decoding unit 16a performs a decoding process and converts the 3D model into shape and texture data required for display. Then, in step S17, the rendering unit 17a performs rendering using the shape and texture data. The rendering result is displayed on the display unit in step S18. Then, when the processing of step S18 ends, the processing of the information processing system 10a ends.

[1-3. Hardware configuration of information processing system]
Next, the hardware configuration of the information processing system 10a will be described with reference to Fig. 4. Fig. 4 is a hardware block diagram showing an example of the hardware configuration of the information processing system.

In Fig. 4, the CPU 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input/output interface 25 is also connected to the bus 24. An input unit 26, an output unit 27, a memory unit 28, a communication unit 29, and a drive 30 are connected to the input/output interface 25.

The input unit 26 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 27 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 28 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 29 includes, for example, a network interface, etc. The drive 30 drives removable media such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The computer configured as above performs the above-mentioned series of processes by the CPU 21, for example, loading a program stored in the storage unit 28 into the RAM 23 via the input/output interface 25 and the bus 24 and executing the program. The RAM 23 also stores data and the like necessary for the CPU 21 to execute various processes as appropriate.

The program executed by the computer can be applied by recording it on a removable medium such as a package medium. In this case, the program can be installed in the storage unit 28 via the input/output interface by inserting the removable medium into the drive.

The program can also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 29 and installed in the storage unit 28.

[1-4. Detailed configuration of the rendering unit]
Next, a detailed configuration of the rendering unit 17a will be described with reference to Fig. 5. Fig. 5 is a functional block diagram showing an example of the functional configuration of the rendering unit.

As shown in FIG. 5, the rendering unit 17a includes a virtual image generation unit 51, a viewpoint setting unit 52, a superimposition ratio calculation unit 53, a 3DCG object image generation unit 54, and an image synthesis unit 55.

The virtual image generating unit 51 acquires the real image captured by the real camera 70, the 3D model 90M of the subject 90 generated by the 3D model generating unit 12 (see FIG. 1), and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52, and generates a virtual image V (see FIG. 7) of the subject 90 viewed from the virtual viewpoint Vp (see FIG. 6). The 3D model 90M includes shape information and texture information of the subject 90 acquired by each of the multiple real cameras 70. The subject 90 is an example of an object in this disclosure.

The virtual image V generated by the virtual image generating unit 51 is a so-called volumetric image in which a 3D model 90M of the subject 90 is projected onto a virtual viewpoint Vp. The method of generating the virtual image V is not limited, but for example, a 3D model 90M of the subject 90 is generated by a method such as Visual Hull using real images captured by multiple real cameras 70. The generated 3D model 90M is then projected onto the virtual viewpoint Vp to generate a virtual image V in which the subject 90 is viewed from the virtual viewpoint Vp. The virtual image V is an example of a first virtual image in this disclosure.

The viewpoint setting unit 52 sets the position of the virtual viewpoint Vp and the observation direction from the virtual viewpoint Vp to the subject 90 (first object) over time, i.e., according to time t. Note that the virtual viewpoint Vp is an example of viewpoint information in this disclosure.

The superimposition ratio calculation unit 53 receives the installation positions of the real cameras 70a and 70b, the real camera accuracy information, and the 3D model 90M of the subject 90 from the 3D model generation unit 12 (see FIG. 1). The superimposition ratio calculation unit 53 then calculates the superimposition ratio r of the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib when generating a composite image L (see FIG. 9) of the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib at the virtual viewpoint Vp set by the viewpoint setting unit 52. The real image Ia is an example of the first real image in this disclosure. The real image Ib is an example of the second real image in this disclosure. The superimposition ratio r is an example of the composite ratio in this disclosure. The real camera accuracy information is calibration information related to the real cameras 70a and 70b (external calibration information related to the installation positions of the real cameras, and internal parameter information related to the optical parameters of the real cameras), etc.

The real images Ia and Ib (see FIG. 7) are images including the subject 90 actually captured by the real cameras 70a and 70b, respectively. The superimposition ratio r specifies the superimposition ratios (brightness ratio, color ratio) of each image to be superimposed and the 3DCG object image O when the image synthesis unit 55 superimposes the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib to generate a synthetic image L. When the superimposition ratio of the real image Ia is _rca (0≦ _rca ≦1), the superimposition ratio of the real image Ib is _rcb (0≦ _rcb ≦1), the superimposition ratio of the virtual image V is _rV (0≦ _rV ≦1), and the superimposition ratio of the 3DCG object image O is _r3DCG (0≦ _r3DCG ≦1), the superimposition ratios _rca , _rcb , and _rV satisfy the condition of formula (1). The blending ratio r is set according to the position of the virtual viewpoint Vp, as will be described in detail later.

r _ca + r _cb + r _V =1...(1)

The 3DCG object image O is synthesized by alpha blending an image in which the real image Ia, the virtual image V, and the real image Ib are superimposed. Details will be described later (see Figure 7).

The 3DCG object image generating unit 54 generates a 3DCG object image O to be superimposed on the composite image L. The 3DCG object image generating unit 54 may generate an image including a character having 3D information as the 3DCG object image O, or may generate a visual effect to be applied to the entire composite image L. Specifically, the 3DCG object image generating unit 54 generates a 3DCG object image O in which the 3DCG object is viewed from a virtual viewpoint Vp by projecting a 3D model 92 possessed by the 3DCG object onto a virtual viewpoint Vp. The 3DCG object image O is an example of a second virtual image in the present disclosure. The 3DCG object is an example of a second object in the present disclosure.

The image synthesis unit 55 generates a synthetic image L by synthesizing real images Ia and Ib including a subject 90 having a 3D model 90M captured from different positions by the real camera 70 (first imaging device), a virtual image V obtained by observing the subject 90 from a virtual viewpoint Vp set by the viewpoint setting unit 52 according to time t, and a 3DCG object image O having a 3D model 92 viewed from the virtual viewpoint Vp, at superimposition ratios r and r _3DCG calculated by the superimposition ratio calculation unit 53 according to the position of the virtual viewpoint Vp. Note that the image synthesis unit 55 is an example of an image generation unit in the present disclosure.

[1-5. Operation of the First Embodiment]
Next, the operation of the decoding device 50a of the first embodiment will be specifically described with reference to Fig. 6 to Fig. 9. Fig. 6 is a diagram showing an example of the arrangement of a real camera and a virtual camera. Fig. 7 is a diagram explaining a method of superimposing a plurality of images. Fig. 8 is a diagram showing an example of setting a superimposition ratio according to the position of a virtual viewpoint. Fig. 9 is a diagram showing an example of the superimposition result of each image.

In the information processing system 10a, the viewpoint setting unit 52 sets the virtual viewpoint Vp to move from the real viewpoint position of the real camera 70a to the real viewpoint position of the real camera 70b along a movement path 88a according to time t, as shown in, for example, FIG. 6.

More specifically, the virtual viewpoint Vp moves to the position of virtual camera 72a at time t1 while continuing to capture subject 90 having 3D model 90M. Thereafter, the virtual viewpoint Vp moves to the position of virtual camera 72b at time t2, to the position of virtual camera 72c at time t3, to the position of virtual camera 72d at time t4, to the position of virtual camera 72e at time t5, and to the position of real camera 70b at time t6.

The superimposition ratio calculation unit 53 calculates the superimposition ratio r between the real images Ia, Ib in the composite image L, the virtual image V obtained by viewing the subject 90 from the virtual viewpoint Vp, and the 3DCG object image O viewed from the virtual viewpoint Vp, based on the spatial distance (distance) between the real camera 70a or real camera 70b and the virtual viewpoint Vp, and the temporal distance from the time when the virtual viewpoint Vp is at the position of the real camera 70a or real camera 70b, i.e., the moving speed of the virtual viewpoint, etc.

7 to generate a composite image L. Note that the description will be given on the assumption that the superimposition ratio calculation unit 53 has calculated the superimposition ratio r _ca of the real image Ia, the superimposition ratio r _cb of the real image Ib, the superimposition ratio r _V of the virtual image V, and the superimposition ratio r _3DCG of the 3DCG object image O.

As shown in FIG. 7, the image synthesis unit 55 includes multipliers M1, M2, M3, M4, and M5, adders A1 and A2, a 3D object superimposition unit 56, and a signal selection unit 57.

Multiplier M1 multiplies real image Ia by the superimposing ratio _rca . Multiplier M2 multiplies real image Ib by the superimposing ratio _rcb . Multiplier M3 multiplies virtual image V by the superimposing ratio _rV . Multiplier M4 multiplies 3DCG object image O by the superimposing ratio _r3DCG . Multiplier M5 multiplies the sum of the output of multiplier M1, the output of multiplier M2, and the output of multiplier M3 by the superimposing ratio (1- _r3DCG ).

Adder A1 adds the output of multiplier M1, the output of multiplier M2, and the output of multiplier M3.

The 3D object superimposition unit 56 generates a superimposed image Ix2 by further compositing a 3DCG object image O onto a superimposed image Ix1 obtained by superimposing the real image Ia, the real image Ib, and the virtual image V. The 3DCG object image O is composited, for example, by alpha blending. Alpha blending is one of the techniques for compositely combining a plurality of images, and is a method of superimposing images while taking into account the transmittance. In other words, the superimposition ratio _r3DCG of the 3DCG object image O represents the transmittance of the 3DCG object image O.

The signal selection unit 57 is a selector that selects a superimposed image Ix1 (input A) obtained by superimposing the real image Ia, the real image Ib, and the virtual image V, and a superimposed image Ix2 (input B). The signal selection unit 57 selects the superimposed image Ix2 (input B) for an area where a 3DCG object exists. On the other hand, the signal selection unit 57 selects the superimposed image Ix1 (input A) for an area where a 3DCG object does not exist. Note that whether or not an area has a 3DCG object is determined based on a 3DCG object presence map Mo. The 3DCG object presence map Mo is an image that indicates the location of the 3DCG object. The 3DCG object presence map Mo is used to identify an area to be used for synthesis by the image synthesis unit 55 from the 3DCG object image O in which the 3DCG object is viewed from the virtual viewpoint Vp.

Then, the image synthesis unit 55 outputs the output of the signal selection unit 57 as a synthetic image L to the display unit 18.

In addition, when the 3DCG object image O is an image that applies a visual effect to the entire composite image L, the signal selection unit 57 selects the superimposed image Ix2 (input B) for the entire image.

7, in order to realize a more complex superimposition, a foreground area map showing the location of the subject 90 in the real images Ia and Ib may be used. Also, a depth map showing the distance to the subject 90 in the virtual image V may be used.

Next, an example of setting the blending ratio r ( _rca , _rcb , _rv , _r3DCG ) according to the position of the virtual viewpoint Vp will be described with reference to FIG. 8.

First, the superimposition ratio _rca of the real image Ia will be described. The superimposition ratio calculation unit 53 gradually decreases the superimposition ratio _rca , which is set to 100% at the position of the real camera 70a, from around time t1, and simultaneously gradually increases the superimposition ratio _rV , which is set to 0% at the position of the real camera 70a, from around time t1. Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV to 100% at time t when the superimposition ratio _rca reaches 0%.

Then, the superimposition ratio calculation unit 53 gradually decreases the superimposition ratio _{r_V} around time t4, and simultaneously gradually increases the superimposition ratio _{r_cb} of the actual image Ib. The superimposition ratio _{r_V} is set to 0% around time t5, and the superimposition ratio _{r_cb} is set to 100% around time t5. Note that the superimposition ratio _{r_ca} is maintained at 0% until time t6.

In this way, when the distance between the real camera 70a and the virtual viewpoint Vp is equal to or smaller than a predetermined value, and when the distance between the real camera 70b and the virtual viewpoint Vp is equal to or smaller than a predetermined value, the superimposition ratio calculation unit 53 sets a lower superimposition ratio _rV than when the distance is greater than the predetermined value. This increases the contribution of the real images Ia and Ib to the composite image L when the virtual viewpoint Vp is near the real cameras 70a and 70b.

Qualitatively, when the virtual viewpoint Vp is located midway between the real cameras 70a and 70b (near time t3 in FIG. 8), the superimposition ratio _rV of the virtual image V is set to 100%. However, when the real cameras 70a and 70b are disposed close to each other, the superimposition ratio _rV of the virtual image _V is estimated to be low, and may not reach 100%.

Furthermore, the superimposition ratio calculation unit 53 sets the superimposition ratio r _3DCG of the 3DCG object image O. In the example of Fig. 8, the superimposition ratio calculation unit 53 sets the superimposition ratio r _3DCG to 100% from around time t1 to around time t5, and then changes the superimposition ratio r _3DCG to 0% at times before and after that.

The superimposition ratio calculation unit 53 may also set the rate of change of the superimposition ratio r ( _rca , _rcb , _rV , _r3DCG ) according to the moving speed of the virtual viewpoint Vp. For example, when the moving speed of the virtual viewpoint Vp is slow, it is desirable to set the rate of change of the superimposition ratio r ( _rca , _rcb , rV _{, r3DCG} ) per unit time to be small. On the other hand, when the moving speed of the virtual viewpoint Vp is fast, it is desirable to set the rate of change of the superimposition ratio r ( _rca , _rcb , _rV , _r3DCG ) per unit time to be large. This makes it possible to make the change in the composite image L accompanying the movement of the virtual viewpoint Vp less noticeable.

Note that the setting pattern of the overlap ratio r is not limited to the example in Figure 8. Other setting patterns of the overlap ratio r will be described later (see Figure 11).

In addition, the superimposition ratio calculation unit 53 may simultaneously control not only the superimposition ratio r of each image, but also the color intensity of each image, the degree of blur of each image, etc.

Next, an example of a composite image generated by the image synthesis unit 55 will be described with reference to Figure 9. Figure 9 shows real images Ia and Ib observed at each time from time t0 to time t6 shown in Figure 6, a virtual image V generated from the real images Ia and Ib, and a 3DCG object image O representing the visual effect at each time. Figure 9 also shows a composite image L obtained by synthesizing these images at each time. Note that while the real images Ia and Ib include the background, the virtual image V was generated only for the area of the subject 90 extracted from the real images Ia and Ib.

The virtual image V is generated based on the real image Ia and the 3D model 90M of the subject 90 depicted in the real image Ib. Therefore, although it is not fully shown in FIG. 9, strictly speaking, the virtual image V is an image in which the real images Ia and Ib overlap.

Also, in Figure 9, 3DCG object image O is a visual effect that depicts particles falling randomly across the entire image.

The composite image L is generated based on the superimposition ratio r set by the superimposition ratio calculation unit 53. Figure 9 shows an example in which the superimposition ratio r is set to be approximately the same as that in Figure 8.

[1-6. Processing flow of the first embodiment]
Next, the flow of processing performed by the information processing system 10a will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of the flow of processing performed by the information processing system of the first embodiment.

The viewpoint setting unit 52 sets a virtual viewpoint Vp for each time (step S21).

The image synthesis unit 55 acquires a real image Ia captured by the real camera 70a and a real image Ib captured by the real camera 70b (step S22).

The virtual image generation unit 51 generates a virtual image V based on the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generation unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52 (step S23).

The 3DCG object image generation unit 54 generates a 3DCG object image O viewed from the virtual viewpoint Vp (step S24).

The blending ratio calculation unit 53 sets the blending ratio r ( _rca , _rcb , _rv , _r3DCG ) (step S25).

The image synthesis unit 55 generates a synthetic image L (step S26).

The image synthesis unit 55 outputs the synthetic image L to the display unit 18 (step S27).

The viewpoint setting unit 52 determines whether the virtual viewpoint Vp has moved to the final position (step S28). If it is determined that the virtual viewpoint Vp has moved to the final position (step S28: Yes), the information processing system 10a ends the processing of Fig. 10. On the other hand, if it is not determined that the virtual viewpoint Vp has moved to the final position (step S28: No), the process returns to step S21.

It should be noted that the above-mentioned operational example is an example in which the volumetric image and the 3DCG object are each converted into a two-dimensional image, and then the composite image L is generated. Of course, the volumetric image and the 3DCG object may be composited as 3D models, and the composite image L may be generated by projecting the composite 3D model onto a two-dimensional image.

For example, a 3D model 90M of a subject 90 and a 3D model 92 of a 3DCG object may be three-dimensionally synthesized based on three-dimensional position information of each 3D model and three-dimensional position information of a virtual viewpoint Vp. Then, a synthetic image L may be generated by projecting the synthesized three-dimensional information onto a two-dimensional virtual image V observed from the virtual viewpoint Vp.

[1-7. Modification of the first embodiment]
The superimposition ratio r set by the superimposition ratio calculation unit 53 is not limited to the example setting shown in Fig. 8. Another example setting of the superimposition ratio r will be described below with reference to Fig. 11. Fig. 11 is a diagram showing another example setting of the superimposition ratio according to the position of the virtual viewpoint.

The superimposition ratio r1 is a setting example in which the real image Ia and the virtual image V, and the virtual image V and the real image Ib are switched instantaneously at a predetermined time. That is, the superimposition ratio _rca of the real image Ia is switched from 100% to 0% at the time ta. Then, the superimposition ratio _rV of the virtual image V is switched from 0% to 100% at the time ta. Also, the superimposition ratio _rcb of the real image Ib is switched from 0% to 100% at the time tb. Then, the superimposition ratio _rV of the virtual image V is switched from 100% to 0% at the time tb. Note that, although each waveform showing the superimposition ratio r1 shown in FIG. 11 is a rectangular wave in the strict sense, it is drawn with a steep slope in order to make it difficult to understand visually.

In this way, a cut (switching) effect is achieved by instantaneously switching between the two images. Note that the superimposition ratio r1 shown in FIG. 11 is also set to instantaneously switch whether or not the 3DCG object image O is superimposed on the superimposition image Ix1 for the 3DCG object image O.

The superimposition ratio r2 is an example in which the superimposition ratio _r3DCG of the 3DCG object image O is set to 100% (maximum value), and the superimposition ratio _rca of the real image Ia and the superimposition ratio _rV of the virtual image V are changed. That is, at time tc when the superimposition ratio _r3DCG of the 3DCG object image O is set to 100%, the superimposition ratio _rca of the real image Ia is gradually decreased from 100%. In addition, after the superimposition ratio _rcb of the real image Ib is gradually increased to 100% at time td, the superimposition ratio _r3DCG of the 3DCG object image O is gradually decreased from 100%.

By setting the overlap ratio r2 in this way, when the overlap ratio r2 of the real image Ia, the virtual image V, and the real image Ib changes, the switch from the real image Ia to the virtual image V and the switch from the virtual image V to the real image Ib can be made inconspicuous. This creates the effect of making the viewer of the composite image L recognize that the virtual image V and the real images Ia and Ib are in the same space.

The blending ratio r3 is an example setting in which the rising timing of the blending ratio _rV of the virtual image V is delayed compared to the falling timing of the blending ratio _rca of the real image Ia, and the falling timing of the blending ratio _rV is advanced compared to the rising timing of the blending ratio _rcb of the real image Ib.

Specifically, the superimposition ratio _rca of the real image Ia is gradually decreased from 100% to 0% from time te to time tf, and then the superimposition ratio _rV of the virtual image V is gradually increased from 0% to 100% at time tf. Then, the superimposition ratio _rV of the virtual image V is gradually decreased to 0% at time tg, and then the superimposition ratio _rcb of the real image Ib is gradually increased from 0% to 100% from time tg to time th. Note that the superimposition ratio _r3DCG of the 3DCG object image O starts to gradually increase from 0% at time te, and after the superimposition ratio _r3DCG is set to 100%, it is gradually decreased to 0% at time th.

By setting the overlap ratio r3 in this manner, a visual effect is obtained in which the subject 90 temporarily disappears and then the virtual image V emerges.

[1-8. Effects of the first embodiment]
As described above, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L by synthesizing real images Ia, Ib including a subject 90 (first object) captured by a real camera 70 (first imaging device), a virtual image V (first virtual image) generated based on a 3D model 90M corresponding to the subject 90 and a specific virtual viewpoint Vp (viewpoint information), and a 3DCG object image O (second virtual image) generated based on a 3D model 92 corresponding to a 3DCG object (second object) different from the subject 90 and the virtual viewpoint Vp, at superimposition ratios r, r _3DCG (synthesis ratios) according to the viewpoint information.

This allows seamless switching between real images Ia, Ib captured by real camera 70 and virtual image V.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L at a superimposition ratio r (synthesis ratio) corresponding to the distance between the virtual viewpoint Vp and the real camera 70 (first imaging device).

This makes it possible to set the superimposition ratio r (composite ratio) between the real images Ia, Ib captured by the real cameras 70a, 70b and the virtual image V through simple calculation processing.

Furthermore, in the decoding device 50 a (information processing device) of the first embodiment, when the distance between the real camera 70 (first imaging device) and the virtual viewpoint Vp is equal to or less than a predetermined value, the image synthesis unit 55 (image generation unit) synthesizes the virtual image V at a lower superimposition ratio r _V (synthesis ratio) compared to when the distance is greater than the predetermined value.

As a result, in the vicinity of the real cameras 70a and 70b, a higher quality composite image L can be obtained by increasing the contribution of the real images Ia and Ib to the composite image L.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the viewpoint setting unit 52 sets a virtual viewpoint Vp, which is viewpoint information, over time.

This allows free movement of the viewpoint by moving the virtual viewpoint Vp.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the viewpoint information is the position of a virtual viewpoint Vp and the observation direction at the virtual viewpoint Vp.

This allows the virtual viewpoint Vp to be moved while keeping the subject 90 in focus.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L at a superimposition ratio r (synthesis ratio) corresponding to the movement speed of the virtual viewpoint Vp.

This makes the transition between the real images Ia, Ib and the virtual image V less noticeable.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L using a real image Ia (first real image) captured by two different real cameras 70a (first imaging device) and a real image Ib (second real image) captured by real camera 70b (first imaging device), a virtual image V (first virtual image) generated by projecting a 3D model 90M of a subject 90 (first object) onto a virtual viewpoint Vp located between the two real cameras 70a and 70b, and a 3DCG object image O (second virtual image) generated from a 3DCG object (second object).

This allows seamless switching between real image Ia captured by real camera 70a, real image Ib captured by real camera 70b, and virtual image V.

In addition, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L by switching the superimposition ratio r (synthesis ratio) between a minimum value (0%) and a maximum value (100%) at a predetermined time.

This allows you to instantly switch between two images, creating a cut (switching) effect.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L by changing the superimposition ratio r (synthesis ratio) of the real images Ia, Ib and the virtual image V while the superimposition ratio r _3DCG (synthesis ratio) of the 3DCG object image O (CG object) is set to the maximum value (100%).

This makes it possible to make the transition from real image Ia to virtual image V, and from virtual image V to real image Ib less noticeable, thereby making the viewer of the composite image L recognize that virtual image V and real images Ia and Ib are in the same space.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L by delaying the timing of changing the superimposition ratio _rV of the virtual image V beyond the falling timing of the superimposition ratio _rca of the real image Ia, or advances the timing of changing the superimposition ratio _rcb of the real image Ib.

This creates a visual effect in which the subject 90 disappears and then the virtual image V appears.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the superimposition ratio r _3DCG (synthesis ratio) of the 3DCG object image O (CG object) is the transmittance when superimposing the real image Ia and virtual image V with the 3DCG object image O, or the real image Ib and virtual image V with the 3DCG object image O.

This allows the 3DCG object image O to be composited at any transparency, thereby expanding the range of expression of the composite image L.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the 3DCG object image O (CG object) includes a visual effect having 3D information.

This makes it possible to reduce the sense of discomfort felt when switching between real images Ia, Ib and virtual image V by adding visual effects.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the 3DCG object image O (CG object) includes a character.

This allows characters to be superimposed on the composite image L, thereby expanding the range of expression of the composite image L.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the observation direction is the direction from the virtual viewpoint Vp toward the subject 90 (object).

This makes it possible to switch between real images Ia, Ib and virtual image V while always capturing the subject 90.

Furthermore, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates a synthetic image L by synthesizing real images Ia, Ib including a subject 90 (first object) captured by a real camera 70 (first imaging device), a 3D model 90M corresponding to the subject 90, and a 3D model 92 corresponding to a 3DCG object (second object) different from the subject 90 at superimposition ratios r, r _3DCG (synthesis ratios) according to specific viewpoint information.

This allows seamless switching between real images Ia, Ib captured by real camera 70 and virtual image V.

2. Second embodiment
The information processing system 10a (see FIG. 1) described in the first embodiment is configured using only a real camera 70 installed, for example, at a live venue as an imaging device. Therefore, since the imaging position is limited, the images capturing the subject 90 (real images Ia, Ib, virtual image V) are uniform. The information processing system 10b (see FIG. 12) of the second embodiment adds a real camera equipped on a mobile terminal carried by an audience member to the configuration of the information processing system 10a, thereby expanding the variety of real images capturing the subject 90 and further enhancing the performance effect.

[2-1. Overview of information processing system]
First, an overview of an information processing system 10b will be described with reference to Fig. 12. Fig. 12 is a diagram illustrating an overview of an information processing system according to a second embodiment.

The information processing system 10b includes real cameras 70 (70a, 70b) installed at a live venue or the like to capture an image of the subject 90, as well as real cameras 76 (76a, 76b, ...) built into multiple mobile terminals 74 (74a, 74b, ...) carried by spectators at the live venue. The real camera 76a captures a real image Ic including the subject 90. The real camera 76b captures a real image Id including the subject 90. The real camera 76 is a real camera different from the real camera 70 (first imaging device) described above. That is, the real images Ia and Ib captured by the real camera 70 are used to generate a 3D model 90M of the subject 90, whereas the real images Ic and Id captured by the real camera 76 are not used to generate a 3D model 90M of the subject 90. The real cameras 76 (76a, 76b, ...) are an example of a second imaging device in this disclosure.

Then, the viewpoint setting unit 52 (viewpoint setting unit) of the information processing system 10b acquires the current position of the real camera 76 and sets the virtual viewpoint Vp based on the position of the real camera 70. As a result, the viewpoint setting unit 52 sets, for example, a movement path for moving the virtual viewpoint Vp to the current position of the real camera 76.

Then, the information processing system 10b gradually moves the virtual viewpoint Vp away from a position near the real camera 70 to the position of the real camera 76 (76a, 76b, ...) built into the mobile terminal 74 (74a, 74b, ...) held by the spectator.

At this time, while moving the virtual viewpoint Vp, the information processing system 10b generates a composite image L of the virtual image V (two-dimensional image) generated from the volumetric video and the real image captured by the real camera 76 at the position of the moving virtual viewpoint Vp. Then, the information processing system 10b displays the generated composite image L on the mobile terminal 74 held by the spectator.

Note that the real cameras 76 (76a, 76b, ...) built into the mobile terminals 74 (74a, 74b, ...) carried by the spectators are not used to generate the 3D model 90M of the subject 90, and therefore the 3D model 90M of the subject 90 generated by the real camera 70 is used.

This allows, for example, a spectator carrying a mobile terminal 74a, 74b to experience the image of the subject 90 captured by the real camera 70 switching over time to the real images Ic, Id captured by the real cameras 76a, 76b on their own mobile terminals 74a, 74b.

Note that the real cameras 76 (76a, 76b, ...) are not limited to cameras built into the mobile terminal 74. In other words, the real cameras 76 (76a, 76b, ...) may be fixed cameras or movable cameras installed at a distance from the subject 90 at the live venue.

The movement path of the virtual viewpoint Vp can be set arbitrarily, but in the example shown in FIG. 12, the virtual viewpoint Vp is moved from the position of the real camera 70a at time t0 to the position of the real camera 70b at time t6, and then moved to the position of the real camera 76a at time t10 (movement path 88b). Also, the virtual viewpoint Vp is moved from the position of the real camera 70a at time t0 to the position of the virtual camera 72c at time t3, and then moved to the position of the real camera 76b at time t14 (movement path 88c). In this way, the information processing system 10b sets the movement path of the virtual viewpoint Vp for the multiple mobile terminals 74 (74a, 74b, ...) based on the positions of the mobile terminals 74 (74a, 74b, ...).

[2-2. Functional configuration of information processing system]
The information processing system 10b includes the encoding device 40a described in the first embodiment and a decoding device 50b (not shown) replacing the decoding device 50a. The information processing system 10b further includes the mobile terminal 74 (74a, 74b, ...) described above. The decoding device 50b is an example of an information processing device in the present disclosure.

The decoding device 50b differs from the decoding device 50a in that it has a function of receiving an image captured by a real camera 76 built into the mobile terminal 74, a function of receiving the current position of the mobile terminal 74, and a function of transmitting a composite image L to the mobile terminal 74. Accordingly, the decoding device 50b has a rendering unit 17b (not shown) instead of the rendering unit 17a. The rendering unit 17b has a function of receiving an image captured by the real camera 76, a function of receiving the current position of the mobile terminal 74, and a function of transmitting a composite image L to the mobile terminal 74, in comparison with the rendering unit 17a. Note that the functional block configuration of the rendering unit 17b is the same as the functional block configuration of the rendering unit 17a, so it is not shown in the figure, and the same reference numerals as those in FIG. 5 are used to describe the functional blocks. That is, the image synthesis unit 55 and the virtual image generation unit 51 receive the image captured by the real camera 76. In addition, the superimposition ratio calculation unit 53 receives the current position of the mobile terminal 74. In addition, the image synthesis unit 55 transmits the synthetic image L to the mobile terminal 74 .

The mobile terminal 74 has the functional configuration shown in Figure 13. Figure 13 is a functional block diagram showing an example of the functional configuration of a mobile terminal.

The mobile terminal 74 (74a, 74b, ...) includes a computer-configured operation control unit 75. The operation control unit 75 includes an imaging control unit 80, a positioning unit 81, an application control unit 82, a display control unit 83, an operation control unit 84, and a communication control unit 85.

The imaging control unit 80 controls the imaging operation of the real camera 76 (76a, 76b, ...) built into the mobile terminal 74 (74a, 74b, ...).

The positioning unit 81 detects the current position of the mobile terminal 74 (74a, 74b, ...) by, for example, a GPS receiver 77 built into the mobile terminal 74 (74a, 74b, ...). The positioning unit 81 may detect the current position of the mobile terminal 74 (74a, 74b, ...) by a function other than the GPS receiver 77, for example, Wi-Fi (registered product) positioning or Bluetooth (registered product) positioning. The positioning unit 81 may also detect the position and orientation of the mobile terminal 74 (74a, 74b, ...) based on images of markers installed in the venue captured by the real cameras 76 (76a, 76b, ...).

The application control unit 82 controls the operation of the video display application running on the mobile terminal 74 (74a, 74b, ...).

The display control unit 83 controls the display state of the display 78 provided on the mobile terminal 74 (74a, 74b, ...).

The operation control unit 84 receives an operation signal from an operation device such as a touch panel 79 provided on the mobile terminal 74 (74a, 74b, ...) and recognizes the operation content.

The communication control unit 85 controls the communication state between the mobile terminal 74 (74a, 74b, ...) and the decoding device 50b.

[2-3. Operation of the second embodiment]
Next, the operation of the information processing system 10b of the second embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of setting the blending ratio according to the position of the virtual viewpoint in the second embodiment. Note that, for the sake of simplicity, Fig. 14 shows only the blending ratio between the real images Ia, Ib, Ic, and Id and the virtual image V, and omits the blending ratio of the 3DCG object image O.

When the viewpoint setting unit 52 sets the virtual viewpoint Vp to move along the movement path 88b, the superimposition ratio calculation unit 53 sets the superimposition ratio r4 shown in Figure 14.

The superimposition ratio calculation unit 53 gradually decreases the superimposition ratio _rca of the real image Ia, which is set to 100%, from time t1 (the position of the virtual camera 72a) to 0% before time t3 (the position of the virtual camera 72c), and maintains _rca = 0% until time t10 (the position of the real camera 76a).

The superimposition ratio calculation unit 53 also sets the superimposition ratio _rV1 of the virtual image V, which was set to 0%, to gradually increase from near time t1 (the position of the virtual camera 72a) and to 100% before time t3 (the position of the virtual camera 72c).Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV1 of the virtual image V to gradually decrease to 0% from time t4 (the position of the virtual camera 72d) to time t5 (the position of the virtual camera 72e).The superimposition ratio _rV1 is the superimposition ratio of the virtual image V generated based on the real image Ia captured by the real camera 70a and the real image Ib captured by the real camera 70b.

The superimposition ratio calculation unit 53 gradually increases the superimposition ratio r _cb of the real image Ib, which was set to 0%, from around time t4 (the position of the virtual camera 72d), and sets it to 100% at time t5 (the position of the virtual camera 72e).Then, the superimposition ratio calculation unit 53 gradually decreases the superimposition ratio r _cb of the real image Ib from time t7 (the position of the virtual camera 72f), and sets it to 0% before time t8 (the position of the virtual camera 72g).

Furthermore, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV2 of the virtual image V, which was set to 0%, to gradually increase from around time t7 (the position of the virtual camera 72f) and to 100% before time t8 (the position of the virtual camera 72g).Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV2 of the virtual image V to gradually decrease to 0% from time t9 (the position of the virtual camera 72h) to time t10 (the position of the real camera 76a).The superimposition ratio _rV2 is the superimposition ratio of the virtual image V generated based on the real image Ib captured by the real camera 70b and the real image Ic captured by the real camera 76a.

Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rcc of the real image Ic, which was set to 0%, so that it gradually increases from around time t9 (the position of the virtual camera 72h) and reaches 100% at time t10 (the position of the real camera 76a).

Note that the superimposition ratio _rV2 is gradually decreased (changed) in the vicinity of the real camera 76a, for example, from time t9 to time t10. This is because the real image Ic captured by the real camera 76a, i.e., the second imaging device, has lower calibration and synchronization accuracy than the real images Ia and Ib captured by the real cameras 70 (70a, 70b), i.e., the first imaging device. That is, when switching from the virtual image V to the real image Ic, there is a possibility that a positional shift of the subject 90 may occur, and this is to make this positional shift less noticeable.

In addition, when the viewpoint setting unit 52 sets the virtual viewpoint Vp to move along the movement path 88c, the superimposition ratio calculation unit 53 sets the superimposition ratio r5 shown in Figure 14.

The superimposition ratio calculation unit 53 gradually decreases the superimposition ratio _rca of the real image Ia, which is set to 100%, from time t1 (the position of the virtual camera 72a) to 0% before time t3 (the position of the virtual camera 72c), and maintains _rca = 0% until time t10 (the position of the real camera 76a).

The superimposition ratio calculation unit 53 also sets the superimposition ratio _rV1 of the virtual image V, which was set to 0%, to be gradually increased from near time t1 (the position of the virtual camera 72a) to be 100% between time t2 (the position of the virtual camera 72b) and time t3 (the position of the virtual camera 72c).Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV1 of the virtual image V to be gradually decreased to 0% between time t3 (the position of the virtual camera 72c) and time t11 (the position of the virtual camera 72i).The superimposition ratio _rV1 is the superimposition ratio of the virtual image V generated based on the real image Ia captured by the real camera 70a and the real image Ib captured by the real camera 70b.

Furthermore, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV2 of the virtual image V, which was set to 0%, to gradually increase from around time t3 (the position of the virtual camera 72c) and to 100% before time t11 (the position of the virtual camera 72i).Then, the superimposition ratio calculation unit 53 sets the superimposition ratio _rV2 of the virtual image V to gradually decrease to 0% from time t13 (the position of the virtual camera 72k) to time t14 (the position of the real camera 76b).The superimposition ratio _rV2 is the superimposition ratio of the virtual image V generated based on the virtual image V generated at time t3 (the position of the virtual camera 72c) and the real image Id captured by the real camera 76b.

Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r _cd of the real image Id, which was set to 0%, to gradually increase from around time t13 (the position of the virtual camera 72k) and reach 100% at time t14 (the position of the real camera 76b).

Note that the superimposition ratio _rV2 is gradually decreased (changed) in the vicinity of the real camera 76b, for example, from time t13 to time t14. This is because the real image Id captured by the real camera 76b, i.e., the second imaging device, has lower calibration and synchronization accuracy than the real images Ia and Ib captured by the real cameras 70 (70a, 70b), i.e., the first imaging device. That is, when switching from the virtual image V to the real image Id, there is a possibility that a positional shift of the subject 90 may occur, and this is to make this positional shift less noticeable.

The superimposition ratio calculation unit 53 may set the superimposition ratios _rcc , _rcd (composite ratios) of the real images Ic, Id according to the distance between the virtual viewpoint Vp and the real cameras 76 (76a, 76b). For example, the superimposition ratios _rcc , _rcd of the real images Ic, Id captured by the real cameras 76 (76a, 76b) when the virtual viewpoint Vp is closer to the real cameras 76 (76a, 76b) than a predetermined distance may be set higher than the superimposition ratios rcc, _rcd of the real images Ic, Id captured by the real cameras 76 when the virtual viewpoint Vp is farther away from the real cameras ₇₆ than a predetermined distance.

In addition, the superimposition ratio calculation unit 53 may simultaneously control not only the superimposition ratio r of each image, but also the color intensity of each image, the degree of blur of each image, etc.

[2-4. Processing flow of the second embodiment]
Next, the flow of processing performed by the information processing system 10b will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of processing performed by the information processing system of the second embodiment. Note that, as an example, the operation in a state where the movement route 88b shown in Fig. 12 is set will be described.

First, the flow of processing performed by the rendering unit 17b (not shown) will be described. After the information processing system 10b launches an application that operates the second embodiment, the rendering unit 17b determines whether it has received location information of the mobile terminal 74a (step S31). If it is determined that the location information of the mobile terminal 74a has been received (step S31: Yes), the process proceeds to step S32. On the other hand, if it is not determined that the location information of the mobile terminal 74a has been received (step S31: No), the process repeats the determination of step S31.

If the answer is Yes in step S31, the viewpoint setting unit 52 sets a virtual viewpoint Vp for each time (step S32).

The image synthesis unit 55 acquires the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, and the real image Ic captured by the mobile terminal 74a (real camera 76a) (step S33).

The virtual image generating unit 51 generates a virtual image V based on the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generating unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52. The virtual image generating unit 51 also generates a virtual image V based on the real image Ib captured by the real camera 70b, the real image Ib captured by the real camera 70b, the real image Ic captured by the mobile terminal 74a (real camera 76a), the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generating unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52. (Step S34).

The superimposing ratio calculation unit 53 sets the superimposing ratio r ( _rca , _rcb , _rV1 , _rV2 ) (step S35).

The image synthesis unit 55 generates a synthetic image L (step S36).

The image synthesis unit 55 transmits the synthetic image L and the position of the virtual viewpoint Vp to the mobile terminal 74a (step S37).

The rendering unit 17b determines whether the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38). If it is determined that the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38: Yes), the rendering unit 17b ends the processing of Fig. 15. On the other hand, if it is not determined that the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38: No), the processing returns to step S33.

Next, the flow of processing performed by the mobile terminal 74a will be described. The application control unit 82 determines whether an application that operates the second embodiment has been started based on the operation signal detected by the operation control unit 84 (step S41). If it is determined that the application has been started (step S41: Yes), the process proceeds to step S42. On the other hand, if it is not determined that the application has been started (step S41: No), the determination of step S41 is repeated.

If the answer is Yes in step S41, the communication control unit 85 transmits the current position of the mobile terminal 74a calculated by the positioning unit 81 to the rendering unit 17b (step S42).

The communication control unit 85 transmits the actual image Ic captured by the real camera 76a by the imaging control unit 80 to the rendering unit 17b (step S43).

The communication control unit 85 receives the composite image L and the position of the virtual viewpoint Vp from the rendering unit 17b (step S44).

The display control unit 83 displays the composite image L on the display 78 (step S45).

The application control unit 82 determines whether the virtual viewpoint Vp and the position of the mobile terminal 74a match (step S46). If it is determined that the virtual viewpoint Vp and the position of the mobile terminal 74a match (step S46: Yes), the rendering unit 17b ends the processing of FIG. 15. On the other hand, if it is not determined that the virtual viewpoint Vp and the position of the mobile terminal 74a match (step S46: No), the process returns to step S43.

In addition, in Figure 15, it is described that an application is launched on the mobile terminal 74a side, but an image may also be transmitted from the decoding device 50b side to the customer's mobile terminal 74 as a part of the production of a certain scene.

[2-5. Effects of the second embodiment]
As described above, in the decoding device 50b (information processing device) of the second embodiment, the viewpoint setting unit 52 sets the position of the virtual viewpoint Vp based on the position of the real camera 76 (76a, 76b) (second imaging device) that is different from the multiple real cameras 70 (first imaging device) used to generate the 3D model 90M of the subject 90 (object).

This allows real images Ic, Id captured by real cameras 76 (76a, 76b) (second imaging device) that are not used to generate the 3D model 90M of the subject 90 to be used as material for the composite image L, thereby expanding the range of expression of the composite image L.

In addition, in the decoding device 50b (information processing device) of the second embodiment, the image synthesis unit 55 (image generation unit) sets the superimposition ratios rcc, _rcd (synthesis ratios) of the real images Ic, Id captured by the real camera 76 (76a, 76b) (second imaging device) when the virtual viewpoint Vp is closer to the real camera 76 than a predetermined distance, to be higher than the superimposition ratios rcc, _rcd of the real images Ic, Id captured by the real camera 76 when the virtual viewpoint _Vp is farther away from the real camera 76 than a predetermined _distance .

As a result, as the virtual viewpoint Vp approaches the real cameras 76 (76a, 76b), the superimposition ratios _rcc , rcd of the real images Ic, _Id are set higher, making it possible to make the switch from the virtual image V to the real images Ic, Id less noticeable.

In addition, in the decoding device 50b (information processing device) of the second embodiment, the image synthesis unit 55 (image generation unit) changes the superimposition ratios rcc, rcd (synthesis ratios) of the real images Ic, Id captured by the real camera 76 (76a, 76b) (second imaging device) when the position of the virtual viewpoint Vp approaches the real camera 76 (76a, 76b) more gradually than the superimposition ratios _rca , _rcb (synthesis ratios) of the real images Ia, Ib captured by the real camera 70 (70a, 70b) when the position of the virtual viewpoint Vp approaches the real camera 70 ( _70a , _70b ) (first imaging device).

This makes it possible to make the switch from the virtual image V to the real images Ic, Id less noticeable when using real images Ic, Id captured by a real camera 76 (76a, 76b) with poor calibration or synchronization accuracy.

In addition, in the decoding device 50b (information processing device) of the second embodiment, the second imaging device is provided in a mobile terminal 74 (74a, 74b).

This allows the real images Ic, Id captured by the real camera 76 (76a, 76b) (second imaging device) equipped on the mobile terminal 74 (74a, 74b) to be transmitted to the decoding device 50b (information processing device), and allows a customer carrying the mobile terminal 74 (74a, 74b) to view the composite image L transmitted from the decoding device 50b (information processing device). In other words, a two-way service can be realized between the decoding device 50b (information processing device) and the mobile terminal 74 (74a, 74b).

(3. Application Examples of the Present Disclosure)
[3-1. Content Creation]
For example, the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 may be synthesized with a 3D model managed by another server to produce video content. In addition, for example, when background data acquired by an imaging device such as Lidar exists, the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 may be combined with the background data to produce content in which the subject 90 appears as if it were in a location indicated by the background data. Note that the video content may be three-dimensional video content or two-dimensional video content converted into two dimensions. The 3D model 90M of the subject 90 generated in this embodiment is, for example, a 3D model generated by the 3D model generation unit 12 or a 3D model reconstructed by the rendering units 17a and 17b.

[3-2. Experience in Virtual Space]
For example, the subject 90 generated in this embodiment can be placed in a virtual space where a user becomes an avatar and communicates with the user. In this case, the user becomes an avatar and can view a real-life image of the subject 90 in the virtual space.

[3-3. Application to remote communication]
For example, by transmitting the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 from the transmission unit 14 to a remote location, a user at the remote location can view the 3D model 90M of the subject 90 through a playback device at the remote location. For example, by transmitting the 3D model 90M of the subject 90 in real time, the subject 90 and a user at a remote location can communicate in real time. For example, a case can be envisioned in which the subject 90 is a teacher and the user is a student, or a case can be envisioned in which the subject 90 is a doctor and the user is a patient.

[3-4. Other]
For example, a free viewpoint video of sports or the like can be generated based on the 3D models 90M of multiple subjects 90 generated by the 3D model generation unit 12. Also, an individual can distribute the image of himself/herself generated by the 3D model generation unit 12 to a distribution platform. In this way, the contents of the embodiments described in this specification can be applied to various technologies and services.

Although the present disclosure has been described above using several embodiments, these embodiments may be executed in any device. In that case, it is sufficient that the device has the necessary functional blocks and is capable of obtaining the necessary information.

Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, if a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step may be executed as multiple step processes. Conversely, processes described as multiple steps may be executed collectively as a single step.

Also, for example, a program executed by a computer may be configured so that the processing of the steps describing the program is executed chronologically in the order described in this specification, or may be executed in parallel, or individually at the required timing, such as when a call is made. In other words, as long as no contradiction arises, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps describing the program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

In addition, for example, the multiple technologies related to the present technology can be implemented independently and individually, as long as no contradiction occurs. Of course, any multiple technologies of the present technology can also be applied and implemented. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Also, part or all of any of the above-mentioned technologies of the present technology can be implemented in combination with other technologies not described above.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may exist. Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

For example, the present disclosure may be configured as follows:

(1)
An information processing device comprising: an image generation unit that generates a composite image by combining a real image including a first object captured by a first imaging device, a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information, and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, at a combination ratio according to the viewpoint information.
(2)
the image generation unit generates the composite image at a composite ratio according to a distance between a virtual viewpoint based on the viewpoint information and the first imaging device.
The information processing device according to (1).
(3)
When a distance between the first imaging device and a virtual viewpoint based on the viewpoint information is equal to or smaller than a predetermined value, the image generation unit synthesizes the first virtual image at a lower synthesis ratio than when the distance is greater than the predetermined value.
The information processing device according to (1) or (2).
(4)
A viewpoint setting unit that sets a virtual viewpoint, which is the viewpoint information, over time.
The information processing device according to any one of (1) to (3).
(5)
The viewpoint information is a position of a virtual viewpoint and an observation direction at the virtual viewpoint.
The information processing device according to any one of (1) to (4).
(6)
The image generation unit generates the composite image at a composite ratio according to a moving speed of the virtual viewpoint.
The information processing device according to (4) or (5).
(7)
the image generation unit generates a composite image using a first real image and a second real image captured by two different first imaging devices, respectively, a first virtual image generated by projecting a 3D model of the first object onto a virtual viewpoint located between the two first imaging devices, and a second virtual image generated from the second object.
The information processing device according to any one of (2) to (6).
(8)
the image generating unit generates the composite image by switching the composite ratio between a minimum value and a maximum value at a predetermined time.
The information processing device according to (6) or (7).
(9)
the image generation unit changes a blending ratio of the real image and the first virtual image to generate the blended image while a blending ratio of the second object is set to a maximum value;
The information processing device according to any one of (6) to (8).
(10)
the image generation unit generates the composite image by delaying a change timing of the combination ratio of the first virtual image to a timing later than a fall timing of the combination ratio of the real image or advances the change timing of the combination ratio of the real image to a timing earlier than a rise timing of the combination ratio of the real image.
The information processing device according to any one of (6) to (9).
(11)
a blending ratio of the second object is a transmittance when the real image and the first virtual image are superimposed on the second object;
The information processing device according to any one of (1) to (10).
(12)
the second object includes a visual effect;
The information processing device according to any one of (1) to (11).
(13)
The second object includes a character.
The information processing device according to any one of (1) to (11).
(14)
the observation direction is a direction from the virtual viewpoint toward the first object;
The information processing device according to any one of (5) to (13).
(15)
The viewpoint setting unit is
setting the virtual viewpoint based on a position of a second image capture device different from the plurality of first image capture devices used to generate the 3D model of the first object;
The information processing device according to any one of (5) to (14).
(16)
the image generation unit sets a synthesis ratio of the real image captured by the second imaging device when the position of the virtual viewpoint is closer than a predetermined distance to the second imaging device, higher than a synthesis ratio of the real image captured by the second imaging device when the position of the virtual viewpoint is farther than the predetermined distance.
The information processing device according to (15) above.
(17)
the image generation unit changes a synthesis ratio of the real image captured by the second imaging device more gradually when the position of the virtual viewpoint approaches the second imaging device than a synthesis ratio of the real image captured by the first imaging device when the position of the virtual viewpoint approaches the first imaging device;
The information processing device according to (16).
(18)
The second imaging device is provided in a mobile terminal.
The information processing device according to any one of (15) to (17).
(19)
An information processing device comprising: an image generation unit that generates a composite image by combining an actual image including a first object captured by a first imaging device, a 3D model corresponding to the first object, and a 3D model corresponding to a second object different from the first object, at a combination ratio according to specific viewpoint information.
(20)
A method for generating image data, comprising: generating a composite image by combining a first virtual image generated based on a real image including a first object captured by a first imaging device, a 3D model corresponding to the first object, and specific viewpoint information; and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, at a combination ratio according to the viewpoint information.

10a...information processing system, 11...data acquisition unit, 12...3D model generation unit, 13a...encoding unit, 14...transmission unit, 15...reception unit, 16a...decoding unit, 17a, 17b...rendering unit, 18...display unit, 40a...encoding device (information processing device), 50a, 50b...decoding device (information processing device), 51...virtual image generation unit, 52...viewpoint setting unit, 53...superimposition ratio calculation unit, 54...3DCG object image generation unit, 55...image synthesis unit (image generation unit), 56...3D object superimposition unit, 57...signal selection unit, 70, 70a, 70b...real camera (first imaging device), 72a, 72b, 72c, 72d, 72e, 72f, 72g, 72h, 72i, 72k...virtual camera, 74, 74 a, 74b...mobile terminal, 75...operation control unit, 76, 76a, 76b...real camera (second imaging device), 80...imaging control unit, 81...positioning unit, 82...application control unit, 83...display control unit, 84...operation control unit, 85...communication control unit, 88a, 88b, 88c...movement path, 90...subject (first object), 90M, 92...3D model, A1, A2...adder, Ia...real image (first real image), Ib...real image (second real image), Ic, Id...real image, Ix1, Ix2...superimposed image, L...synthetic image, M1, M2, M3, M4, M5...multiplier, Mo...3DCG object presence map, O...3DCG object image (second virtual image), r, r1, r2, r3, r _ca , _rcb , rcc, _rcd , _rV , _rV1 , _{rV2 , r3DCG} ...superimposition ratio (composite ratio), V...virtual image (first virtual image), Vp...virtual viewpoint (viewpoint information)

Claims (20)

a real image including a first object captured by a first imaging device;
a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information; and
a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information; and
an image generating unit that generates a composite image by combining the above-mentioned images at a combination ratio according to the viewpoint information,
Information processing device. The image generating unit includes:
generating the composite image at a composite ratio according to a distance between a virtual viewpoint based on the viewpoint information and the first imaging device;
The information processing device according to claim 1 . The image generating unit includes:
When the distance between the first imaging device and a virtual viewpoint based on the viewpoint information is equal to or less than a predetermined value,
The first virtual image is synthesized at a lower synthesis ratio compared to when the distance is greater than a predetermined value.
The information processing device according to claim 1 . A viewpoint setting unit that sets a virtual viewpoint, which is the viewpoint information, over time.
The information processing device according to claim 1 . The viewpoint information is a position of a virtual viewpoint and an observation direction at the virtual viewpoint.
The information processing device according to claim 4. The image generating unit includes:
generating the composite image at a composite ratio according to the moving speed of the virtual viewpoint;
The information processing device according to claim 5 . The image generating unit includes:
A first actual image and a second actual image captured by two different first imaging devices, respectively;
a first virtual image generated by projecting a 3D model of the first object onto a virtual viewpoint located between the two first imaging devices; and
a second virtual image generated from the second object; and
Generate a synthetic image using
The information processing device according to claim 2 . The image generating unit includes:
generating the composite image by switching the composite ratio between a minimum value and a maximum value at a predetermined time;
The information processing device according to claim 6. The image generating unit includes:
generating the composite image by changing a blending ratio of the real image and the first virtual image while a blending ratio of the second object is set to a maximum value;
The information processing device according to claim 6. The image generating unit includes:
generating the composite image by delaying a change timing of the blending ratio of the first virtual image from a falling timing of the blending ratio of the real image or from a rising timing of the blending ratio of the real image;
The information processing device according to claim 6. The blending ratio of the second object is
is a transmittance when the real image, the first virtual image, and the second object are superimposed.
The information processing device according to claim 1 . the second object includes a visual effect;
The information processing device according to claim 1 . The second object includes a character.
The information processing device according to claim 1 . the observation direction is a direction from the virtual viewpoint toward the first object;
The information processing device according to claim 5 . The viewpoint setting unit is
setting the virtual viewpoint based on a position of a second image capture device different from the plurality of first image capture devices used to generate the 3D model of the first object;
The information processing device according to claim 5 . The image generating unit includes:
a synthesis ratio of the real image captured by the second imaging device when the virtual viewpoint is closer to the second imaging device than a predetermined distance is set higher than a synthesis ratio of the real image captured by the second imaging device when the virtual viewpoint is farther away from the second imaging device than a predetermined distance;
The information processing device according to claim 15. The image generating unit includes:
when the position of the virtual viewpoint approaches the second imaging device, a synthesis ratio of the real image captured by the second imaging device is changed more gradually than a synthesis ratio of the real image captured by the first imaging device when the position of the virtual viewpoint approaches the first imaging device.
The information processing device according to claim 16. The second imaging device is provided in a mobile terminal.
The information processing device according to claim 15. an image generating unit that generates a composite image by combining an actual image including a first object captured by a first imaging device, a 3D model corresponding to the first object, and a 3D model corresponding to a second object different from the first object, at a combination ratio according to specific viewpoint information;
Information processing device. a real image including a first object captured by a first imaging device;
a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information; and
a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information; and
and generating a composite image by combining the above-mentioned images at a combination ratio according to the viewpoint information.
A method for generating image data.

Images