JP7701843B2 - 3D image generating device and program thereof - Google Patents

Description

The present invention relates to a stereoscopic image generating device and a program for generating stereoscopic images as interactive content.

Interactive content using video is used in a variety of fields, including entertainment and education. A typical example of interactive content that is now widely used is video games. Video games are extremely popular because they can be enjoyed easily by either one person or multiple people. In recent years, the spread of the Internet has made it possible to enjoy video games together with people in far-flung locations from the comfort of their own homes. In addition to video games, interactive content using video is used in a variety of situations in everyday life.

Interactive content is content that reflects user operations, that is, content that has two-way interaction between the user and the creator. Examples of interactive content include video games, digital guide signs, and interactive television.

Three-dimensional images are a type of video media that has been attracting attention in recent years. Unlike typical two-dimensional images, three-dimensional images have binocular parallax and motion parallax, and therefore give a stronger sense of realism and presence. In particular, realism and presence are very important elements in interactive content such as video games. In light of the above, there is a demand for devices that allow users to experience interactive content using three-dimensional images.

Various stereoscopic methods for displaying three-dimensional images have been proposed. Since interactive content such as video games is often viewed continuously for long periods of time, it is preferable to apply a stereoscopic method that causes less eye fatigue. Therefore, a spatial image reproduction method capable of displaying natural three-dimensional images has been proposed (Non-Patent Documents 1 and 2). This spatial image reproduction method is capable of reproducing light rays that are almost identical to the light rays emitted by the real thing, and is said to be less likely to cause eye fatigue due to a mismatch between eye convergence and accommodation (Non-Patent Document 3).

G. Lippmann, “Epreuves, reversibles donnant la sensation du relief", J. Phys., 7, 1, pp.821-825 (1908) F. Okano, et al. “Real-time pickup method for a three-dimensional image based on integral photography”, Applied optics, Vol. 36, No. 7, pp. 1598-1603 (1997) H. Hiura, et al. “Measurement of static convergence and accommodation responses to images of integral photography and binocular stereoscopy”, Optics express, Vol. 25, No. 4, pp. 3454-3468 (2017)

In the interactive content described above, it is necessary to realize the camerawork of a virtual camera that shoots three-dimensional images in a virtual space in order to produce visual effects. However, no prior art has been proposed that allows for easy realization of virtual camerawork in interactive content.

Therefore, the objective of the present invention is to provide a stereoscopic image generating device and a program for the device that can easily realize the camerawork of a virtual camera.

To solve the above problems, the stereoscopic image generating device according to the present invention is a stereoscopic image generating device that generates a stereoscopic image using a spatial image reproduction method as interactive content that reflects user operations, and is configured to include an operation signal conversion unit, a multi-viewpoint image generation unit, and a stereoscopic image conversion unit.

According to this configuration, the operation signal conversion unit receives an operation signal from a user and converts the input operation signal into the position, orientation, and size of a spatial image reproduction type virtual display that virtually displays a three-dimensional CG scene based on a predetermined conversion rule.

The multi-viewpoint image generation unit calculates the camera parameters of a virtual camera array in which virtual cameras are arranged using a rigid transformation matrix in accordance with the position, orientation, and size of the virtual display, and generates a multi-viewpoint image by photographing the virtual display displaying a three-dimensional CG scene with the virtual camera array.
The stereoscopic image conversion unit converts the multi-viewpoint image generated by the multi-viewpoint image generation unit into a stereoscopic image.

The camerawork of the virtual camera is considered to be equivalent to changes in the position, orientation, and size of the virtual display. Therefore, the stereoscopic image generating device reflects the user's operations in the position, orientation, and size of the virtual display, and captures this virtual display with a virtual camera array. In this way, the stereoscopic image generating device can easily reflect the user's operations in the camerawork of the virtual camera.

The present invention can also be realized as a program for causing a computer to function as the stereoscopic image generating device described above.

The present invention provides a stereoscopic image generating device and a program for the device that can easily realize virtual camera work.

1 is a schematic diagram showing a first system configuration in an interactive content experience system. FIG. 1 is an explanatory diagram illustrating display of a stereoscopic image for a plurality of users. FIG. 11 is a schematic diagram showing a second system configuration in an interactive content experience system. FIG. 11 is a schematic diagram showing a third system configuration in an interactive content experience system. FIG. 11 is a schematic diagram showing a fourth system configuration in an interactive content experience system. 1 is a block diagram showing a configuration of a stereoscopic image generating device according to a first embodiment. FIG. 2 is an explanatory diagram for explaining a virtual camera array in the first embodiment. 4A to 4C are explanatory diagrams illustrating camerawork of a virtual camera in the first embodiment. FIG. 4 is an explanatory diagram for explaining transformation of a coordinate system in the first embodiment. FIG. 4 is an explanatory diagram illustrating a change in size of a virtual display in the first embodiment. FIG. 4 is an explanatory diagram illustrating a change in size of a virtual display in the first embodiment. FIG. 4 is an explanatory diagram for explaining generation of an element image in the first embodiment. 4 is a flowchart showing the operation of the stereoscopic image generating device according to the first embodiment. FIG. 11 is a block diagram showing the configuration of a stereoscopic image generating device according to a second embodiment. FIG. 13 is a block diagram showing the configuration of a stereoscopic image generating device according to a third embodiment. 13 is an explanatory diagram illustrating generation of a multi-viewpoint image according to a viewpoint position in the third embodiment. FIG. FIG. 13 is a block diagram showing the configuration of a stereoscopic image generating device according to a fourth embodiment.

Each embodiment of the present invention will be described below with reference to the drawings. However, each embodiment described below is intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[System configuration of the content experience system]
Before describing each embodiment in detail, the overall configuration of the interactive content experience system will be described. Hereinafter, the "interactive content experience system" will be abbreviated to simply "content experience system." As described below, the content experience system has various system configurations depending on the number of users and the presence or absence of a viewpoint tracking function.

<First Configuration: Single user, no viewpoint tracking function>
With reference to FIG. 1, a system configuration in which there is only one user U and the system does not have a gaze point tracking function will be described.
The content experience system 1 is a system in which a user U experiences interactive content such as a game. The content experience system 1 in Fig. 1 is the most basic system configuration among the various system configurations described below. As shown in Fig. 1, the content experience system 1 includes a controller 2, a stereoscopic image generating device 3, and a stereoscopic display 4. Here, the controller 2, the stereoscopic image generating device 3, and the stereoscopic display 4 are connected to each other via wired cables or wireless communication.

The controller 2 is used by the user U to operate the interactive content. If the interactive content is a video game, the user U uses the controller 2 to operate a player character. For example, the controller 2 is a game pad equipped with a stick and buttons. The controller 2 also outputs operation signals input by the user to the stereoscopic image generating device 3.

The stereoscopic image generating device 3 generates a stereoscopic image by a spatial image reproduction method as interactive content (e.g., a video game) that reflects the operation of the user U. In this embodiment, the stereoscopic image generating device 3 generates a stereoscopic image (a group of elemental images) by an integral method (Non-Patent Documents 1 and 2, Reference 1). The stereoscopic image generating device 3 then outputs the generated stereoscopic image to the stereoscopic display 4.

Reference 1: Misaka, "Outline of the integral system," Japan Broadcasting Corporation, NHK Research and Development R&D/No. 144, March 2014

The three-dimensional display 4 is a spatial image reproduction type display that displays the three-dimensional image input from the three-dimensional image generating device 3. In this embodiment, the three-dimensional display 4 is equipped with a lens array and a display element (not shown), similar to a general integral type display. For example, the lens array is an array in which element lenses corresponding to element images are arranged in a two-dimensional direction. For example, the display element can be a general flat panel display such as a liquid crystal display, an organic EL (Electro Luminescence) display, or a plasma display.

<Second Configuration: Multiple users, no viewpoint tracking function>
A system configuration in which there are multiple users U and the system does not have a gaze point tracking function will be described with reference to FIG. 2 and FIG.
A spatial image reproduction type display allows stereoscopic viewing not only in the main lobe, which is a viewing zone formed in front of the display, but also in the side lobes formed on both sides of the main lobe. As shown in FIG. 2, the content experience system 1 can display a common three-dimensional CG scene M for two users U (U ₁ , U ₂ ) using the side lobes, which is suitable for a fighting game, for example. When there are multiple users U, as shown in FIG. 3, the content experience system 1 includes a controller 2 (2 ₁ , ..., 2 _N ) for each user U, and only one stereoscopic display 4 common to all users U. Here, N is an integer of 1 or more indicating the number of users U. In other respects, the content experience system 1 of the second configuration is similar to the first configuration, so further description will be omitted.

<Third Configuration: Single user, with viewpoint tracking function>
A system configuration in which there is only one user U and the system has a gaze tracking function will be described with reference to FIG.
4, the content experience system 1B includes a controller 2, a stereoscopic image generation device 3B, a stereoscopic display 4, and a camera 5. Components other than the stereoscopic image generation device 3B and the camera 5 are similar to those in the first configuration, and therefore descriptions thereof will be omitted.

The stereoscopic image generating device 3B generates stereoscopic images using a spatial image reproduction method, similar to the stereoscopic image generating device 3 in FIG. 1. At this time, the stereoscopic image generating device 3B detects the viewpoint position (position of both eyes) of the user U using a viewpoint tracking function, and displays a stereoscopic image only in the viewing zone of the user U. This viewpoint tracking function enables the stereoscopic display 4 in the stereoscopic image generating device 3B to use a lens array with a long focal length, and the depth range determined by the light density can be expanded. Furthermore, the stereoscopic image generating device 3B can expand the viewing zone by dynamically tracking the viewing zone of the user U.

Camera 5 is a camera for detecting the viewpoint position. For example, camera 5 is a general web camera that captures an image of the face area of user U. Camera 5 then outputs the captured face area image to stereoscopic image generating device 3B.

<Fourth Configuration: Multiple users, with viewpoint tracking function>
A system configuration in the case where there are multiple users U and the system has a viewpoint tracking function will be described with reference to Fig. 5. As shown in Fig. 5, the content experience system 1B includes a controller 2 ( ₂₁ , ..., _2N ), a stereoscopic display 4 ( ₄₁ , ..., _4N ), and a camera 5 ( ₅₁ , ..., _5N ) for each user U. A stereoscopic image generating device 3B in Fig. 5 generates a stereoscopic image for each user U. In other respects, the content experience system 1B in the fourth configuration is similar to the third configuration, and therefore description thereof will be omitted.

As described above, the content experience systems 1 and 1B can accommodate the number of users U and the presence or absence of a viewpoint tracking function with only simple configuration changes, improving versatility. In other words, the content experience systems 1 and 1B can flexibly change their system configuration depending on the number of users U and the desired video quality.

Furthermore, the spatial image reproduction type 3D display 4 can reproduce light rays that are almost identical to those emitted by objects, and is said to be less likely to cause eye fatigue due to a mismatch between eye convergence and adjustment (Non-Patent Document 3). In this way, the content experience systems 1 and 1B employ a spatial image reproduction type 3D display 4 that can display natural 3D images, thereby reducing eye fatigue even in interactive content that is often viewed for long periods of time, such as games.

However, in conventional video games using two-dimensional images, the player character is simply displayed on a flat screen, and it is impossible to visually sense the thickness of the player character's body, or for the character to jump out of the screen or move into the distance. In other words, it is difficult to get a strong sense of realism or presence in conventional video games using two-dimensional images. On the other hand, content experience systems 1 and 1B provide a sense of three-dimensionality compared to video games using two-dimensional images, which not only improves the sense of realism and character presence, but also makes it possible to create new effects using binocular parallax and motion parallax.

First Embodiment
[Configuration of stereoscopic image generating device]
The configuration of the stereoscopic image generating device 3 will be described with reference to FIG.
As shown in Fig. 6, the stereoscopic image generating device 3 includes a storage unit 30, an operation signal conversion unit 31, a multi-viewpoint image generating unit 32, and a stereoscopic image conversion unit 33. Note that the stereoscopic image generating device 3 in Fig. 6 corresponds to the system configuration (the first configuration in Fig. 1) in the case where there is only one user U and there is no viewpoint tracking function.

The storage unit 30 is a storage device such as a memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) that stores three-dimensional CG scenes M of interactive content. Here, it is assumed that a user of the content experience system 1 stores a desired three-dimensional CG scene M in advance in the storage unit 30. For example, in the case of a fighting game, the three-dimensional CG scene M is composed of two or more player character models and a background model. In the following explanation, the processing related to the camerawork of the virtual camera will be mainly explained, and a description of the general production method of interactive content will be omitted.

The operation signal conversion unit 31 receives an operation signal from a user U and converts the input operation signal into the position, attitude, and size of a spatial image reproduction type virtual display that virtually displays a three-dimensional CG scene M based on a predetermined conversion rule. In this embodiment, the operation signal conversion unit 31 receives an operation signal from the controller 2 operated by each user U and converts this operation signal into the position, attitude, and size of the virtual display. The operation signal conversion unit 31 then outputs the position, attitude, and size of the converted virtual display to the multi-viewpoint image generation unit 32.

Virtual displays and virtual cameras in 3D CG scenes
A virtual display 90 and a virtual camera array 91 in a three-dimensional CG scene M will now be described.

As shown in FIG. 7, a spatial image reproduction type virtual display 90 is placed in a three-dimensional CG scene M to be displayed as a stereoscopic image. A stereoscopic image (element image E) can be generated by photographing this virtual display 90 with a virtual camera array 91. For example, the virtual display 90 is a virtual display equipped with a display and a lens array, similar to the integral method. The virtual camera array 91 has multiple virtual cameras 92 arranged in a two-dimensional direction. Here, the virtual camera array 91 is arranged at equal intervals at positions a predetermined viewing distance D away from the display surface (virtual display surface) of the virtual display 90 in the depth direction. The viewing distance D is the distance from the user U to the virtual display 90.

7, virtual camera array 91 is composed of five virtual cameras 92 ₁ to 92 ₅ arranged in the horizontal direction. All virtual cameras 92 ₁ to 92 ₅ are arranged facing virtual display 90, and internal parameters are set in advance so that the angles of view of the cameras match the area of virtual display 90 when capturing images.

Here, the midpoint of the user U's eyes (viewpoint position P) coincides with the center of the virtual camera array 91. In addition, in the depth direction, the center of the virtual camera array 91 coincides with the center of the virtual display 90. In addition, in FIG. 7, the horizontal direction is the x-axis, the vertical direction is the y-axis, and the depth direction is the z-axis.

The camera work when shooting a general two-dimensional image consists of dolly (position change), pan/tilt/roll (posture change), and zoom in/zoom out (focal length change). When these operations are performed in a three-dimensional virtual space, the camera work is equivalent to the operation of the virtual display 90. As shown in FIG. 8(a), dollying the virtual camera 92 corresponds to changing the position of the virtual display 90. In FIG. 8, a person and a dog are illustrated as the three-dimensional CG scene M. In addition, the angle of view α of the virtual camera 92 before the camera work is illustrated with a dashed line, and the angle of view β of the virtual camera 92 after the camera work is illustrated with a solid line. In other words, the angles of view α and β of the virtual camera 92 represent the changes in the position, posture, and size of the virtual display 90 before and after the camera work. As shown in FIG. 8(b), panning, tilting, or rolling the virtual camera 92 corresponds to changing the posture of the virtual display 90. As shown in FIG. 8(c), zooming in or zooming out of the virtual camera 92 corresponds to reducing or enlarging the size of the virtual display 90.

As described above, the camerawork of the virtual camera 92 can be considered equivalent to the operation of the virtual display 90. Therefore, the operation signal conversion unit 31 converts the operation signal from the controller 2 into the operation (position, orientation, and size) of the virtual display 90 in order to reflect the operation by the user U in the interactive content.

The conversion rules for converting the user U's operation signal into an operation on the virtual display 90 can be set arbitrarily. For example, the conversion rules may be such that, in the case of a first-person or third-person viewpoint, the user U's operation is treated as an operation on the virtual display 90 as is. In addition, the conversion rules may be such that, in the case of a fixed viewpoint within a game stage, when the user U moves the player character to a specific position on the game stage, the viewpoint is switched to one that corresponds to that position.

Returning to FIG. 6, the description of the stereoscopic image generating device 3 will be continued.
The multi-viewpoint image generating unit 32 calculates the camera parameters of a virtual camera array 91 in which virtual cameras 92 are arranged, using a rigid transformation matrix, in accordance with the position, orientation, and size of the virtual display 90. The multi-viewpoint image generating unit 32 then generates a multi-viewpoint image by capturing an image of the virtual display 90 displaying a three-dimensional CG scene M with the virtual camera array 91.

<Generating multi-view images reflecting camera work>
Generation of a multi-viewpoint image reflecting camera work will be described with reference to FIG.
The camera parameters of the virtual camera 92 include its position, orientation, and internal parameters. Furthermore, the camera parameters of the virtual camera 92 change depending on the position, orientation, and size of the virtual display 90. Hereinafter, in generating the multi-view image V, the position and orientation of the virtual display 90 and the size of the virtual display 90 will be considered separately.

9 illustrates each coordinate system in the virtual space. The world coordinate system _ΣW is a three-dimensional coordinate system that serves as a reference in the virtual space, and has a desired position as its origin. The virtual display coordinate system _ΣD is a three-dimensional coordinate system that uses the virtual display 90 as its reference, and has, for example, the center of the virtual display surface as its origin. The camera coordinate system _ΣC is a coordinate system that uses one virtual camera 92 that constitutes the virtual camera array 91 as its reference, and has, for example, the optical principal point of the virtual camera 92 as its origin.

Generally, a coordinate system can be transformed by a rigid transformation matrix. Here, transformation from the coordinate system _ΣA to the coordinate system _ΣB is expressed by the rigid transformation matrix ^B M _A. In this case, the coordinates ^A X of a three-dimensional point in the coordinate system _ΣA can be transformed into three-dimensional coordinates ^B X in the coordinate system _ΣB by the following formula (1). In the three-dimensional coordinates X, the upper left subscript indicates the coordinate system, and the superscript tilde indicates homogeneous coordinates.

Moreover, the homogeneous coordinates of three-dimensional coordinate X=[X, Y, Z] ^T are expressed by the following formula (2): where the subscript T represents the transpose of a vector.

9, a rigid transformation matrix ^D M _W represents a coordinate transformation from the world coordinate system Σ _W to the virtual display coordinate system Σ _D. Also, a rigid transformation matrix ^C M _D represents a coordinate transformation from the virtual display coordinate system Σ _D to the camera coordinate system Σ _C.

First, consider the position and orientation of the virtual display 90. The position ^D P of the camera in the virtual display coordinate system Σ _D can be converted to the position ^W P of the camera in the world coordinate system Σ _W as shown in the following equation (3).

Here, if no camera work is performed, the rigid transformation matrix ^D M _W will not change and will be a fixed value. However, as described above, since camera work is performed, the rigid transformation matrix ^D M _W changes according to the position and attitude of the virtual display 90. For example, since the rigid transformation matrix ^D M _W is expressed by a rotation matrix and a translation vector, it is only necessary to change the translation vector of the rigid transformation matrix ^D M _W in accordance with the position of the virtual display 90, and change the rotation matrix of the rigid transformation matrix ^D M _W in accordance with the attitude of the virtual display 90.

Next, consider the size of the virtual display 90. Changing the size of the virtual display 90 is equivalent to changing the lens pitch or focal length of the virtual display 90. The viewing distance also changes proportionally accordingly. Therefore, the position of the virtual camera 92 in the virtual display coordinate system _ΣD changes.

As shown in Fig. 10 and Fig. 11, when the size of the virtual display 90 changes, the position of the virtual camera 92 changes. As shown in Fig. 10, the virtual display 90 has a predetermined size, and virtual cameras 92 ₁ , ..., 92 _K ..., 92 _N are arranged at regular intervals with a predetermined angle of view (where 1<K<N). In addition, the viewing distance is D, and the camera interval between each virtual camera 92 is W. At this time, the center of the virtual camera array 91 (virtual camera 92 _K ) and the center of the virtual display 90 coincide with each other in the depth direction. As shown in Fig. 11, when the size of the virtual display 90 is multiplied by a, the viewing distance becomes aD, and the camera interval between each virtual camera 92 becomes aW. At this time, the attitude and internal parameters of the virtual camera 92 do not change.

As described above, the multi-viewpoint image generation unit 32 uses the rigid transformation matrix of equation (3) to calculate camera parameters for each virtual camera 92 according to the position, orientation, and size of the virtual display 90. The multi-viewpoint image generation unit 32 then generates a multi-viewpoint image by capturing an image of the virtual display 90 with each virtual camera 92, and outputs the generated multi-viewpoint image to the stereoscopic image conversion unit 33.

Returning to FIG. 6, the description of the stereoscopic image generating device 3 will be continued.
The stereoscopic image conversion unit 33 converts the multi-viewpoint image generated by the multi-viewpoint image generation unit 32 into a stereoscopic image. In this embodiment, as shown in Fig. 12, the stereoscopic image conversion unit 33 converts the multi-viewpoint image V input from the multi-viewpoint image generation unit 32 into an element image E by rearranging the pixel positions of the multi-viewpoint image V. Then, the stereoscopic image conversion unit 33 outputs the converted element image E to the stereoscopic display 4. When this element image E is displayed on the stereoscopic display 4, the user U can see a stereoscopic image.

[Operation of stereoscopic image generating device]
The operation of the stereoscopic image generating device 3 will be described with reference to FIG.
It is assumed here that a three-dimensional CG scene M is stored in advance in the storage unit 30 .

In step S1, the operation signal conversion unit 31 receives an operation signal from the user U and converts the input operation signal into the position, attitude, and size of the virtual display 90 based on a predetermined conversion rule. For example, the conversion rule may be such that, in the case of a first-person or third-person viewpoint, the operation of the user U is treated as an operation of the virtual display 90 as is. In addition, the conversion rule may be such that, in the case of a fixed viewpoint within a game stage, when the user U moves the player character to a predetermined position in the game stage, the viewpoint is switched to one that corresponds to that position.

In step S2, the multi-viewpoint image generation unit 32 calculates the camera parameters of the virtual camera array 91 in which the virtual cameras 92 are arranged, using a rigid transformation matrix, according to the position, orientation, and size of the virtual display 90. Here, the multi-viewpoint image generation unit 32 calculates the camera parameters for each virtual camera 92, according to the position, orientation, and size of the virtual display 90, using the rigid transformation matrix of the above-mentioned formula (3). Then, the multi-viewpoint image generation unit 32 generates a multi-viewpoint image V by photographing the virtual display 90 with each virtual camera 92.

In step S3, the stereoscopic image conversion unit 33 converts the multi-viewpoint image V generated by the multi-viewpoint image generation unit 32 into a stereoscopic image. In this embodiment, the stereoscopic image conversion unit 33 converts the multi-viewpoint image V into an element image E by rearranging the pixel positions of the multi-viewpoint image V input from the multi-viewpoint image generation unit 32, as shown in FIG. 12.

[Action and Effects]
The camerawork of the virtual camera 92 is considered to be equivalent to changes in the position, attitude, and size of the virtual display. Therefore, the stereoscopic image generating device 3 according to the first embodiment reflects the operation of the user U in the position, attitude, and size of the virtual display 90, and captures the virtual display 90 with the virtual camera array 91. In this way, the stereoscopic image generating device 3 can easily reflect the operation of the user U in the camerawork of the virtual camera in order to produce interactive content such as a video game.

Second Embodiment
[Configuration of stereoscopic image generating device]
With reference to FIG. 14, the configuration of a stereoscopic image generating device 3 according to the second embodiment will be described with respect to differences from the first embodiment.
The integral type three-dimensional display 4 has a characteristic that the spatial resolution decreases the further away from the display surface. In this case, when an object far from the display surface is reproduced, it is difficult to visually recognize it correctly, resulting in a three-dimensional image that is very difficult to see. In other words, the three-dimensional display 4 has a narrow depth range of the three-dimensional CG scene M that can be reproduced as a three-dimensional image. For this reason, the three-dimensional image generating device 3 differs from the first embodiment in that it compresses the depth of the three-dimensional CG scene M to match the three-dimensional display 4.

As shown in FIG. 14, the stereoscopic image generating device 3 includes a storage unit 30, an operation signal conversion unit 31, a multi-viewpoint image generating unit 32, a stereoscopic image conversion unit 33, and a depth compression unit 34. Note that each unit other than the multi-viewpoint image generating unit 32 and the depth compression unit 34 is the same as in the first embodiment, and therefore a description thereof will be omitted.

The depth compression unit 34 compresses the depth of the three-dimensional CG scene M using a preset depth compression function. In this embodiment, the depth compression unit 34 compresses the depth of the three-dimensional CG scene M stored in the storage unit 30. Here, the depth compression unit 34 may use either a linear depth compression function or a nonlinear depth compression function (References 2 and 3). At this time, the user of the content experience system 1 sets in the depth compression unit 34 which depth compression function to use. The linear depth compression function is a function that uniformly compresses the entire three-dimensional CG scene M in the depth direction, and is expressed by, for example, a linear function. The nonlinear depth compression function is a function that compresses the three-dimensional CG scene M in the depth direction so that the compression rate is small in the range where the spatial resolution is high, and is expressed by, for example, a sigmoid function. Then, the depth compression unit 34 outputs the three-dimensional CG scene M after the depth compression to the multi-viewpoint image generation unit 32.

Reference 2: Y. Sawahata and T. Morita, “Estimating Depth Range Required for 3-D Displays to Show Depth-Compressed Scenes Without Inducing Sense of Unnaturalness”, IEEE Transactions on Broadcasting, Vol. 64, No. 2, pp. 488 - 497 (2018)
Reference 3: Sawabata, "Natural integral 3D display technology based on spatial cognition characteristics," Japan Broadcasting Corporation, NHK Research and Development R&D/No. 164, August 2017

The multi-viewpoint image generating unit 32 uses a virtual camera array 91 to capture an image of a virtual display 90 displaying the depth-compressed 3D CG scene M. In this embodiment, the multi-viewpoint image generating unit 32 is similar to the first embodiment except that the 3D CG scene M is input from the depth compression unit 34, and therefore further explanation is omitted.

[Action and Effects]
As described above, the stereoscopic image generating device 3 according to the second embodiment can easily reflect the operation of the user U in the camerawork of the virtual camera, similarly to the first embodiment.
Furthermore, since the stereoscopic image generating device 3 compresses the depth of the three-dimensional CG scene M, even for interactive content having a wide depth range, degradation of spatial resolution is suppressed, and high-quality stereoscopic images can be displayed.

(Variation 1)
In the first and second embodiments, the content experience system 1 is described as including only one controller 2 and one stereoscopic display 4, and corresponds to the system configuration of FIG. 1, but is not limited to this.
For example, the content experience system 1 may be provided with a plurality of controllers 2, and may have a system configuration (second configuration in FIG. 3) corresponding to a plurality of users U. In this case, the operation signal conversion unit 31 averages the operation signals from the respective controllers 2 and converts the operations by the plurality of users U into camerawork. Furthermore, the operation signal conversion unit 31 may convert the operation signal into camerawork so that the operation signal of the user U who is performing the operation first is reflected.

Third Embodiment
[Configuration of stereoscopic image generating device]
With reference to FIG. 15, the configuration of a stereoscopic image generating device 3B according to the third embodiment will be described with respect to differences from the first embodiment.
The stereoscopic image generating device 3B according to the third embodiment is different from the first embodiment in that it has a viewpoint tracking function. Note that the stereoscopic image generating device 3B in Fig. 15 corresponds to the system configuration (the third configuration in Fig. 4) in the case where there is only one user U and the viewpoint tracking function is provided.

As shown in FIG. 15, the stereoscopic image generating device 3B includes a storage unit 30, an operation signal conversion unit 31, a multi-viewpoint image generating unit 32B, a stereoscopic image conversion unit 33B, and a viewpoint position detection unit 35. Note that each unit other than the multi-viewpoint image generating unit 32 and the viewpoint position detection unit 35 is the same as in the first embodiment, and therefore a description thereof will be omitted.

The viewpoint position detection unit 35 detects the viewpoint position of each user U. In this embodiment, a known method is used to detect the midpoint between the eyes of the user U from the facial area image of each user U input from the camera 5 as the viewpoint position (References 4 and 5). In this embodiment, since there is only one user U, the viewpoint position detection unit 35 detects the viewpoint position of one person. Then, the viewpoint position detection unit 35 outputs the detected viewpoint position of the user U to the multi-viewpoint image generation unit 32B.

Reference 4: JP 2019-213127 A Reference 5: Okaichi, "Development of a viewpoint tracking integral 3D image display system," Journal of the Institute of Image Information and Television Engineers, Vol. 75, No. 1, pp. 125-130 (2021)

The multi-viewpoint image generating unit 32B changes the rigid transformation matrix according to the viewpoint position input from the viewpoint position detecting unit 35, and generates a multi-viewpoint image V for each user U using the rigid transformation matrix for each user U. Here, the multi-viewpoint image generating unit 32B finds a rigid transformation matrix from the viewpoint position of each user U, and generates a multi-viewpoint image V corresponding to the viewpoint position of each user U. In this embodiment, since there is only one user U, the multi-viewpoint image generating unit 32B detects a multi-viewpoint image for one person.

<Generating multi-viewpoint images according to viewpoint position>
The generation of a multi-viewpoint image V according to the viewpoint position will be described with reference to FIG.
When the viewpoint tracking function is provided, the position and internal parameters of each virtual camera 92 in the virtual camera array 91 change according to the viewpoint position of the user U. As shown in Fig. 16, when the viewpoint position P of the user U moves, the position and internal parameters of each virtual camera 92 in the virtual camera array 91 are set according to the viewpoint position P after the movement.

Specifically, when the viewpoint tracking function is provided, the rigid transformation _matrix ^CMD from the virtual display coordinate system _ΣD to the camera coordinate system _ΣC changes. At this time, the position of the virtual camera 92 changes, but the attitude of the virtual camera 92 does not change. That is, _in the rigid transformation _matrix ^CMD , as shown in the following formula (4), the translation vector ^CT changes according to the viewpoint position, while the rotation matrix ^CRD becomes a unit matrix.

From the above, the multi-viewpoint image generation unit 32B uses equation (4) to change the rigid transformation matrix ^C M _D in accordance with the viewpoint position of the user U, and sets a virtual camera array 91 for each user U. Thereafter, the multi-viewpoint image generation unit 32B generates a multi-viewpoint image in the same manner as in the first embodiment, and outputs the generated multi-viewpoint image to the stereoscopic image conversion unit 33B.

Returning to FIG. 15, the description of the configuration of the stereoscopic image generating device 3B will be continued.
The stereoscopic image conversion unit 33B converts the multi-viewpoint image into a stereoscopic image for each user U. In this example, since there is only one user U, the stereoscopic image conversion unit 33B converts a stereoscopic image for one person. Note that the method of converting a stereoscopic video by the stereoscopic image conversion unit 33B is the same as in the first embodiment, and therefore a description thereof will be omitted. Then, the stereoscopic image conversion unit 33B outputs the stereoscopic image (element image E) generated for each user U to the stereoscopic display 4 corresponding to each user U.

[Action and Effects]
As described above, the stereoscopic image generating device 3B according to the third embodiment can easily reflect the operation of the user U in the camerawork of the virtual camera, similarly to the first embodiment.
Furthermore, the stereoscopic image generating device 3B can display a stereoscopic image only in the area where both eyes of the user U are located by using a viewpoint tracking function. This allows the stereoscopic display 4 to use a lens array with a long focal length, and the depth range can be expanded by the light density. Furthermore, the stereoscopic image generating device 3B can expand the viewing zone by using the viewpoint tracking function, and display a high-quality stereoscopic image.

(Variation 2)
The viewpoint position detection unit 35 may determine whether there is a plurality of users U based on the number of detected viewpoint positions. For example, the viewpoint position detection unit 35 may divide the number of eyes included in the face area image by 2 (rounding down to the nearest whole number) and determine the result as the number of users U.

Then, if it is determined that there are multiple users U, the multi-viewpoint image generating unit 32B may generate a multi-viewpoint image V for each user U. Furthermore, if it is determined that there are multiple users U, the stereoscopic image converting unit 33B may convert the multi-viewpoint image V into a stereoscopic image (element image E) for each user U.

In this way, it is possible to switch between the presence and absence of the viewpoint tracking function depending on the number of users U, making the content experience system 1B's system configuration more flexible and further improving its versatility.

Fourth Embodiment
[Configuration of stereoscopic image generating device]
With reference to FIG. 17, the configuration of a stereoscopic image generating device 3B according to the fourth embodiment will be described with respect to differences from the third embodiment.
A stereoscopic image generating device 3B according to the fourth embodiment differs from the third embodiment in that the depth of the three-dimensional CG scene M is compressed.

As shown in FIG. 17, the stereoscopic image generating device 3B includes a memory unit 30, an operation signal conversion unit 31, a multi-viewpoint image generating unit 32B, a stereoscopic image conversion unit 33B, a depth compression unit 34, and a viewpoint position detection unit 35.

The multi-viewpoint image generating unit 32B uses a virtual camera array 91 to capture an image of a virtual display 90 on which the depth-compressed 3D CG scene M is being played. In this embodiment, the multi-viewpoint image generating unit 32B is the same as in the third embodiment except that the 3D CG scene M is input from the depth compression unit 34, and therefore further explanation is omitted.

The storage unit 30, the operation signal conversion unit 31, the stereoscopic image conversion unit 33B, and the viewpoint position detection unit 35 are similar to those in the third embodiment, and therefore their description will be omitted.
Moreover, the depth compression section 34 is similar to that in the second embodiment, and therefore a description thereof will be omitted.

[Action and Effects]
As described above, the stereoscopic image generating device 3B according to the third embodiment can easily reflect the operation of the user U in the camerawork of the virtual camera, similarly to the third embodiment.
Furthermore, similarly to the third embodiment, the stereoscopic image generating device 3B can expand the depth range and also expand the viewing zone by using the viewpoint tracking function, thereby displaying a high-quality stereoscopic image.
Furthermore, similarly to the second embodiment, the stereoscopic image generating device 3B can suppress a decrease in spatial resolution and display a high-quality stereoscopic image.

(Variation 3)
In the third and fourth embodiments, the content experience system 1B is described as including only one controller 2 and one stereoscopic display 4, and corresponds to the system configuration of FIG. 4, but the present invention is not limited to this.
For example, the content experience system 1B may have a plurality of controllers 2 and three-dimensional displays 4, and may have a system configuration corresponding to a plurality of users U (fourth configuration in FIG. 5).

(Other Modifications)
Although each embodiment has been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.
In the above-described embodiments, the spatial image reproduction method is described as being an integral method, but is not limited thereto. For example, the spatial image reproduction method may be a lenticular method. In this case, the stereoscopic display may include a lenticular lens array in which semi-cylindrical lenticular lenses are arranged in a one-dimensional direction, and a display element. In this case, the multi-viewpoint image generating unit may use a virtual camera array in which virtual cameras are arranged in a one-dimensional direction.

In each of the above-described embodiments, the stereoscopic image generating device has been described as independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for causing hardware resources such as a CPU, memory, and hard disk of a computer to function as the stereoscopic image generating device described above. This program may be distributed via a communication line, or may be written to a recording medium such as a CD-ROM or flash memory and distributed.

Reference Signs List 1, 1B Content experience system 2 Controller 3, 3B Stereoscopic image generating device 4 Stereoscopic display 5 Camera 30 Memory unit 31 Operation signal conversion unit 32, 32B Multi-viewpoint image generating unit 33, 33B Stereoscopic image conversion unit 34 Depth compression unit 35 Viewpoint position detection unit 90 Virtual display 91 Virtual camera array 92 Virtual camera E Element image M Three-dimensional CG scene V Multi-viewpoint image

Claims (5)

A stereoscopic image generating device that generates a stereoscopic image by a spatial image reproduction method as interactive content reflecting a user's operation, comprising:
an operation signal conversion unit which receives an operation signal from the user and converts the input operation signal into a position, an attitude and a size of a spatial image reproduction type virtual display which virtually displays a three-dimensional CG scene based on a predetermined conversion rule;
a multi-viewpoint image generating unit that calculates camera parameters of a virtual camera array in which virtual cameras are arranged, based on a rigid body transformation matrix, in accordance with a position, orientation, and size of the virtual display, and generates a multi-viewpoint image by capturing an image of the virtual display displaying the three-dimensional CG scene with the virtual camera array;
a stereoscopic image conversion unit that converts the multi-viewpoint image generated by the multi-viewpoint image generation unit into the stereoscopic image;
A stereoscopic image generating device comprising: a depth compression unit that compresses the depth of the three-dimensional CG scene using a preset depth compression function;
2. The stereoscopic image generating device according to claim 1, wherein the multi-viewpoint image generating section captures the virtual display displaying the depth-compressed three-dimensional CG scene with the virtual camera array. A viewpoint position detection unit that detects a viewpoint position for each user is further provided,
the multi-viewpoint image generation unit changes the rigid transformation matrix in accordance with the viewpoint position, and generates the multi-viewpoint image for each user by using the rigid transformation matrix for each user;
3. The stereoscopic image generating device according to claim 1, wherein the stereoscopic image conversion unit converts the multi-view image into the stereoscopic image for each of the users. the viewpoint position detection unit determines whether or not there are a plurality of users based on the number of the detected viewpoint positions;
When it is determined that there are a plurality of users, the multi-viewpoint image generating unit generates the multi-viewpoint image for each of the users,
The stereoscopic image generating device according to claim 3 , wherein when it is determined that there are a plurality of users, the stereoscopic image conversion unit converts the multi-view image into the stereoscopic image for each of the users. A program for causing a computer to function as a stereoscopic image generating device according to any one of claims 1 to 4.

Images