JP7614765B2 - Content presentation device and program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies (1) Name of the publisher: Japan Broadcasting Corporation Date of website posting: July 27, 2020 Posting address: https://www.nhk.or.jp/info/pr/marukaji/assets/pdf/450.pdf https://www.nhk.or.jp/strl/news/2020/7.html (2) Name of the publisher: Japan Broadcasting Corporation Name of publication: GiKen Dayori, August 2020 Issue Date of posting: August 6, 2020 Posting address: https://www.nhk.or.jp/strl/ https://www.nhk.or. jp/strl/publica/giken_dayori/185/1.html https://www.nhk.or.jp/strl/publica/giken_dayori/185/pdf/dayori185.pdf (3) Publisher: Dempa Shimbun Co., Ltd. Publication name: Dempa Shimbun, dated August 3, 2020, page 3 Publication date: August 3, 2020 (4) Publisher: Dempa Shimbun Co., Ltd. Publication name: Dempa Shimbun, dated September 2, 2020, page 1 Publication date: September 2, 2020

The present invention relates to a content presentation device and a program.

2. Description of the Related Art When experiencing Augmented Reality (AR) content and Virtual Reality (VR) content, there is a demand for a system that provides a sense of experiencing the content together with other people.
For example, in Non-Patent Document 1, in order to verify the effectiveness of VR shared experiences such as games, movies, and meetings, a system is constructed in which a person in another space is reproduced next to the user by capturing and transmitting three-dimensional images in real time, while the user is reproduced next to the person in the other space.
In addition, Non-Patent Document 2 proposes an AR system that enables natural communication with a person in another space, in which a person in another space is photographed in three dimensions and the person in the other space is reproduced in real time in the AR space that the person is experiencing. Non-Patent Document 2 also mentions using the system for VR by changing the device.
Furthermore, there is a demand for an AR/VR system that allows people who are physically separated as well as those in the same space to share content experiences in real time while feeling the presence of the other person.

Simon et al, “EVERYDAY PHOTO-REALISTIC SOCIAL VR:COMMUNICATE AND COLLABORATE WITH AN ENHANCED CO-PRESENCE AND IMMERSION”, IBC 2019 Sergio et al, “Holoportation: Virtual 3D Teleportation in Real-time”, UIST ’16, October 16-19, 2016

To be able to share content in the same space with people in different spaces, it is necessary to share the relative positions of each other without any contradictions, including the position of the content (AR content, VR content, or AR/VR mixed content that is a mixture of AR content and VR content).

The present invention has been made in consideration of the above points, and provides a content presentation device and program that can provide the sensation of experiencing content with people in different spaces while sharing the same space.

[1] In order to solve the above problem, a content presentation device according to one aspect of the present invention includes a reference position setting unit that sets, in a three-dimensional space, first subject reference position information, which is information indicating a reference position and orientation for a first subject video, which is a video including an image of a first subject, and second subject reference position information, which is information indicating a reference position and orientation for a second subject video, which is a video including an image of a second subject, a surrounding video acquisition unit that acquires a surrounding video, which is a video of the surroundings of the device including the first subject, a first subject video extraction unit that extracts the first subject video from the surrounding video, a background video acquisition unit that acquires a background video, which is a video used as a background, a second subject video acquisition unit that acquires the second subject video from another content presentation device, a content acquisition unit that acquires content, a first subject position setting unit that sets a position and orientation of the first subject video in the three-dimensional space based on the first subject reference position information, and a position and orientation of the second subject video in the three-dimensional space based on the first subject reference position information. a second subject position setting unit that sets a position in the three-dimensional space where the content is to be displayed based on a predetermined relative positional relationship between the position indicated by the first subject reference position information and the position indicated by the second subject reference position information; and a presentation unit that outputs a presentation image which is an image in the three-dimensional space where the first subject video, the position and orientation of which in the three-dimensional space is set, the second subject video, the position and orientation of which in the three-dimensional space is set, and the content, the position of which in the three-dimensional space is set, are displayed against the background of the background image, wherein the relative positional relationship between the position in the three-dimensional space where the content is to be displayed, the position indicated by the first subject reference position information, and the position indicated by the second subject reference position information, the first subject reference position information, and the second subject reference position information are each shared with the other content presentation device, and the presentation unit generates the presentation image using a video see-through method .

[2] In one aspect of the present invention, in the content presentation device described above, the presentation unit outputs the sound included in the content by localizing the sound based on the position in the three-dimensional space where the content is displayed.

[3] In one aspect of the present invention, in the content presentation device described above, the presentation unit localizes and outputs the sound included in the second object image based on the position of the second object image in the three-dimensional space.

[4] In one aspect of the present invention, the content presentation device further includes an occlusion reproduction unit that determines whether the first object image or the second object image is in the foreground in the presented image based on the position of the first object image in the three-dimensional space and the position of the second object image in the three-dimensional space, and the presentation unit outputs the presented image based on the determination result of the occlusion reproduction unit.

[5] In one aspect of the present invention, in the content presentation device described above, the second subject position setting unit sets the position of the second subject image in the three-dimensional space by matching the position of the feet of the second subject included in the second subject image with the position of the bottom surface of the three-dimensional space based on the second subject reference position information.

[6] In one aspect of the present invention, in the content presentation device described above, the presentation unit displays the second subject image at a transparency level less than a predetermined level.

[8] Also, one aspect of the present invention includes a reference position setting process for setting, in a three-dimensional space, first subject reference position information, which is information indicating a reference position and orientation for a first subject video, which is a video including an image of a first subject, and second subject reference position information, which is information indicating a reference position and orientation for a second subject video, which is a video including an image of a second subject, a surrounding image acquisition process for acquiring a surrounding image, which is an image of the surroundings of a content presentation device including the first subject, a first subject image extraction process for extracting the first subject video from the surrounding image, a background image acquisition process for acquiring a background image, which is an image used as a background, a second subject image acquisition process for acquiring the second subject video from another content presentation device, a content acquisition process for acquiring content, a first subject position setting process for setting a position and orientation of the first subject video in the three-dimensional space based on the first subject reference position information, and a second subject image acquisition process for setting a position and orientation of the second subject video in the three-dimensional space based on the second subject reference position information. A program that causes a computer to execute the processes of: a subject position setting process; a content position setting process that sets a position in the three-dimensional space where the content is to be displayed based on a predetermined relative positional relationship between the position indicated by the first subject reference position information and the position indicated by the second subject reference position information; and a presentation process that outputs a presentation image, which is an image displayed in the three-dimensional space against the background image, of the first subject image whose position and orientation in the three-dimensional space are set, the second subject image whose position and orientation in the three-dimensional space are set, and the content whose position in the three-dimensional space is set, wherein the relative positional relationship between the position in the three-dimensional space where the content is displayed, the position indicated by the first subject reference position information, and the position indicated by the second subject reference position information, the first subject reference position information, and the second subject reference position information are each shared with the other computer , and the presentation process is a program that generates the presentation image using a video see-through method .

The present invention provides the feeling that people in different spaces are experiencing content while sharing the same space.

1 is a diagram illustrating an example of a configuration of a content presentation system according to an embodiment of the present invention. 1 is a diagram illustrating an example of a configuration of a content presentation device according to an embodiment. FIG. 2 is a diagram illustrating an example of a configuration of an imaging unit according to the embodiment. FIG. 2 is a diagram illustrating an example of a configuration of a display unit according to the embodiment. FIG. 4 is a diagram illustrating an example of an imaging process according to the embodiment. FIG. 11 is a diagram illustrating an example of a display process according to the embodiment. FIG. 11 is a diagram illustrating an example of a position setting process according to the embodiment. FIG. 11 is a diagram showing an example of an image displayed on a head mounted display for setting the position of a subject according to the embodiment. 5A to 5C are diagrams illustrating an example of a relative positional relationship between content, a first object image, and a second object image according to an embodiment. FIG. 1 is a diagram illustrating an example of drawing an AR or VR object according to an embodiment. 10A to 10C are diagrams illustrating an example of reproduction of occlusion for a first object image and a second object image according to an embodiment. 11A to 11C are diagrams illustrating an example of reproduction of occlusion for a second object image and content according to the embodiment. FIG. 1 is a diagram showing an example of content that combines AR and VR according to an embodiment. FIG. 1 is a diagram showing an example of content that combines AR and VR according to an embodiment. FIG. 1 is a diagram showing an example of content that combines AR and VR according to an embodiment.

(Embodiment)
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a content presentation system S according to this embodiment. The content presentation system S includes a plurality of content presentation devices 1 (content presentation devices 1-1, 1-2), a server device 2, a plurality of head mounted displays H1 (head mounted displays H1-1, H1-2), a plurality of stereo cameras SC1 (stereo cameras SC1-1, SC1-2), and a plurality of RGBD cameras CM1 (RGBD cameras CM1-1, CM1-2). Since a user is photographed by the RGBD camera CM1, in this embodiment, the user is referred to as a subject.

In the content presentation system S, a plurality of subjects exist in different spaces that are separate from each other. In the example shown in FIG. 1, the first subject P1 and the second subject P2 exist in the first space R1 and the second space R2, respectively. The first space R1 and the second space R2 exist in locations that are separate from each other. The distance that the first space R1 and the second space R2 are separated from each other may be any distance. The first space R1 and the second space R2 may be separated by a wall of a building or the like, or may not be separated. The first space R1 and the second space R2 may be different locations in the same room. The first space R1 is, for example, a room in the home of the first subject P1, and the second space R2 is, for example, a room in the home of the second subject P2 that is located away from the home of the first subject P1.

The multiple content presentation devices 1 communicate with each other and work together. In FIG. 1, the content presentation device 1-1 and the content presentation device 1-2 are installed in a first space R1 and a second space R2, respectively, and operate simultaneously. Each content presentation device 1 is used by one subject. In FIG. 1, the content presentation device 1-1 and the content presentation device 1-2 are used by a first subject P1 and a second subject P2, respectively. The multiple content presentation devices 1 exchange information with each other via a network NW and a server device 2. The content presentation device 1 and the server device 2 are each connected to the network NW by wireless communication or wired communication.

The content presentation device 1 is, for example, a personal computer (PC). The content presentation device 1 may be a mobile terminal device such as a smartphone. The content presentation device 1 may be provided integrally with the head mounted display H1. For example, the content presentation device 1 may be built into the head mounted display H1 as a small terminal device, or the functions of the content presentation device 1 may be realized as a program, and the program may be executed by a computing device built into the head mounted display H1.

The configuration on the side of the first subject P1 will be described below, but the configuration on the side of the second subject P2 is similar to the configuration on the side of the first subject P1.
The first subject P1 is wearing a head mounted display H1-1. The head mounted display H1-1 includes a display device, a microphone, a speaker, and an HMD position measurement unit. The HMD position measurement unit measures the position and orientation of the head mounted display H1-1.
As an example, the stereo camera SC1-1 is provided integrally with the head mounted display H1. The stereo camera SC1-1 captures the scenery of the first space R1 as seen from the first subject P1, and a part of the body of the first subject P1. The part of the body of the first subject P1 is, for example, the first subject P1's own hand, arm, etc. as seen from the first subject P1's perspective. The head mounted display H1-1 reproduces the image captured by the stereo camera SC1-1 on a display device provided in the head mounted display H1-1.

The first subject P1 is photographed by the RGBD camera CM1-1 along with the scenery behind the first subject P1 in the first space R1. The first subject video PV1 is a video including an image of the first subject P1. The first subject video PV1 includes the image of the first subject P1 as well as the sound of the first subject P1 picked up by a microphone provided on the head-mounted display H1-1.

The image captured by the RGBD camera CM1-1 is transmitted to the content presentation device 1-2 via the content presentation device 1-1. On the other hand, the image captured by the RGBD camera CM1-2 on the side of the second subject P2 is received by the content presentation device 1-1 via the content presentation device 1-2. The head mounted display H1-1 causes the display device provided in the device to present the presentation image V1-1. The presentation image V1-1 includes the second subject image PV2 included in the image captured by the RGBD camera CM1-2, together with an image of a part of the body of the first subject P1 against the background of the background image BG1-1, and the content C1. Here, the background image BG1 is an image used as the background in the image reproduced by the head mounted display H1. In this embodiment, the background image BG1-1 used as the background in the image reproduced by the head mounted display H1-1 is, as an example, an image of the scenery of the first space R1 captured by the stereo camera SC1-1. The second subject video PV2 is a video including an image of the second subject P2 and includes the sound of the second subject P2.
The content C1 is displayed as an augmented reality (AR) image reproduced by the head mounted display H1. The content C1 is, for example, a television broadcast or a three-dimensional image using AR.

Similarly, on the side of the second subject P2, the head mounted display H1-2 causes the display device provided in the head mounted display H1-2 to present the presented image V1-2. The presented image V1-2 includes an image of part of the body of the second subject P2 against the background image BG1-2, as well as the first subject image PV1 contained in the image captured by the RGBD camera CM1-1, and content C1. The content C1 is common to the presented image V1-1 presented on the side of the first subject P1 and the presented image V1-2 presented on the side of the second subject P2.

As described above, the background image BG1 uses the scenery of the space that the subject exists in. Since the first subject P1 and the second subject P2 exist in different spaces that are separate from each other, the background image BG1-1 used on the side of the first subject P1 and the background image BG1-2 used on the side of the second subject P2 are different from each other.
In addition, a common image may be used on the side of the first subject P1 and the side of the second subject P2 as the background image BG1. For example, an image of a landscape of a third space other than the first space R1 or the second space R2 may be used as the background image BG1. The third space is, for example, a virtual space. This virtual space includes, for example, various facilities (for example, broadcast studios, conference rooms, movie theaters, aquariums, art museums, concert venues, entertainment facilities, etc.), outdoors (coasts, mountains, parks, tourist attractions, etc.), etc. The virtual space may be an image of a current landscape by real-time streaming, or an image of a landscape photographed in the past. The virtual space may be a place that does not actually exist created by computer graphics.

In the content presentation system S, common content is presented to multiple subjects that exist in different spaces separated from each other, making them feel as if they are in the same space.

[Configuration of the content presentation device]
The configuration of the content presentation device 1 will now be described with reference to FIGS.
2 is a diagram showing an example of the configuration of the content presentation device 1 according to the present embodiment. The content presentation device 1 includes a shooting unit 11 and a display unit 12. The shooting unit 11 transmits a first subject video PV1 to the content presentation device 1 on the side of the second subject P2. The display unit 12 displays the presentation video on a display device provided in the head mounted display H1.

At least some of the functions of each of the functional units of the content presentation device 1 may be realized using electronic circuits. Furthermore, some or all of these functional units may be realized using a computer and a program. Each functional unit has a storage means as necessary. The storage means is, for example, a flip-flop that maintains a predetermined state on the electronic circuit, a variable in the program when a program is used, or a memory that is allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may be used as necessary.

[Configuration of the imaging section]
FIG. 3 is a diagram showing an example of the configuration of the imaging unit 11 according to the present embodiment.
The RGBD image capturing unit 31 captures an RGBD image, which is an image of the surroundings of the device itself (RGBD camera CM1-1). The RGBD image includes a first subject P1 existing in a first space R1 and the scenery behind the first subject P1. The RGBD image capturing unit 31 corresponds to the RGBD camera CM1-1 shown in FIG.

The first subject position measuring unit 32 measures the position and attitude of the head mounted display H1-1. The first subject position measuring unit 32 is, for example, an HMD position measuring unit built into the head mounted display H1-1. Here, the position is information expressed by position coordinates in a three-dimensional space. Also, the attitude is information expressing the orientation of the head mounted display H1-1, for example, by three-dimensional angle information. The HMD position measuring unit may acquire the position and attitude by, for example, incorporating a gyro sensor or a camera. Also, since self-position estimation can be performed from an image captured by a stereo camera SC1-1 provided integrally with the head mounted display H1-1, the HMD position measuring unit may calculate the position and attitude from an image captured by the stereo camera SC1-1. Also, the HMD position measuring unit may acquire the position and attitude by receiving a beacon signal from an external source, or by receiving a signal such as infrared light emitted from an external source in synchronization with a clock to identify a location in real space. In addition, the HMD position measurement unit may have multiple cameras (e.g., RGBD camera CM1-1) installed outside (e.g., within a room which is the viewing space for the content) photograph the head mounted display H1-1 and receive the determined position and attitude information.
The first subject position measuring unit 32 outputs information indicating the measured position and orientation of the head mounted display H1-1 to the content presentation device 1 as HMD position information.

The first subject sound acquisition unit 33 acquires the sound of the first subject P1 present in the first space R1. The first subject sound acquisition unit 33 is a microphone built into the head mounted display H1-1. The first subject sound acquisition unit 33 outputs information indicating the acquired sound of the first subject P1 to the content presentation device 1 as first subject sound information.

The imaging unit 11 includes a first object image cropping unit 111 and a first object image supplying unit 112 .
The first subject image cutout unit 111 cuts out the first subject image PV1 from the RGBD image captured by the RGBD shooting unit 31. As an example, the first subject image cutout unit 111 generates distance information (depth map) based on distance information included in the RGBD image, and cuts out the first subject image PV1 from the RGBD image based on a preset distance (depth) based on the generated distance information. Here, the distance information is information indicating the distance (depth) seen from the RGBD camera CM1-1 for each pixel of the RGBD image. In the RGBD image captured by the RGBD shooting unit 31, the distance (depth) seen from the RGBD camera CM1-1 of the part corresponding to the first subject P1 is closer than the distance (depth) seen from the RGBD camera CM1-1 of the background part other than the first subject P1, so the first subject image cutout unit 111 can cut out the first subject image PV1 from the RGBD image based on the distance information. The preset distance (depth) is set in advance according to the distance between the first subject P1 and the RGBD camera CM1-1. It is preferable that no other object is placed between the first subject P1 and the RGBD camera CM1-1.

The method by which the first subject video cutout unit 111 cuts out the first subject video PV1 from the RGBD video is not limited to a method based on distance information. The first subject video cutout unit 111 may cut out the first subject video PV1 from the RGBD video based on machine learning. In this case, the first subject video cutout unit 111 has previously learned the characteristics of the first subject P1 in the video, for example, by machine learning. The first subject video cutout unit 111 cuts out the first subject video PV1 by identifying the location in the RGBD video where the first subject P1 appears (information such as the coordinates of an area in an image) by referring to the learned model. As another example, the first subject video cutout unit 111 may identify the location in the RGBD video where the first subject P1 appears based on pattern matching using a template.

The first subject video supply unit 112 supplies the first subject video PV1 cut out by the first subject video cut-out unit 111, the HMD position information output from the first subject position measurement unit 32, and the first subject audio information output from the first subject audio acquisition unit 33 to another content presentation device 1 (content presentation device 1-2). Note that the first subject audio information is supplied while being included in the first subject video PV1.

[Display Configuration]
4 is a diagram showing an example of the configuration of the display unit 12 according to the present embodiment. The display unit 12 includes an external information acquisition unit 13, a control command acquisition unit 14, a content acquisition unit 15, a processing unit 16, a data sharing unit 17, a second subject information acquisition unit 18, and a storage unit 19.

The external information acquisition unit 13 includes a surrounding image acquisition unit 131, a distance information acquisition unit 132, and a first subject position and orientation acquisition unit 133. The surrounding image acquisition unit 131 acquires a surrounding image E1 captured by the stereo camera SC1-1. The surrounding image E1 is an image of the surroundings of the own device (content presentation device 1-1) including the first subject P1. The surrounding image E1 includes the sound of the first subject P1.

The distance information acquisition unit 132 acquires distance information (depth map) corresponding to the surrounding image E1 acquired by the surrounding image acquisition unit 131. Here, the distance information acquisition unit 132 acquires (generates) the distance information (depth map) using stereo matching based on the surrounding image E1 captured by the stereo camera SC1-1. Note that distance information may be calculated from the surrounding image E1 captured by the stereo camera SC1-1 by a device other than the content presentation device 1, and the distance information acquisition unit 132 may acquire the distance information calculated by the other device. Note that the acquisition of a distance image itself can be achieved by using existing technology.

The first subject position/orientation acquisition unit 133 acquires first subject position/orientation information from the head mounted display H1-1. The first subject position/orientation information is information indicating the position and orientation of the first subject P1. The first subject position/orientation acquisition unit 133 acquires information indicating the position and orientation of the head mounted display H1-1 from the head mounted display H1-1 as the first subject position/orientation information.

The control command acquisition unit 14 acquires a control command from an operation unit (not shown). The control command indicates an operation on the content. The operation unit accepts operations from the first subject P1. The operation unit includes, for example, a remote controller, a keyboard, a controller, etc. The internal state of the content is updated in response to the operation on the content. The internal state of the content includes information indicating the selected content (such as a program), whether to play or stop, the volume, the position and orientation in which the content is displayed, the size of the area in which the content is displayed, etc. Selecting content means selecting a program (or channel), etc. The position in which the content is displayed is specified by the position in the left-right direction or the depth direction as viewed from the first subject P1.

The content acquisition unit 15 acquires the content C1 from the storage unit 19. Note that, in this embodiment, an example in which the content C1 is pre-stored in the storage unit 19 will be described, but this is not limiting. The content acquisition unit 15 may acquire content from a source other than the content presentation device 1. For example, the content acquisition unit 15 may acquire content from a recording medium such as a DVD, a Blu-ray disc, or a hard disk device. Alternatively, the content acquisition unit 15 may acquire content distributed by a signal for communication, broadcasting, or the like.

The processing unit 16 processes information from the external information acquisition unit 13, the control command acquisition unit 14, the content acquisition unit 15, the data sharing unit 17, and the second subject information acquisition unit 18, and calculates and outputs the presentation image V1 to be displayed on the display device 4. The configuration of the processing unit 16 will be described in detail later.

The data sharing unit 17 performs processing for the content presentation device 1 (content presentation device 1-1) to share control commands with another content presentation device 1 (content presentation device 1-2). The data sharing unit 17 transmits the control commands acquired by the control command acquisition unit 14 to the other content presentation device 1. The data sharing unit 17 receives control commands from the other content presentation device 1. As described above, operations on the content may be performed by operations from the first subject P1, or may be performed by receiving operations from the second subject P2 from the other content presentation device 1 (content presentation device 1-2).

The second subject information acquisition unit 18 includes a second subject video acquisition unit 181 and a second subject position/posture acquisition unit 182. The second subject video acquisition unit 181 acquires the second subject video PV2 from the other content presentation device 1 (content presentation device 1-2). The second subject video PV2 includes the sound of the second subject P2. The second subject video PV2 acquired by the second subject video acquisition unit 181 from the other content presentation device 1-2 is the second subject video PV2 cut out from an RGBD image captured by an RGBD capture unit (corresponding to the RGBD capture unit 31 shown in FIG. 3) provided on the side of the other content presentation device 1-2, and the sound of the second subject P2 acquired by a second subject audio acquisition unit (corresponding to the first subject audio acquisition unit 33 shown in FIG. 3) provided on the side of the other content presentation device 1-2. The second subject position/orientation acquisition unit 182 acquires second subject position/orientation information from another content presentation device 1 (content presentation device 1-2) of the second subject P2. The second subject position/orientation information is information that indicates the position and orientation of the second subject P2.

The storage unit 19 stores various information. The various information includes the content C1 and the shared reference information 192. The shared reference information 192 is information indicating the reference of the position and orientation at which the first object video PV1, the second object video PV2, and the content C1 are displayed in the presentation video V1. The shared reference information 192 includes the first object reference position information M1 and the second object reference position information M2, which will be described later. The shared reference information 192 also includes the relative positional relationship between the position in the three-dimensional space T1 where the content C1 is displayed, the position indicated by the first object reference position information M1, and the position indicated by the second object reference position information M2. The shared reference information 192 is shared between the content presentation device 1 (content presentation device 1-1) and another content presentation device 1 (content presentation device 1-2). The position indicated by the first object reference position information M1 and the position indicated by the second object reference position information M2 will be described later. The storage unit 19 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device.

In the present embodiment, an example in which the shared criteria information 192 is stored in advance in the storage unit 19 will be described, but the present invention is not limited to this. The content presentation device 1 may obtain the shared criteria information 192 from an external device other than the content presentation device 1 (for example, the server device 2).

The processing unit 16 includes a position setting unit 161 , a first object image extraction unit 166 , a background image acquisition unit 167 , a mask generation unit 168 , an occlusion reproduction unit 169 , and a presentation unit 1610 .
The position setting unit 161 sets the positions and orientations at which the first object video PV1, the second object video PV2, and the content C1 are displayed in the presentation video V1. The position setting unit 161 includes a reference position setting unit 162, a first object position setting unit 163, a second object position setting unit 164, and a content position setting unit 165.

The reference position setting unit 162 sets the first subject reference position information M1 and the second subject reference position information M2 in the three-dimensional space T1. The three-dimensional space T1 is a three-dimensional space for calculating the position and orientation of each element (the first subject video PV1, the second subject video PV2, and the content C1) to generate the presented video V1. The first subject reference position information M1 is information indicating the reference position and orientation for the first subject video PV1. The second subject reference position information M2 is information indicating the reference position and orientation for the second subject video PV2. The first subject reference position information M1 and the second subject reference position information M2 are, as an example, three points (also called markers) arranged in the three-dimensional space T1. The first subject reference position information M1 and the second subject reference position information M2 each indicate the position and orientation in the three-dimensional space T1 by the three markers. Here, the orientation includes two orientations: the orientation in the vertical direction and the orientation in the rotational direction that is perpendicular to the vertical direction. Since the orientation indicated by the first subject reference position information M1 includes the orientation in the vertical direction and the orientation in the rotational direction perpendicular to the vertical direction, the first subject P1 can check whether the orientation of the head mounted display H1-1 worn by the first subject P1 is tilted with respect to the three-dimensional space T1. For example, when the first subject P1 wearing the head mounted display H1-1 stands upright on the floor of the first space R1, the floor is tilted with respect to the horizontal plane in the three-dimensional space T1. When the orientation of the head mounted display H1-1 is tilted with respect to the three-dimensional space T1, the first subject P1 adjusts his/her own posture to adjust the orientation of the head mounted display H1-1.

The reference position setting unit 162 sets the second subject reference position information M2 in the three-dimensional space T1 based on a predetermined relative positional relationship between the position indicated by the second subject reference position information M2 and the position indicated by the first subject reference position information M1. The position indicated by the first subject reference position information M1 is, for example, the position of one of the three points indicated by the first subject reference position information M1. Similarly, the position indicated by the second subject reference position information M2 is, for example, the position of one of the three points indicated by the second subject reference position information M2. The predetermined relative positional relationship is included in the shared reference information 192. This predetermined relative positional relationship is a relationship based on the origin and coordinate system set in the three-dimensional space T1. When the origin and coordinate system are set in the three-dimensional space T1 and the position indicated by the first subject reference position information M1 is set, the position indicated by the second subject reference position information M2 is determined relative to the position indicated by the first subject reference position information M1. The position indicated by the second subject reference position information M2 is, for example, a position included between the position indicated by the first subject reference position information M1 and the position where the RGBD camera CM1 is installed.

In this embodiment, an example in which the first subject reference position information M1 and the second subject reference position information M2 each include three markers will be described, but the present invention is not limited to this. The first subject reference position information M1 and the second subject reference position information M2 may be other information as long as they indicate the position in the three-dimensional space T1, the orientation in the vertical direction, and the orientation in the rotational direction. For example, the first subject reference position information M1 and the second subject reference position information M2 may each be a three-dimensional local coordinate system. The local coordinate system is set for each point in the three-dimensional space T1, separate from the coordinate system set in the three-dimensional space T1.
As another example, each of the first subject reference position information M1 and the second subject reference position information M2 may include four or more markers.

The first subject position setting section 163 sets the position and orientation of the first subject video PV1 in the three-dimensional space T1 based on the first subject reference position information M1.
The second subject position setting section 164 sets the position and orientation of the second subject video PV2 in the three-dimensional space T1 based on the second subject reference position information M2.
The content position setting unit 165 sets the position in the three-dimensional space T1 where the content C1 is displayed based on a predetermined relative positional relationship between the position indicated by the first subject reference position information M1 and the position indicated by the second subject reference position information M2. This predetermined relative positional relationship is included in the shared reference information 192. For example, this predetermined relative positional relationship is a relative positional relationship in which the position indicated by the first subject reference position information M1, the position indicated by the second subject reference position information M2, and the position of the content C1 form a triangle having three sides of a predetermined length.

The first subject image extraction unit 166 extracts an image of a predetermined subject (such as a person) from the surrounding image E1. The image of the predetermined subject includes, together with the first subject image PV1, an image of the body of another person who is experiencing the same virtual reality content together in the same space (first space R1). The first subject image extraction unit 166 performs a process of recognizing the first subject P1 in the surrounding image E1 acquired by the surrounding image acquisition unit 131. The first subject image extraction unit 166 generates distance information (depth map) from the surrounding image E1, and identifies a location in the surrounding image E1 where the predetermined subject is reflected (information such as coordinates of an area in an image) based on the generated distance information and a preset depth. The first subject image extraction unit 166 supplies position information of the area in the surrounding image E1 to the mask generation unit 168 as a result of the recognition process.
As described above, the image of the predetermined object includes the first object image PV1, so the first object image extractor 166 extracts the first object image PV1 from the surrounding image E1.

The first subject video extraction unit 166 may recognize the first subject P1 based on machine learning. In this case, the first subject video extraction unit 166 has learned the characteristics of a specific subject in a video in advance by machine learning. The first subject video extraction unit 166 identifies a location in the peripheral video E1 where the specific subject is captured (information such as the coordinates of an area in an image) by referring to the learned model, and outputs the information.
Furthermore, the first subject image extraction unit 166 may select an algorithm for extraction depending on the type of subject to be extracted. For example, when another person who is experiencing the same virtual reality content together in the same space (first space R1) is present at a distance, the first subject image extraction unit 166 may use machine learning to extract an image of the other person and may use distance information to extract an image of a subject other than the other person.

The background image acquisition unit 167 acquires the background image BG1. The background image acquisition unit 167 cuts out an image (image) of the presentation area from the peripheral image E1 acquired by the peripheral image acquisition unit 131. The background image acquisition unit 167 acquires the cut-out image of the presentation area as the background image BG1. The image of the presentation area may be only a part of the entire peripheral image E1 (for example, a part near the center). This makes it possible to match the viewing angle of the peripheral image E1 acquired by the peripheral image acquisition unit 131 with the viewing angle of the presentation image V1 displayed on the display device 4. The background image acquisition unit 167 supplies the cut-out image (image) to the presentation unit 1610. Note that the processing of the background image acquisition unit 167 may include central projection processing.

The mask generation unit 168 generates mask information. The mask generation unit 168 generates mask information based on distance information and mask information based on the recognition result. The mask information based on distance information indicates whether or not the area is an area in which the presentation image should be presented based on the distance information supplied from the distance information acquisition unit 132. The mask information based on the recognition result indicates whether or not the area is an area in which the subject recognized by the first subject image extraction unit 166 (the body of the first subject P1, or a subject that exists in the first space R1 and shares content with the first subject P1) exists. In this embodiment, the mask information is information indicating whether the area is an area in which the first subject image PV1, the second subject image PV2, and the content C1 should be displayed, or an area in which the surrounding image E1 should be displayed.

As described above, the mask generating unit 168 generates a mask based on the distance information and a mask based on the recognition result. This allows the presenting unit 1610 to perform the following presentation. For example, the body of the user (i.e., the first subject P1), the body of another person experiencing the same virtual reality content together in the same space (first space R1), and the body of another person (second subject P2) experiencing the same virtual reality content together in another space (second space R2) can be presented in the virtual reality video. For a specific subject (person, etc.) recognized by the first subject video clipping unit 111, the peripheral video E1 can be displayed regardless of the distance from the device. For subjects other than the specific subject (person, etc.), presentation based on the distance information is performed. In other words, objects that are relatively close to the device are presented in the virtual reality space as included in the peripheral video E1. Also, objects that are relatively far from the device are not presented as included in the peripheral video E1. In such an area where there are objects that are relatively far from the device, the spherical image Z1 is presented in the background.

Here, the "relatively close range" is, for example, a range where a person can reach out and touch without moving from the spot (for example, while sitting). For example, it is a range of about 1 meter. Conversely, the "relatively far range" is, for example, a range of about 2 meters or more. In the intermediate distance range (1 meter or more and 2 meters or less), an image that combines the peripheral image E1 for a short distance and the omnidirectional image Z1 for a long distance may be presented.
The three-dimensional space T1 is made up of a short distance region TN1 and a long distance region TF1. The short distance region TN1 is the above-mentioned "relatively close range." The long distance region TF1 is the above-mentioned "relatively far range."

The occlusion reproduction unit 169 performs processing regarding occlusion. Occlusion refers to rendering an object in the foreground so as to hide an object in the background. The occlusion reproduction unit 169 determines whether the first object video PV1 or the second object video PV2 is in the foreground in the presented video V1, based on the position of the first object video PV1 in the three-dimensional space T1 and the position of the second object video PV2 in the three-dimensional space T1. The occlusion reproduction unit 169 supplies the determination result to the presentation unit 1610.

The presentation unit 1610 generates the presentation video V1. The presentation video V1 is an image in which the first object video PV1, the position and orientation of which are set in the three-dimensional space T1, the second object video PV2, the position and orientation of which are set in the three-dimensional space T1, and the content C1, the position of which is set in the three-dimensional space T1, are displayed in the three-dimensional space T1 against the background of the surrounding video E1. The presentation unit 1610 generates the presentation video V1 by reflecting the positional relationship between the first object video PV1, the second object video PV2, and the content C1 in the presentation video V1 based on the mask information generated by the mask generation unit 168 and the determination result by the occlusion reproduction unit 169. The process of generating the presentation video V1 is also referred to as rendering. The presentation unit 1610 outputs the generated presentation video V1 to the display device 4.

In the process of outputting the presentation video V1 to the display device 4, the presentation unit 1610 localizes and outputs the audio included in the second object video PV2 based on the position of the second object video PV2 in the three-dimensional space T1. In addition, in the process of outputting the presentation video V1 to the display device 4, the presentation unit 1610 localizes and outputs the audio included in the content C1 based on the position in the three-dimensional space T1 where the content C1 is displayed. Note that the presentation unit 1610 may output the audio included in the content C1 from a position other than the position in the three-dimensional space T1 where the content C1 is displayed.

The display device 4 displays the presentation image V1 output by the presentation unit 1610. The display device 4 is built into the head-mounted display H1. The display device 4 displays the presentation image V1 output from the presentation unit 1610 for each area on the screen. Note that the display device 4 may perform stereo display for stereoscopic vision, for example.

[Shooting processing]
Here, the shooting process according to this embodiment will be described. The shooting process is a process in which a subject is captured by the RGBD camera CM1, and the image of the subject is transmitted to another content presentation device 1 by the shooting unit 11. FIG. 5 is a diagram showing an example of the shooting process according to this embodiment. The shooting process shown in FIG. 5 corresponds to processing for one frame of the presentation image V1 presented by the content presentation device 1. Note that the shooting process shown in FIG. 5 will be described taking the first subject image PV1 as an example, but the shooting process for the second subject image PV2 is similar.

Step S10: The first subject image cropping unit 111 acquires the RGBD image captured by the RGBD imaging unit 31. The RGBD image is captured by the RGBD imaging unit 31 (RGBD camera CM1-1), is an image of the periphery of the RGBD camera CM1-1, and includes the first subject P1.

Step S20: The first object image cutout unit 111 cuts out the first object image PV1 from the RGBD image captured by the RGBD imaging unit 31. Here, the first object image cutout unit 111 cuts out the first object image PV1 from the RGBD image based on a preset depth. Note that the first object image cutout unit 111 may cut out the first object image PV1 from the RGBD image based on machine learning. The first object image cutout unit 111 supplies the cut out first object image PV1 to the first object image supply unit 112.
Step S30: The first object video supplying unit 112 transmits the first object video PV1 supplied from the first object video clipping unit 111 to the other content presentation device 1-2.

Step S40: The first subject image supply unit 112 acquires HMD position information from the first subject position measurement unit 32. The HMD position information indicates the position and attitude of the head mounted display H1-1 measured by the first subject position measurement unit 32 (the gyro sensor built into the head mounted display H1-1) as described above.
Step S50: The first subject video supply unit 112 acquires first subject audio information from the first subject audio acquisition unit 33. The first subject audio information indicates the audio of the first subject P1 acquired by the first subject audio acquisition unit 33 (the microphone built into the head mounted display H1-1) as described above.
Step S60: The first object video supply unit 112 transmits the acquired HMD position information and first object audio information to the other content presentation device 1-2.

Step S70: The photographing unit 11 determines whether or not the end condition of the photographing process is satisfied. The end condition of the photographing process is, for example, that the photographing unit 11 accepts an operation to end the photographing process, or that the power of the head mounted display H1-1 is turned off. If the photographing unit 11 determines that the end condition of the photographing process is satisfied (step S70; YES), the photographing unit 11 ends the photographing process. On the other hand, if the photographing unit 11 determines that the end condition of the photographing process is not satisfied (step S70; NO), the photographing unit 11 executes the process of step S10 again.
With this, the photographing unit 11 ends the photographing process.

In the shooting process, the processes from step S10 to step S20 until the first subject video PV1 is extracted, the process of acquiring HMD position information in step S40, and the process of acquiring first subject audio information in step S50 may be executed in parallel. Also, the process of transmitting the first subject video PV1 in step S30 and the process of transmitting the HMD position information and the first subject audio information in step S60 may be executed in parallel or at the same time. The first subject audio information may be included in the first subject video PV1 and transmitted before being transmitted to another content presentation device 1-2.

[Display Processing]
Next, the display process according to this embodiment will be described. The display process is a process for displaying the presented image V1 on the display device 4 built in the head mounted display H1. Fig. 6 is a diagram showing an example of the display process according to this embodiment. The display process shown in Fig. 6 corresponds to a process for one frame of the presented image V1 presented by the content presentation device 1. Note that the display process shown in Fig. 6 will be described taking the first object image PV1 as an example, but the same applies to the shooting process for the second object image PV2.

Step S110: The presentation unit 1610 causes the display device 4 to display the first subject reference position information M1 and the second subject reference position information M2 set in the three-dimensional space T1 by the reference position setting unit 162.

Step S120: The surrounding image acquisition unit 131 acquires the surrounding image E1 captured by the stereo camera SC1-1. The surrounding image E1 captured by the stereo camera SC1-1 is a set of an image captured by the left camera and an image captured by the right camera.
Step S130: The presentation unit 1610 displays the image of the presentation area cut out by the background image acquisition unit 167 on the display device 4. Here, the first object image PV1 is extracted from the surrounding image E1 by the first object image extraction unit 166 and displayed on the display device 4.

Step S140: The first subject position setting unit 163 sets the position and orientation of the first subject video PV1 in the three-dimensional space T1 based on the first subject reference position information M1. Here, the first subject position setting unit 163 adjusts the position of the first subject video PV1 in the three-dimensional space T1 by adjusting the position and orientation of the viewpoint in the presentation video V1 generated by rendering. Before the position of the first subject video PV1 is set (adjusted) based on the first subject reference position information M1, the position and orientation of the viewpoint in the presentation video V1 are set to a predetermined position and orientation in the initial display set based on the specifications of the head-mounted display H1-1, and the presentation video V1 is displayed. Therefore, the position and orientation of the first subject video PV1 in the three-dimensional space T1 are generally deviated from the position and orientation indicated by the first subject reference position information M1. The first subject position setting unit 163 matches the position and orientation of the viewpoint in the presentation video V1 to the position and orientation indicated by the first subject reference position information M1. As a result, the first subject position setting unit 163 matches the position and orientation of the first subject image PV1 in the three-dimensional space T1 with the position and orientation indicated by the first subject reference position information M1.

Step S150: The presentation unit 1610 displays the second subject video PV2 acquired by the second subject video acquisition unit 181 on the display device 4. Here, the presentation unit 1610 displays the second subject video PV2 at a transparency level less than a predetermined level. Here, the transparency level less than the predetermined level means a transparency level at which the first subject P1 does not feel that the second subject video PV2 is transparent. Note that the presentation unit 1610 may display the second subject video PV2 at a transparency level equal to or greater than a predetermined level.
The presentation unit 1610 displays the second object image PV2 as an actual image. Displaying as an actual image means displaying an image of the second object P2 captured by the RGBD camera CM1-2 using pixel values as is without compromising the texture.

The presentation unit 1610 displays the second object image PV2 based on the result of the determination by the occlusion reproduction unit 169 as to which of the second object image PV2 and an image corresponding to another object included in the presentation area is in the foreground. The other objects included in the presentation area include the first object image PV1 and the scenery captured in the surrounding image E1.

Step S160: The second subject position setting unit 164 sets the position and orientation of the second subject video PV2 in the three-dimensional space T1 based on the second subject reference position information M2. Here, the stereo camera SC1-2 that captures the second subject P2 outputs the image of the scenery captured from the position of the device as an RGBD image with the height and left/right positions of the device as the center of the captured image. Therefore, the second subject position setting unit 164 adjusts the position and orientation of the second subject video PV2 in the three-dimensional space T1 by adjusting the position and orientation of the virtual camera (projector) that projects the second subject video PV2 in the three-dimensional space T1. In the following description, the virtual camera that projects the second subject video PV2 in the three-dimensional space T1 is called the virtual camera VC1-1. Before the position of the second subject video PV2 is set (adjusted) based on the second subject reference position information M2, the position and orientation of the virtual camera VC1-1 are set to a predetermined position and orientation in the initial display that is set based on the specifications of the head mounted display H1-1. Therefore, the position and orientation of the second subject video PV2 in the three-dimensional space T1 generally deviates from the position and orientation indicated by the second subject reference position information M2. The second subject position setting unit 164 adjusts the position and orientation of the virtual camera VC1-1 in the three-dimensional space T1 to match the position and orientation of the second subject video PV2 in the three-dimensional space T1 with the position and orientation indicated by the second subject reference position information M2.

As described above, in steps S140 and S160, the positions and orientations of the first and second object images PV1 and PV2 in the three-dimensional space T1 are set based on the first and second object reference position information M1 and M2, respectively. Once the positions and orientations of the first and second object images PV1 and PV2 in the three-dimensional space T1 are set, in subsequent frames, the changes in the positions and orientations of the first and second object images PV1 and PV2 in the three-dimensional space T1 are relative changes from the positions and orientations set based on the first and second object reference position information M1 and M2, respectively. In the content presentation system S, the first and second object reference position information M1 and the relative positional relationship between the position indicated by the first and second object reference position information M1 and the position indicated by the second object reference position information M2 are shared between multiple content presentation devices 1. Therefore, the relative changes from the position and orientation set based on the first subject reference position information M1 and the second subject reference position information M2 in the subsequent frames are shared between multiple content presentation devices 1.

Step S170: The presentation unit 1610 hides the first subject reference position information M1 and the second subject reference position information M2 displayed on the display device 4.

Step S180: The presentation unit 1610 localizes and outputs (plays) the audio included in the second subject video PV2 based on the position of the second subject video PV2 in the three-dimensional space T1 based on the HMD position information received from the other content presentation device 1-2. The HMD position information received from the other content presentation device 1-2 indicates the position of the head-mounted display H1-2 attached to the second subject P2. The audio included in the second subject video PV2 is included in the second subject video PV2 as second subject audio information. Note that the presentation unit 1610 may output the audio included in the second subject video PV2 from a position other than the position in the three-dimensional space T1 of the second subject video PV2.

Step S190: The presentation unit 1610 determines whether or not a control command has been input to an operation unit (not shown). The operation units refer to both the operation unit on the first subject video PV1 side and the operation unit on the second subject video PV2 side. The control command input to the operation unit on the first subject video PV1 side is acquired from the control command acquisition unit 14. The control command input to the operation unit on the second subject video PV2 side is acquired by the data sharing unit 17 from another content presentation device 1-2. If the control command acquisition unit 14 determines that a control command has been input (step S190; YES), it executes the process of step S200. On the other hand, if the control command acquisition unit 14 determines that a control command has not been input (step S190; NO), it executes the process of step S220.

Step S200: The data sharing unit 17 transmits the control command acquired by the control command acquisition unit 14 to the other content presentation device 1-2.
Step S210: The control command acquisition unit 14 updates the internal state of the content C1 in accordance with the acquired control command.

Step S220: The mask generating unit 168 generates mask information for sharing the same space. The mask information for sharing the same space indicates whether or not a region includes a subject present in the first space R1 (the body of the first subject P1, or a subject present in the first space R1 and sharing content with the first subject P1).
Step S230: The mask generating unit 168 adds, to the same space sharing mask information generated in step S220, information indicating whether or not the area includes the second object image PV2.

Step S240: The mask generating unit 168 adds information indicating whether or not the area includes the content C1 according to the internal state of the content to the mask information for sharing the same space.
Step S250: The presentation unit 1610 adds the content C1 to the presentation image V1 based on the same space sharing mask information. Here, the presentation unit 1610 adds the video and audio included in the content C1 to the presentation image V1.

Step S260: The presentation unit 1610 outputs the generated presentation image V1 to the display device 4 for display. Here, the presentation unit 1610 generates the presentation image V1 using a video see-through method. The display device 4 displays the presentation image V1 using the video see-through method. Note that the display device 4 may display the presentation image V1 using an optical see-through method.

Step S270: The processing unit 16 determines whether or not the end condition of the display process is satisfied. The end condition of the display process is, for example, that the power supply of the head mounted display H1-1 is turned off. When the processing unit 16 determines that the end condition of the display process is satisfied (step S270; YES), the processing unit 16 ends the display process. On the other hand, when the processing unit 16 determines that the end condition of the display process is not satisfied (step S270; NO), the processing unit 16 returns to step S120 and executes the subsequent processes again. Here, in the second and subsequent display processes, the processes of steps S110, S140, S160, and S170 are omitted.
This is how the display unit 12 ends the display process.

Note that, in the display process shown in FIG. 6, an example has been described in which the display position of the second object image PV2 is set and then the display position of the content C1 is set, but this is not limited thereto. The display position of the second object image PV2 may be set after the display position of the content C1 is set.

[Position Settings]
Here, referring to FIG. 7 and FIG. 8, the process in which the position setting unit 161 sets the positions of the first object image PV1, the second object image PV2, and the content C1 in the three-dimensional space T1 will be described. FIG. 7 is a diagram showing an example of the position setting process according to this embodiment. FIG. 7(A) shows the positional relationship in the three-dimensional space T1-1 on the side of the first object P1. FIG. 7(B) shows the positional relationship in the three-dimensional space T1-2 on the side of the second object P2. An origin O1 is set in advance in the three-dimensional space T1-1 and the three-dimensional space T1-2. Note that, although FIG. 7(A) shows the entire body of the first object P1, in the presented image V1 displayed on the head-mounted display H1-1, only the part of the body of the first object P1 captured by the stereo camera SC1-1 is displayed based on the video see-through. Similarly, while Figure 7 (B) shows the entire body of the second subject P2, in the presented image V1 displayed on the head-mounted display H1-2, only the part of the body of the second subject P2 captured by the stereo camera SC1-2 is displayed based on video see-through.

The set of the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13 is an example of the first subject reference position information M1. The first subject first marker M11, the first subject second marker M12, and the first subject third marker M13 are set at a predetermined relative position with respect to the origin O1. The distance from the origin O1 and the direction seen from the origin O1 of each of the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13 are determined in advance. In other words, the values of each coordinate when the positions of the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13 are represented by a three-dimensional coordinate system set in advance in the three-dimensional space T1-1 are determined in advance.

The set of the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 is an example of the second subject reference position information M2. The second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are set at a predetermined relative position with respect to the origin O1. The distance from the origin O1 and the direction seen from the origin O1 of each of the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are determined in advance. In other words, the value of each coordinate when the position of each of the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 is represented by a three-dimensional coordinate system set in advance in the three-dimensional space T1-1 is determined in advance.

As described above, the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13 are set at a predetermined relative position to the origin O1, and therefore the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are set at a predetermined relative position to the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13.

The first subject video PV1 is displayed, for example, so that the head, waist, and predetermined positions in the direction of the front of the first subject P1 in the first subject video PV1 correspond to the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13, respectively. However, in the presented video V1 displayed on the head-mounted display H1-1, only the part of the body of the first subject P1 that is captured by the stereo camera SC1-1 is displayed as the first subject video PV1 based on the video see-through.

The second subject video PV2 is projected and displayed by the virtual camera VC1-1 in the three-dimensional space T1-1 as described above. The second subject video PV2 is displayed so that, for example, the head, waist, and predetermined positions in the direction of the front of the second subject P2 in the second subject video PV2 correspond to the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23, respectively. The sound included in the second subject video PV2 as the second subject sound information is localized and output at the second subject sound position PP2. The second subject sound position PP2 is set based on the head position of the second subject video PV2 (the position of the second subject first marker M21). The second subject sound position PP2 is set, for example, to the head position of the second subject video PV2. Alternatively, the second subject sound position PP2 may be set to a position that is a predetermined distance away in the height direction from the head position of the second subject video PV2. The second subject sound position PP2 is set based on the head position in the second subject video PV2, and then moves in conjunction with the movement of this head position.

7(B), the first subject first marker M11, the first subject second marker M12, the first subject third marker M13, the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are set at the same positions as those set in the three-dimensional space T1-1 shown in Fig. 7(A). The distance from the origin O1 and the direction as viewed from the origin O1 of each of the first subject first marker M11, the first subject second marker M12, the first subject third marker M13, the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are the same between the three-dimensional space T1-1 and the three-dimensional space T1-2. In other words, the coordinate values of the first subject first marker M11, the first subject second marker M12, the first subject third marker M13, the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23 are the same between the three-dimensional space T1-1 and the three-dimensional space T1-2. The first subject sound position PP1 is set, for example, at the position of the head in the first subject video PV1. Alternatively, the first subject sound position PP1 may be set at a position a predetermined distance away in the height direction from the position of the head in the first subject video PV1. The first subject sound position PP1 is set based on the position of the head in the first subject video PV1, and then moves in conjunction with the movement of the position of the head.

The first subject video PV1 is projected and displayed by the virtual camera VC1-2 in the three-dimensional space T1-2. In the example shown in FIG. 7, the head, waist, and predetermined positions in the front direction of the first subject P1 of the first subject video PV1 are displayed to match the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13, respectively, and the head, waist, and predetermined positions in the front direction of the second subject P2 of the second subject video PV2 are displayed to match the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23, respectively, but this is not limited to this. Of the parts included in the first subject video PV1, three predetermined parts according to the position indicated by the first subject reference position information M1 may be displayed to match the three positions indicated by the first subject reference position information M1, respectively. For example, the positions of the shoulders, arms, and legs of the first subject video PV1 may be used for alignment to the position indicated by the first subject reference position information M1 of the first subject video PV1, depending on the respective positions of the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13. Also, depending on the position indicated by the first subject reference position information M1, a predetermined position in a predetermined direction (besides the front direction, the back direction, the left and right directions, etc.) around the first subject P1 may be used for alignment to the position indicated by the first subject reference position information M1 of the first subject video PV1.
Similarly, with regard to aligning the second subject video PV2 to the position indicated by the second subject reference position information M2, three specific parts of the second subject video PV2 corresponding to the position indicated by the second subject reference position information M2 may be displayed so as to coincide with the positions indicated by the second subject reference position information M2.

Here, the setting of the positions of the first subject image PV1 and the second subject image PV2 will be explained using images displayed on the head mounted display H1-1. FIG. 8 is a diagram showing an example of an image displayed on the head mounted display H1-1 to set the position of the subject according to this embodiment. In the example shown in FIG. 8, the first subject P1 is sitting on a chair OB1 in the first space R1 with the head mounted display H1-1 attached to his head. The second subject P2 is sitting on a chair in the second space R2. In the example shown in FIG. 8, the second subject image PV2 includes an image of the chair OB2 on which the second subject P2 is sitting, as well as an image of the second subject P2.

Figure 8 (A) shows a state in which the position of the second subject video PV2 is misaligned with respect to the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23. In Figure 8 (A), the position of the waist of the second subject P2 in the second subject video PV2 (i.e., the position of the seat of the chair OB2) is misaligned with the position of the second subject second marker M22. Meanwhile, the position of the seat of the chair OB1 on which the first subject P1 is sitting matches the position of the first subject second marker M12, and the position of the first subject P1 in the first subject video PV1 is not misaligned with respect to the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13.

The position of the second subject video PV2 is adjusted in the up-down, front-back, and left-right directions. The position of the feet of the second subject P2 included in the second subject video PV2 is adjusted in the up-down direction so as to match the position of the bottom surface of the three-dimensional space T1. In other words, the second subject position setting unit 164 sets the position of the second subject video PV2 in the three-dimensional space T1 by matching the position of the feet of the second subject P2 included in the second subject video PV2 to the position of the bottom surface of the three-dimensional space T1 based on the second subject reference position information M2. Note that the second subject position setting unit 164 may set the position of the second subject video PV2 in the three-dimensional space T1 by matching the position of the feet of the second subject P2 to a position other than the position of the bottom surface of the three-dimensional space T1.

When the position of the second subject video PV2 is adjusted relative to the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23, the state shown in FIG. 8(C) is obtained. In FIG. 8(C), the position of the waist of the second subject P2 in the second subject video PV2 (i.e., the position of the seat of the chair OB2) coincides with the position of the second subject second marker M22.

On the other hand, FIG. 8(B) shows a state in which the position of the first subject P1 is misaligned with respect to the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13. In FIG. 8(B), the position of the seat of the chair OB1 on which the first subject P1 is sitting is misaligned with the position of the first subject second marker M12. On the other hand, the position of the waist of the second subject P2 in the second subject video PV2 (i.e., the position of the seat of the chair OB2) is consistent with the position of the second subject second marker M22, and the position of the second subject video PV2 is not misaligned with respect to the second subject first marker M21, the second subject second marker M22, and the second subject third marker M23.

The position of the first object image PV1 is adjusted in the front-back and left-right directions. In the image displayed on the head mounted display H1-1, the bottom surface of the three-dimensional space T1 is pre-adjusted based on the up-down position of the head mounted display H1-1. If the up-down position and front direction of the head mounted display H1-1 have been pre-adjusted to match the bottom surface position and horizontal position of the three-dimensional space T1, respectively, then the up-down position of the first object image PV1 is not adjusted. If the up-down position and front direction of the head mounted display H1-1 have not been pre-adjusted to match the bottom surface position and horizontal position of the three-dimensional space T1, respectively, then the up-down position of the first object image PV1 is adjusted.

When the position of the first subject image PV1 is adjusted relative to the first subject first marker M11, the first subject second marker M12, and the first subject third marker M13, the state shown in Fig. 8C is obtained. In Fig. 8C, the position of the seat of the chair OB1 on which the first subject P1 is sitting coincides with the position of the first subject second marker M12.
When the adjustment of the position of the first object image PV1 or the second object image PV2 is completed, each marker is made non-displayable.

The second subject video PV2 is an image obtained by capturing the second subject P2 from one direction by the RGBD camera CM1-2. The vertical position of the second subject P2 in the second subject video PV2 is adjusted by the shooting unit 11. This is because it is preferable that the vertical position (height) of the RGBD camera CM1-2 is at a vertical position (height) approximately the same as the line of sight of the second subject P2.

The position of the second subject P2 in the second subject video PV2 in the direction perpendicular to the up-down direction and the orientation in that direction (rotation direction) may be adjusted by the position of the RGBD camera CM1-2, or by the display unit 12, or may be adjusted by combining the adjustment by the position of the RGBD camera CM1-2 and the adjustment by the display unit 12. In this embodiment, if the orientation of the second subject P2 in the second subject video PV2 in the direction perpendicular to the up-down direction is not appropriate, as an example, the position of the RGBD camera CM1-2 that captures the second subject P2 in the direction perpendicular to the up-down direction and the orientation in that direction (rotation direction) are adjusted. If the position of the second subject P2 in the second subject video PV2 in the direction perpendicular to the up-down direction and the orientation in that direction (rotation direction) are adjusted by the position of the RGBD camera CM1-2, the position or orientation of the second subject video PV2 in the capture range of the RGBD camera CM1-2 can be kept constant.

In addition, instead of adjusting the orientation of the RGBD camera CM1-2 as described above, the display unit 12 may change the position of the second subject P2 in the direction perpendicular to the up-down direction in the second subject video PV2 and the orientation (rotation direction) in that direction. In this case, the second subject position setting unit 164 provided in the display unit 12 changes the position of the second subject P2 in the direction perpendicular to the up-down direction in the second subject video PV2 and the orientation (rotation direction) in that direction based on image processing. The second subject position setting unit 164 may select an appropriate image from multiple images of the second subject P2 captured in advance from multiple directions. Alternatively, the second subject position setting unit 164 may change the orientation of the second subject P2 in the second subject video PV2 by generating an image of the second subject P2 captured from another direction based on image processing using machine learning or the like.

Returning to FIG. 7, the description of the position setting will be continued.
The position in the three-dimensional space T1 where the content C1 is displayed is set to a predetermined relative position with respect to the origin O1. Since the position indicated by the first subject reference position information M1 and the position indicated by the second subject reference position information M2 are set to predetermined relative positions with respect to the origin O1, the position where the content C1 is displayed is set to a predetermined relative position with respect to the position indicated by the first subject reference position information M1 and the position indicated by the second subject reference position information M2.
The sound contained in the content C1 is localized at a content sound position PC1 and played back. The content sound position PC1 is set based on the position where the content C1 is displayed. The content sound position PC1 is set, for example, at the position where the content C1 is displayed. The content sound position PC1 may be set at a position a predetermined distance away in the height direction from the position where the content C1 is displayed. After being set based on the position where the content C1 is displayed, the content sound position PC1 moves in conjunction with the movement of the position where the content C1 is displayed.

Here, the relationship between the position where the content C1 is displayed, the position where the first object video PV1 is displayed, and the position where the second object video PV2 is displayed will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the relative positional relationship between the content C1, the first object video PV1, and the second object video PV2 according to this embodiment.
9A shows the positional relationship in the three-dimensional space T1-1 on the side of the first subject P1. In FIG. 9A, positions A1-1, A2-1, and A3-1 respectively indicate the position of the first subject P1, the position where the second subject video PV2 is displayed, and the position where the content C1 is displayed. Position A2-1 is set at a position between position A1-1 and the position where the RGBD camera CM1-1 is installed. Alternatively, the RGBD camera CM1-1 may be installed at a position a predetermined distance away from position A1-1 in the direction toward position A2-1. Position A3-1 is set so that positions A1-1, A2-1, and A3-1 correspond to the vertices of a triangle having three sides of a predetermined length.

Figure 9 (B) shows the positional relationship in the three-dimensional space T1-2 on the side of the second subject P2. In Figure 9 (B), positions A1-2, A2-2, and A3-2 respectively indicate the position of the first subject video PV1, the position of the second subject P2, and the position where the content C1 is displayed. Position A1-2 is set at a position between position A2-2 and the position where the RGBD camera CM1-2 is installed. Alternatively, the RGBD camera CM1-2 may be installed at a position a predetermined distance away from position A2-2 in the direction toward position A1-2. Furthermore, position A3-2 is set so that positions A1-2, A2-2, and A3-2 correspond to the vertices of a triangle having three sides of predetermined lengths.

Here, in the three-dimensional space T1-1, a triangle with three sides of a predetermined length formed by positions A1-1, A2-1, and A3-1 is congruent with a triangle with three sides of a predetermined length formed by positions A1-2, A2-2, and A3-2 in the three-dimensional space T1-2. In other words, in the content presentation device 1, the relative positional relationship between the position in the three-dimensional space T1 where the content C1 is displayed and the position indicated by the first subject reference position information M1 and the position indicated by the second subject reference position information M2 is shared with other content presentation devices 1.

[Drawing AR or VR objects]
The rules for drawing an AR or VR object in the presented image V1 will be described. Fig. 10 is a diagram showing an example of drawing an AR or VR object according to the present embodiment. As described above, the three-dimensional space T1 is composed of a close distance area TN1 and a long distance area TF1 that are distinguished from each other according to the distance from the first subject P1.

The presented image V1 is made up of a background image BG1 and an object image B1. The background image BG1 is made up of a surrounding image E1 and a celestial sphere image Z1. The surrounding image E1 is used as a background image in the near distance area TN1. The surrounding image E1 includes images of various objects existing in the first space R1 in which the first subject P1 exists, as well as images of the floor or ground, for example, in the vicinity of the first subject P1. In FIG. 10, an image E10, which is an image of a pillar of a house, is shown as an image of an object included in the surrounding image E1. The celestial sphere image Z1 is a VR object, and is used as a background image in the far distance area TF1.

The images included in the object image B1 include a first subject image PV1, a second subject image PV2, content C1, and a third subject image PV3. The third subject image PV3 is an image of a person who exists in the long distance area TF1 among people who exist in the first space R1. The second subject image PV2 and content C1 are AR objects.

In this embodiment, a video of the scenery in the first space R1 of the first subject P1 captured by the stereo camera SC1 is displayed in the near field TN1. A celestial sphere video Z1, which is a VR video, may be displayed as the backmost background in the far field TF1. Note that the celestial sphere video Z1 does not necessarily have to be displayed in the far field TF1.

In the following description, the background image BG1 and the object image B1 other than the spherical image Z1 will be referred to as group A objects, and the spherical image Z1 will be referred to as group B objects.
In the presented image V1, the A group object is displayed in front of the group objects. The A group objects are displayed with occlusion reproduced based on the distance from the first subject P1. The A group object may be displayed in the close distance area TN1, the long distance area TF1, or on the boundary between the close distance area TN1 and the long distance area TF1. Here, the images of various objects existing in the first space R1 and the third subject image PV3 are displayed according to their positions since their positions in the three-dimensional space T1 are determined.

[Recreating occlusion]
Next, reproduction of occlusion will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram showing an example of reproduction of occlusion for the first object video PV1 and the second object video PV2 according to this embodiment. In Fig. 11(A), an image of the hand of the first object P1 is shown as the first object video PV1. When the position of the hand of the first object P1 is closer than the position of the second object video PV2, the part of the presented video V1 where the hand overlaps with the second object video PV2 is displayed closer than the second object video PV2. That is, in this case, the first object video PV1 is displayed in a manner in which the hand is visible.

On the other hand, in FIG. 11(B), the position of the hand of the first subject P1 is further back than the position of the second subject image PV2. In this case, the part of the hand in the presented image V1 that overlaps with the second subject image PV2 is displayed hidden by the second subject image PV2. In other words, in this case, the first subject image PV1 is displayed in such a way that a part of the hand is not visible.

Figure 12 is a diagram showing an example of reproducing occlusion for the second subject video PV2 and content according to this embodiment. In Figure 12, content C3 (an AR object of a trainer doing exercises) is displayed as the content. Because the second subject video PV2 is in front of content C3, content C3 is displayed in such a way that part of content C3 (the legs) is hidden by the chair included in the second subject video PV2.

[Content example]
Next, various examples of content combining AR and VR presented by the content presentation device 1 will be described with reference to Fig. 13 to Fig. 15. In Fig. 13, content C13 is displayed as the content. Content C13 is an AR object on a television screen. In Fig. 13, the television screen is displayed in the air. In Fig. 14, content C14 is displayed as the content. Content C14 is an AR object of a trainer performing exercises. Content C14 is a three-dimensional image of the trainer.
13 and 14, the celestial sphere video Z1, which is a group B object, is not displayed, and the peripheral video E1 is displayed as a background in the far distance area TF1. In Fig. 13 and 14, the first subject video PV1, the second subject video PV2, and the content, which are group A objects, are displayed with occlusion reproduced according to their positions in the three-dimensional space T1.

In FIG. 15, a celestial sphere image Z15 is displayed as the background in the far distance area TF1. The celestial sphere image Z15 is a view of an aquarium. The first subject image PV1 and the second subject image PV2, which are group A objects, are displayed in front of the celestial sphere image Z15, which is a group B object. The first subject image PV1 and the second subject image PV2, which are group A objects, are displayed with occlusion reproduced according to their positions in the three-dimensional space T1.

As described above, the content presentation device 1 according to the present embodiment includes the reference position setting unit 162, the surrounding image acquisition unit 131, the first object image extraction unit 166, the background image acquisition unit 167, the second object image acquisition unit 181, the content acquisition unit 15, the first object position setting unit 163, the second object position setting unit 164, the content position setting unit 165, and the presentation unit 1610. The reference position setting unit 162 sets, in the three-dimensional space T1, the first object reference position information M1, which is information indicating a reference position and orientation for the first object image PV1, which is an image including an image of the first object P1, and the second object reference position information M2, which is information indicating a reference position and orientation for the second object image PV2, which is an image including an image of the second object P2. The surrounding image acquisition unit 131 acquires the surrounding image E1, which is an image of the surroundings of the device including the first object P1. The first object image extraction unit 166 extracts the first object image PV1 from the surrounding image E1. The background image acquisition unit 167 acquires the background image BG1, which is an image used as the background. The second object image acquisition unit 181 acquires the second object image PV2 from another content presentation device 1-2. The content acquisition unit 15 acquires the content C1. The first object position setting unit 163 sets the position and orientation of the first object image PV1 in the three-dimensional space T1 based on the first object reference position information M1. The second object position setting unit 164 sets the position and orientation of the second object image PV2 in the three-dimensional space T1 based on the second object reference position information M2. The content position setting unit 165 sets the position in the three-dimensional space T1 where the content C1 is displayed based on a predetermined relative positional relationship between the position indicated by the first object reference position information M1 and the position indicated by the second object reference position information M2. The presentation unit 1610 outputs a presentation video V1 in which a first subject video PV1, the position and orientation of which are set in the three-dimensional space T1, a second subject video PV2, the position and orientation of which are set in the three-dimensional space T1, and a content C1, the position of which is set in the three-dimensional space T1, are displayed in the three-dimensional space T1 against a background of a background video BG1. In the content presentation device 1, the relative positional relationship between the position in the three-dimensional space T1 where the content C1 is displayed, the position indicated by the first subject reference position information M1, and the position indicated by the second subject reference position information M2, the first subject reference position information M1, and the second subject reference position information M2 are each shared with another content presentation device 1-2.

With this configuration, the content presentation device 1 according to this embodiment allows people in different spaces to share each other's positional relationships, including the position of the content, without any inconsistencies, providing the sensation of experiencing content (AR content, VR content, or AR/VR mixed content that is a mixture of AR content and VR content) with people in different spaces while sharing the same space.

It is expected that future broadcast content will be displayed by seamlessly switching between AR and VR. A system that makes this possible and can display real space and virtual objects by compositing them with the same quality is required. In such a system, it is necessary to specifically design which position each user will use as the reference point to develop the space and how to share different spaces without contradiction.
In the content presentation device 1, a user can experience viewing AR/VR mixed content while engaging in non-verbal communication not only with people in the same area but also with people who are not physically in the same place, without any contradiction in the relative positional relationship between the user, a distant person, and the content. AR/VR mixed content is envisioned as future broadcast content. In the content presentation device 1, a user can view AR content, VR content, or AR/VR mixed content together with family and friends with a sufficient viewing angle while checking their status via a local area and a network, and an environment can be provided in which real space and virtual space can be mixed and simulated with the same level of quality.

Furthermore, in the content presentation device 1 according to this embodiment, the presentation unit 1610 localizes and outputs the sound included in the content C1 based on the position in the three-dimensional space T1 where the content C1 is displayed.
With this configuration, in the content presentation device 1 of this embodiment, the sound contained in the content C1 can be heard from a position within the three-dimensional space T1 in which the content C1 is displayed, thereby increasing the sense of realism when viewing the content C1.

Furthermore, in the content presentation device 1 according to this embodiment, the presentation unit 1610 localizes and outputs the sound included in the second object video PV2 based on the position of the second object video PV2 in the three-dimensional space T1.
With this configuration, in the content presentation device 1 of this embodiment, the sound of the second subject P2 is heard from a position within the three-dimensional space T1 of the second subject image PV2, making it feel as if a person in a different space is making a sound next to you.

Moreover, the content presentation device 1 according to this embodiment further includes an occlusion reproduction unit 169. The occlusion reproduction unit 169 determines which of the first object video PV1 and the second object video PV2 is in the foreground in the presentation video V1, based on the position of the first object video PV1 in the three-dimensional space T1 and the position of the second object video PV2 in the three-dimensional space T1. The presentation unit 1610 outputs the presentation video V1 based on the determination result of the occlusion reproduction unit 169.
With this configuration, the content presentation device 1 according to this embodiment allows people in different spaces to share each other's front-to-back positional relationships without any inconsistencies, making it feel as if people in different spaces are next to each other more naturally than when there is an inconsistency in the front-to-back positional relationships.

Furthermore, in the content presentation device 1 according to this embodiment, the second subject position setting unit 164 sets the position of the second subject P2 included in the second subject video PV2 within the three-dimensional space T1 by matching the position of the feet of the second subject P2 included in the second subject video PV2 with the position of the bottom surface of the three-dimensional space T1 based on the second subject reference position information M2.
With this configuration, the content presentation device 1 of this embodiment allows people in different spaces to share each other's positional relationships in the height (up and down) direction without any inconsistencies, making it feel as if people in different spaces are next to each other more naturally than when there is an inconsistency in the positional relationships in the height (up and down) direction.

Furthermore, in the content presentation device 1 according to this embodiment, the presentation section 1610 displays the second object image PV2 at a transparency level lower than a predetermined level.
With this configuration, the content presentation device 1 of this embodiment allows people in different spaces to see each other's appearances closer to their actual appearances than when they are displayed as transparent images, making it possible to feel as if the appearance of a person in a different space is closer to their actual appearance as if that person is present next to them.

Moreover, in the content presentation device 1 according to this embodiment, the presentation unit 1610 generates the presentation image V1 using a video see-through method.
With this configuration, in the content presentation device 1 according to the present embodiment, one's own body is displayed in a natural manner in the presentation image V1, so that the sensation of sharing the same space with a person in a different space can be made more natural. A natural manner means that the sense of distance, texture, responsiveness (delay), etc. are close to the actual ones.

The data sharing unit 17 may have the function of a synchronization unit. The synchronization unit synchronizes the playback position in the time direction of the content C1 or the spherical image Z1 in the presentation image V1 output by the presentation unit 1610 with another content presentation device 1-2. The content presentation device 1-1 and the other content presentation device 1-2 can exchange information at any time through communication.

In the present embodiment, an example in which the first subject image cutout unit 111 is provided in the imaging unit 11 has been described, but the present invention is not limited to this. The first subject image cutout unit 111 may be provided in the display unit 12. In this case, the first subject image cutout unit 111 is provided in the display unit 12 as a second subject image cutout unit. Here, a case in which the first subject image cutout unit 111 is provided in the display unit 12 as a second subject image cutout unit will be described. The subject image supply unit provided in the imaging unit 11 supplies the RGBD image of the second subject P2 captured by the RGBD imaging unit 31 to the other content presentation device 1. The second subject image acquisition unit 181 acquires the RGBD image provided from the other content presentation device 1. The second subject image cutout unit provided in the display unit 12 cuts out the second subject image PV2 from the RGBD image acquired by the second subject image acquisition unit 181.

In this embodiment, as an example, a situation in which two content presentation devices 1 are operating simultaneously is shown, but the number of content presentation devices 1 that operate in conjunction with each other is arbitrary. Depending on the number of content presentation devices 1 that operate in conjunction with each other, multiple subjects can view the same content while sharing their mutual positional relationships, including the position of the content, without any contradictions.

At least a part of the functions of the content presentation device in each of the above-mentioned embodiments can be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that temporarily and dynamically hold a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may further be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the present invention.

The present invention can be used, for example, in devices for presenting video content, services for presenting video content, etc. However, the scope of use of the present invention is not limited to the examples given here.

1, 1-1, 1-2... Content presentation device, 162... Reference position setting unit, 131... Surrounding image acquisition unit, 166... First subject image extraction unit, 167... Background image acquisition unit, 181... Second subject image Acquisition unit, 15...Content acquisition unit, 163...First subject position setting unit, 164...Second subject position setting unit, 1 65...Content position setting section, 1610...Presentation section, P1...First subject, P2...Second subject, PV1...First subject video, PV2...Second subject video, C1...Content, M1...First subject reference position information , M2...second subject reference position information, T1...three-dimensional space, E1...surrounding image, BG1...background image

Claims (7)

a reference position setting unit that sets, in a three-dimensional space, first subject reference position information, which is information indicating a reference position and orientation for a first subject video, which is a video including an image of a first subject, and second subject reference position information, which is information indicating a reference position and orientation for a second subject video, which is a video including an image of a second subject;
A surrounding image acquisition unit that acquires a surrounding image that is an image of the surroundings of the own device including the first subject;
a first object image extraction unit that extracts the first object image from the surrounding image;
a background image acquisition unit that acquires a background image that is an image to be used as a background;
a second object image acquisition unit that acquires the second object image from another content presentation device;
a content acquisition unit that acquires content;
a first object position setting unit that sets a position and a direction of the first object image in the three-dimensional space based on the first object reference position information;
a second object position setting unit that sets a position and a direction of the second object image in the three-dimensional space based on the second object reference position information;
a content position setting unit that sets a position in the three-dimensional space where the content is to be displayed based on a predetermined relative positional relationship between a position indicated by the first subject reference position information and a position indicated by the second subject reference position information;
a presentation unit that outputs a presentation image in which the first object image, the position and orientation of which are set in the three-dimensional space, the second object image, the position and orientation of which are set in the three-dimensional space, and the content, the position of which is set in the three-dimensional space, are displayed in the three-dimensional space against the background of the background image;
Equipped with
a relative positional relationship between a position in the three-dimensional space where the content is displayed, a position indicated by the first subject reference position information, and a position indicated by the second subject reference position information, the first subject reference position information, and the second subject reference position information are each shared with the other content presentation device ;
The presentation unit generates the presentation image using a video see-through method.
Content presentation device. The content presentation device according to claim 1 , wherein the presentation unit outputs the sound included in the content by localizing the sound based on a position in the three-dimensional space where the content is displayed. The content presentation device according to claim 1 , wherein the presentation section is configured to output the sound included in the second object video by localizing the sound based on a position of the second object video in the three-dimensional space. An occlusion reproduction unit that determines which of the first object image and the second object image is in front in the presentation image based on a position of the first object image in the three-dimensional space and a position of the second object image in the three-dimensional space,
The content presentation device according to claim 1 , wherein the presentation unit outputs the presentation video based on a determination result of the occlusion reproduction unit. 5. The content presentation device according to claim 1, wherein the second subject position setting unit sets the position of the second subject image in the three-dimensional space by matching the position of the foot of the second subject included in the second subject image with the position of the bottom surface of the three-dimensional space based on the second subject reference position information. The content presentation device according to claim 1 , wherein the presentation section displays the second object image at a transparency lower than a predetermined value. a reference position setting process for setting, in a three-dimensional space, first subject reference position information, which is information indicating a reference position and orientation for a first subject image, which is an image including an image of a first subject, and second subject reference position information, which is information indicating a reference position and orientation for a second subject image, which is an image including an image of a second subject;
a surrounding image acquisition step of acquiring a surrounding image that is an image of the surroundings of the own device including the first object;
extracting the first object image from the surrounding image;
a background image acquisition step of acquiring a background image to be used as a background;
a second object image acquiring step of acquiring the second object image from another content presentation device;
a content acquisition process for acquiring content;
a first object position setting step of setting a position and a direction of the first object image in the three-dimensional space based on the first object reference position information;
a second object position setting step of setting a position and a direction of the second object image in the three-dimensional space based on the second object reference position information;
a content position setting process for setting a position in the three-dimensional space where the content is to be displayed based on a predetermined relative positional relationship between a position indicated by the first object reference position information and a position indicated by the second object reference position information;
a presentation process of outputting a presentation image in which the first object image, the position and orientation of which are set in the three-dimensional space, the second object image, the position and orientation of which are set in the three-dimensional space, and the content, the position of which is set in the three-dimensional space, are displayed in the three-dimensional space against the background of the background image;
A program for causing a computer to execute the process of
a relative positional relationship between a position in the three-dimensional space where the content is displayed, a position indicated by the first subject reference position information, and a position indicated by the second subject reference position information, the first subject reference position information, and the second subject reference position information are each shared with the other computer ,
The presenting step generates the presented image using a video see-through method.
program.

Images