JP7549062B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Conventionally, systems that control three-dimensional models in response to the movements of people are known. Patent Document 1 discloses a technology for generating moving images in which a three-dimensional character (hereinafter, referred to as an "avatar") moves in response to the movements of a person.

JP 2015-184689 A

However, with the above technology, only the same video images were displayed to multiple viewers viewing an avatar reflecting a person's movements, and the display of the avatar could not be freely set for each viewer.

The present invention was made in consideration of these points, and aims to provide a mechanism that allows the display of avatars to be configured for each viewer.

The information processing device according to the first aspect of the present invention has a first terminal acquisition unit that acquires motion information indicating the motion of a person captured by an imaging unit of a first terminal, a second terminal acquisition unit that acquires a second terminal display request for displaying an avatar corresponding to the person from a second terminal different from the first terminal, an avatar generation unit that generates the avatar based on the second terminal display request each time the second terminal acquisition unit acquires the second terminal display request, a motion control unit that controls the motion of the avatar generated by the avatar generation unit based on the motion information, and a transmission unit that transmits display information to the second terminal for displaying the avatar whose motion is controlled by the motion control unit.

The second terminal display request may include type selection information indicating a type of the avatar selected from among a plurality of types of the avatar, and the avatar generation unit may generate the avatar of a type indicated by the type selection information included in the second terminal display request.

The second terminal acquisition unit may acquire, after the transmission unit transmits the display information, from the second terminal position information indicating the position of a virtual viewpoint for viewing the avatar in the virtual space in which the avatar is placed and orientation information indicating the orientation from the virtual viewpoint, the information processing device may further include a display information generation unit that generates the display information in which the avatar is displayed as viewed from the virtual viewpoint based on the position information and orientation information, and the transmission unit may transmit the display information generated by the display information generation unit to the second terminal.

The first terminal acquisition unit may further acquire body part setting information for setting a correspondence between a body part of the person that moves and a body part to be moved on the avatar, and the movement control unit may apply the movement of the body part of the person indicated by the movement information to the body part of the avatar that is associated with the body part of the person in the body part setting information.

The second terminal display request may include mode selection information indicating a mode selected from a VR (Virtual Reality) mode in which the avatar is displayed in a virtual space and an AR (Augmented Reality) mode in which the avatar is displayed in a real space, and the information processing device may further include a display information generation unit that generates the display information in which the avatar placed in the virtual space is displayed when the mode selection information indicates that the VR mode is selected, and generates the display information in which the avatar placed in the real space is displayed when the mode selection information indicates that the AR mode is selected, and the transmission unit may transmit the display information generated by the display information generation unit to the second terminal.

The information processing device may further include a determination unit that determines the quality level to be applied to the second terminal from among a plurality of quality levels based on a selected level selected by a user using the second terminal from among a plurality of quality levels each having a different display quality of the avatar displayed on the second terminal, or based on a usage state of a communication service used by the user, and the transmission unit may transmit the display information to the second terminal based on the quality level determined by the determination unit.

The information processing device may further include an acquisition unit for a third terminal that acquires, from a third terminal different from the first terminal and the second terminal, a third terminal display request for displaying the avatar displayed on the second terminal, the acquisition unit for the second terminal may acquire, from the second terminal, a display image that is displayed based on the display information, and the transmission unit may transmit the display image acquired by the acquisition unit for the second terminal to the third terminal.

The second terminal acquisition unit may further acquire motion instruction information for causing the avatar to perform a motion different from the motion of the person, and the motion control unit may control the motion of the avatar generated by the avatar generation unit based on the motion instruction information.

The information processing method according to the second aspect of the present invention includes the steps of: acquiring motion information indicating the motion of a person captured by an imaging unit of a first terminal, acquiring a second terminal display request for displaying an avatar corresponding to the person from a second terminal different from the first terminal, generating the avatar based on the second terminal display request each time the second terminal display request is acquired, controlling the motion of the avatar generated based on the motion information, and transmitting display information for displaying the avatar whose motion has been controlled to the second terminal.

The program according to the third aspect of the present invention causes a computer to function as: a first terminal acquisition unit that acquires motion information indicating the motion of a person captured by an imaging unit of a first terminal; a second terminal acquisition unit that acquires a second terminal display request for displaying an avatar corresponding to the person from a second terminal different from the first terminal; an avatar generation unit that generates the avatar based on the second terminal display request each time the second terminal acquisition unit acquires the second terminal display request; a motion control unit that controls the motion of the avatar generated by the avatar generation unit based on the motion information; and a transmission unit that transmits display information to the second terminal for displaying the avatar whose motion has been controlled by the motion control unit.

The present invention has the advantage of being able to provide a mechanism for allowing the display of avatars to be set for each viewer.

FIG. 1 is a diagram for explaining an overview of an information processing system. FIG. 2 is a diagram illustrating a functional configuration of an information processing device. 13 is a diagram showing a schematic representation of display images displayed simultaneously on a plurality of second terminals, each of which shows an avatar corresponding to a certain performer. FIG. 13 is a diagram showing a schematic representation of display images displayed simultaneously on a plurality of second terminals, each of which shows an avatar corresponding to a certain performer. FIG. 13 is a flowchart showing a flow of processing in the information processing device.

[Overview of Information Processing System S]
1 is a diagram for explaining an overview of an information processing system S. The information processing system S is a system used for providing an avatar presentation service to a user. The avatar presentation service is a service for presenting an avatar corresponding to a performer P to an audience, and is used for various purposes such as events and lessons.

A performer P is a person (user) who provides a predetermined performance to an audience via an avatar, such as a performer appearing at an event or a student taking a lesson. A viewer is a user who watches a predetermined performance given by performer P via an avatar, such as an audience member attending an event or an instructor who instructs a student in a lesson. The information processing system S has a first terminal 1, a second terminal 2, and an information processing device 3.

The first terminal 1 is a terminal managed by the performer P. The first terminal 1 is, for example, a smartphone, a tablet terminal, a personal computer, etc. The first terminal 1 has an imaging unit such as a camera that captures an image of a subject. The imaging unit may be an external camera electrically connected to the first terminal 1. The first terminal 1 may be a camera. The first terminal 1 is capable of communicating with the information processing device 3.

The second terminal 2 is a terminal different from the first terminal 1 and is used by the viewer. For example, a dedicated application program used to use the avatar presentation service is installed on the second terminal 2. The second terminal 2 is, for example, a smartphone, a tablet terminal, a personal computer, etc. The second terminal 2 has a display unit such as a display that displays information, and an operation unit such as a touch panel that accepts operations by the viewer. The second terminal 2 is capable of communicating with the information processing device 3.

The information processing device 3 is a device that manages the avatar display service. The information processing device 3 is, for example, a server. The information processing device 3 is capable of communicating with the first terminal 1 and the second terminal 2.
The processes executed by the information processing system S will be described below.

First, the first terminal 1 transmits movement information to the information processing device 3 ((1) in FIG. 1). The movement information is information indicating the movement of the performer P captured by the imaging unit of the first terminal 1, and is, for example, point cloud data of the entire body of the performer P. The movement information may also be point cloud data of a part of the performer P (for example, the face, upper body, lower body, etc.).

The information processing device 3 receives a second terminal display request from the second terminal 2 ((2) in FIG. 1). The second terminal display request is a request to display an avatar corresponding to the performer P. In the example shown in FIG. 1, the information processing device 3 receives a second terminal display request from each of the second terminal 2a and the second terminal 2b.

Each time the information processing device 3 receives a second terminal display request, it generates an avatar corresponding to the performer P ((3) in FIG. 1). In the example shown in FIG. 2, the information processing device 3 generates a first avatar corresponding to the second terminal 2a and a second avatar corresponding to the second terminal 2b.

The information processing device 3 controls the motion of each generated avatar based on the motion information acquired from the first terminal 1 ((4) in FIG. 1). In the example shown in FIG. 1, the information processing device 3 controls the motion of each of the first avatar and the second avatar based on the motion information acquired from the first terminal 1.

Then, the information processing device 3 transmits display information for displaying the avatar whose action has been controlled to the second terminal 2 ((5) in FIG. 1). The display information is, for example, a display image including an avatar. In the example shown in FIG. 1, the information processing device 3 transmits first display information for displaying the first avatar whose action has been controlled to the second terminal 2a, and transmits second display information for displaying the second avatar whose action has been controlled to the second terminal 2b.

Then, when the second terminal 2 acquires display information from the information processing device 3, it displays an avatar whose movement is controlled based on the display information. In the example shown in FIG. 1, the second terminal 2a displays avatar A1, which is the first avatar whose movement is controlled, based on the first display information. Also, the second terminal 2b displays avatar A2, which is the second avatar whose movement is controlled, based on the second display information.

In this way, the information processing system S can present an avatar for each viewer. As a result, even if a viewer sets the display of an avatar for that viewer (for example, setting the avatar's appearance, setting the angle, etc.), the information processing system S can prevent the settings made by that viewer from being reflected in the display of avatars for other viewers. As a result, the information processing system S can provide a mechanism that enables multiple viewers to each set the display of an avatar for each viewer.

In the above, an example has been described in which the first terminal 1 and the second terminal 2 are different terminals, but this is not limiting. For example, the first terminal 1 may have a function that the second terminal 2 has and display an avatar. In this way, the first terminal 1 displays an avatar controlled according to the movement of the performer P, so that the performer P can perform a performance while checking the avatar whose movement is being controlled.
The configuration of the information processing device 3 will be described below.

[Configuration of information processing device 3]
FIG. 2 is a diagram showing a schematic functional configuration of the information processing device 3. The information processing device 3 includes a communication unit 31, a storage unit 32, and a control unit 33. In FIG. 2, the arrows indicate main data flows, and there may be data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between the functional blocks via any means, such as a data bus, a network, or a portable storage medium.

The communication unit 31 is a communication interface for connecting to a network, and has a communication controller for receiving data from external terminals and external servers.

The storage unit 32 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the information processing device 3, a RAM (Random Access Memory) that serves as the working area of the information processing device 3, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referenced when the application programs are executed.

The storage unit 32 stores information about users who use the avatar presentation service. Information about users includes, for example, the user's ID, the user's name, etc.

The control unit 33 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the information processing device 3, and functions as an acquisition unit for a first terminal 331, an acquisition unit for a second terminal 332, an avatar generation unit 333, an operation control unit 334, a display information generation unit 335, a transmission unit 336, a determination unit 337, and an acquisition unit for a third terminal 338 by executing a program stored in the storage unit 32.

The acquisition unit for the first terminal 331 acquires motion information indicating the motions of performer P captured by the imaging unit of the first terminal 1. For example, the first terminal 1 first captures an image (e.g., a moving image) of performer P moving with the imaging unit, and then transmits frame images for each frame of the moving image to a data generation server (not shown). The data generation server generates point cloud data based on the frame images acquired from the first terminal 1. The acquisition unit for the first terminal 331 then acquires the point cloud data generated by the data generation server for each frame image as motion information.

The first terminal 1 may have a function of a data generation server and generate point cloud data based on the captured image captured by the imaging unit. The first terminal acquisition unit 331 may have a function of a data generation server and acquire operation information by generating point cloud data based on the captured image acquired from the first terminal 1.

The first terminal acquisition unit 331 may acquire the motion information in association with an avatar ID. The avatar ID is a user ID for identifying the performer P or an avatar corresponding to the performer P.

The second terminal acquisition unit 332 acquires a second terminal display request for displaying an avatar corresponding to performer P from the second terminal 2. Specifically, the second terminal acquisition unit 332 acquires a second terminal display request from each of the multiple second terminals 2. For example, the second terminal display request includes the avatar ID and the viewer ID. The viewer ID is a user ID for identifying the viewer.

For example, first, when the viewer performs an operation on the second terminal 2 to access a site that provides an avatar presentation service, the second terminal 2 displays a list screen showing a list of one or more presentable avatars. When the viewer performs an operation to select an avatar on the list screen displayed on the second terminal 2, the second terminal 2 transmits a second terminal display request including the ID of the avatar selected by the viewer and the ID of the viewer. The second terminal acquisition unit 332 then acquires the second terminal display request from the second terminal 2.

The avatar generation unit 333 generates an avatar based on a second terminal display request each time the second terminal acquisition unit 332 acquires the second terminal display request. Specifically, the avatar generation unit 333 generates an avatar corresponding to the second terminal 2 that sent the second terminal display request each time the second terminal acquisition unit 332 acquires the second terminal display request.

The avatar generation unit 333 generates an avatar, for example, by generating an instance corresponding to the second terminal 2 that sent the second terminal display request acquired by the second terminal acquisition unit 332. The instance is, for example, an area generated to present an avatar corresponding to the avatar ID included in the second terminal display request to the second terminal 2 that sent the second terminal display request. The instance includes information for rendering an image including the avatar, and includes information for configuring the appearance of the avatar, such as shape, color, size, etc.

The information processing device 3 can present the avatar in VR or AR. When the avatar is presented in VR, the instance may further include information indicating the position and orientation of a virtual viewpoint for viewing the avatar in the virtual space in which the avatar is placed, and information for configuring objects other than the avatar (e.g., background, etc.). When the avatar is presented in AR, the instance may further include information indicating the position in real space in which the avatar is placed. The avatar generation unit 333 generates the instance in association with, for example, the avatar ID and the viewer ID included in the second terminal display request.

The motion control unit 334 controls the motion of the avatar generated by the avatar generation unit 333 based on the motion information. Specifically, for each avatar generated by the avatar generation unit 333, the motion control unit 334 controls the motion of the avatar by reflecting the motion of the performer P indicated by the motion information acquired by the first terminal acquisition unit 331 in association with the avatar ID included in the second terminal display request, for the avatar corresponding to the avatar ID included in the second terminal display request.

For example, for each instance generated by the avatar generation unit 333, the movement control unit 334 reflects the movement of performer P indicated by the movement information acquired by the first terminal acquisition unit 331 in association with the ID of the avatar associated with the instance, to the avatar included in the instance. The movement control unit 334 reflects the movement of performer P indicated by the movement information in the avatar, for example, using known technology.

The movement control unit 334, for example, reflects the movements of performer P in the avatar in real time. Specifically, the movement control unit 334 synchronizes the movements of performer P indicated by the movement information acquired by the first terminal acquisition unit 331 with the movements of the avatar. For example, each time the first terminal acquisition unit 331 acquires new movement information, the movement control unit 334 reflects the movements of performer P indicated by the new movement information in the avatar.

The transmission unit 336 transmits to the second terminal 2 display information for displaying the avatar whose operation has been controlled by the operation control unit 334. Specifically, the information processing device 3 transmits the display information to the second terminal 2 by executing the following two steps.

As a first step, the display information generation unit 335 generates, for each avatar generated by the avatar generation unit 333, display information for displaying the avatar whose action is controlled by the action control unit 334. For example, for each instance generated by the avatar generation unit 333, the display information generation unit 335 generates display information based on information included in the instance, by rendering an image including an avatar in the instance whose action is controlled by the action control unit 334 based on the action information.

As a second step, for each piece of display information generated by the display information generation unit 335, the transmission unit 336 transmits the display information to the second terminal 2 corresponding to the display information. Specifically, the transmission unit 336 transmits the display information generated by the display information generation unit 335 to the second terminal 2 corresponding to the ID of the viewer associated with the instance based on the display information. For example, each time the display information generation unit 335 performs rendering, the transmission unit 336 transmits the display information corresponding to the rendering to the second terminal 2.

For each viewer, the information processing device 3 presents an avatar according to the settings made by the viewer. For example, the information processing device 3 presents the avatar in a mode selected by the viewer from among a VR mode in which the avatar is displayed in a virtual space and an AR mode in which the avatar is displayed in real space. Specifically, the information processing device 3 presents the avatar in the mode selected by the viewer by executing the following three steps.

As a first step, the second terminal acquisition unit 332 acquires a second terminal display request including mode selection information. The mode selection information is information indicating the mode selected from VR mode and AR mode. For example, the avatar list screen is provided with a selection item for selecting either VR mode or AR mode, and when the viewer selects either mode from the selection item, the second terminal 2 transmits a second terminal display request including mode selection information indicating the mode selected by the viewer to the information processing device 3.

In a second step, when the mode selection information indicates that the VR mode has been selected, the display information generation unit 335 generates display information that displays an avatar placed in a virtual space. For example, the display information generation unit 335 generates a display image as display information that displays the avatar and other objects (e.g., objects that make up the background) as viewed from a specific virtual viewpoint (e.g., a predetermined virtual viewpoint).

When the mode selection information indicates that the AR mode has been selected, the display information generating unit 335 generates display information in which the avatar placed in the real space is displayed. For example, first, the second terminal 2 displays the captured image captured by the imaging unit on the display unit after the viewer selects an avatar and AR mode on the avatar list screen. When the viewer performs an operation to specify the position where the avatar is to be placed on the display unit of the second terminal 2 on which the captured image is displayed (for example, an operation to tap a place on the display unit corresponding to the position where the avatar is to be placed), the second terminal 2 transmits positional relationship information (for example, distance, position coordinates of each position, etc.) indicating the relationship between the presence position of the second terminal 2 and the specified position specified by the viewer to the information processing device 3. Then, when the second terminal acquisition unit 332 acquires the positional relationship information, the display information generating unit 335 generates a display image as display information in which the avatar placed at the specified position is displayed with the presence position of the second terminal 2 as the viewpoint, based on the positional relationship information.

As a third step, the transmission unit 336 transmits the display information generated by the display information generation unit 335 to the second terminal 2. The second terminal 2 then displays on the display unit a display image in which an avatar based on the display information acquired from the information processing device 3 is displayed. For example, if the viewer selects the VR mode, the second terminal 2 displays on the display unit the display image acquired from the information processing device 3. Also, for example, if the viewer selects the AR mode, the second terminal 2 displays on the display unit the display image acquired from the information processing device 3 superimposed on the captured image captured by the imaging unit. In this way, the information processing device 3 can present the avatar in the mode desired by the viewer.

The information processing device 3 may present an avatar of a type set by the viewer. The avatar type may be, for example, a person, an animal, a robot, etc. Specifically, the information processing device 3 presents an avatar of a type set by the viewer by executing the following three steps.

As a first step, the second terminal acquisition unit 332 acquires a second terminal display request including type selection information. The type selection information is information indicating the type of avatar selected from a plurality of avatar types. For example, the avatar list screen has selection items for selecting the type of avatar, and when the viewer performs an operation to select one avatar type from the selection items on the avatar list screen displayed on the second terminal 2, the second terminal 2 transmits to the information processing device 3 a second terminal display request that further includes type selection information indicating the type of avatar selected by the viewer.

In a second step, the avatar generation unit 333 generates an avatar of a type indicated by the type selection information included in the second terminal display request acquired by the second terminal acquisition unit 332. The avatar generation unit 333 stores, for example, information for configuring the appearance of the avatar of the type indicated by the type selection information included in the second terminal display request acquired by the second terminal acquisition unit 332 in an instance.

As a third step, the transmission unit 336 transmits to the second terminal 2 display information for displaying an avatar generated by the avatar generation unit 333 and of a type indicated by the type selection information included in the second terminal display request. After that, the second terminal 2 causes the display unit to display a display image showing the avatar based on the display information acquired from the information processing device 3.

Figure 3 is a schematic diagram showing display images displayed on multiple second terminals 2 in which an avatar corresponding to a certain performer P is simultaneously presented. The diagram shown in Figure 3(a) shows, for example, a display screen displayed on one of two second terminals 2. In the example shown in Figure 3(a), avatar A3, which has the appearance of a "human", is displayed as the type of avatar selected by a viewer using one of the second terminals 2.

The diagram shown in FIG. 3(b) shows, for example, a display screen displayed on the other of the two second terminals 2. In the example shown in FIG. 3(b), avatar A4, which has the appearance of a "dog," is displayed as the type of avatar selected by the viewer using the other second terminal 2. In this way, the information processing device 3 can present, for each viewer, the type of avatar selected by that viewer.

Here, when the information processing device 3 presents to the viewer a type of avatar other than a human, such as an animal or a robot, it is desirable to associate the body parts of the performer P that move (e.g., arms, legs, etc.) with the body parts of the avatar that are to move (e.g., front legs, hind legs, tail, etc. when the avatar type is "dog"). Thus, the information processing device 3 may apply the movement of the body parts of the performer P to the body parts of the avatar that are associated with the body parts of the performer P.

Specifically, first, the first terminal acquisition unit 331 further acquires body part setting information for setting a correspondence between a body part moving on performer P and a body part to be moved on the avatar. The body part setting information is, for example, information that associates a body part moving on performer P with a body part of the avatar that is to be made to correspond to the body part of performer P. The body part setting information is, for example, set by performer P for each type of avatar. Then, the movement control unit 334 applies the movement of the body part of performer P indicated by the movement information to the body part of the avatar that is made to correspond to the body part of the person in the body part setting information acquired by the first terminal acquisition unit 331. In this way, the information processing device 3 can cause performer P to perform the movement desired according to the type of avatar.

When a viewer selects the VR mode, the information processing device 3 may present the avatar at an angle set by the viewer. Specifically, the information processing device 3 presents the avatar at the angle set by the viewer by executing the following three steps.

As a first step, after the transmission unit 336 transmits the display information, the second terminal acquisition unit 332 acquires from the second terminal 2 position information indicating the position of a virtual viewpoint for viewing the avatar in the virtual space in which the avatar is placed, and orientation information indicating the direction from the virtual viewpoint. For example, a controller for controlling the angle is provided on the display screen on which the avatar is displayed on the second terminal 2, and when the viewer performs an operation to change the angle via the controller, the second terminal 2 transmits position information and orientation information corresponding to the changed angle to the information processing device 3.

In a second step, the display information generating unit 335 generates display information that displays the avatar as viewed from a virtual viewpoint, based on the position information and orientation information acquired by the second terminal acquisition unit 332. For example, the display information generating unit 335 first updates the position information and orientation information included in the instance to the position information and orientation information acquired by the second terminal acquisition unit 332. Then, based on the information included in the instance after updating the position information and orientation information, the display information generating unit 335 generates display information by rendering an image including an avatar whose behavior is controlled by the behavior control unit 334 based on the behavior information in the instance.

As a third step, the transmission unit 336 transmits the display information generated by the display information generation unit 335 to the second terminal 2. FIG. 4 is a schematic diagram showing display images displayed on multiple second terminals 2 simultaneously presenting an avatar corresponding to a certain performer P.

The diagram shown in FIG. 4(a) shows, for example, a display screen displayed on one of the two second terminals 2. In the example shown in FIG. 4(a), avatar A5 is displayed as viewed from a virtual viewpoint V1 located in front of the avatar.

The diagram shown in FIG. 4(b) shows, for example, a display screen displayed on the other of the two second terminals 2. In the example shown in FIG. 4(b), avatar A6 is displayed as viewed from a virtual viewpoint V2 located to the side of the avatar. In this way, the information processing device 3 can present the avatar at an angle desired by the viewer.

The information processing device 3 may present the avatar with a quality appropriate to the viewer. Specifically, the information processing device 3 presents the avatar with a quality appropriate to the viewer by performing the following two steps.

As a first step, the determination unit 337 determines a quality level corresponding to the viewer to be applied to the second terminal 2 from among a plurality of quality levels. The quality level is the display quality of the avatar displayed on the second terminal 2, such as the number of pixels of the display image of the avatar, the frame rate of the display image of the avatar, etc. The information processing device 3 is set with a plurality of quality levels each having a different display quality of the avatar, such as "high", "medium", "low", etc.

Specifically, the determination unit 337 determines the quality level to be applied to the second terminal 2 based on the selection level selected by the viewer using the second terminal 2. For example, the second terminal acquisition unit 332 first acquires a second terminal display request including level information indicating the selection level selected by the viewer from among multiple quality levels. Then, the determination unit 337 determines the selection level indicated by the level information included in the second terminal display request acquired by the second terminal acquisition unit 332 as the quality level to be applied to the second terminal 2.

The determination unit 337 may determine the quality level to be applied to the second terminal 2 based on the usage status of the communication service used by the viewer. For example, the administrator who manages the information processing device 3 is a communication carrier that provides communication services, and the memory unit 32 stores information about users who use the communication services. The information about the users includes, for example, the user's ID, the user's usage history of the communication services, the communication service usage plan subscribed to by the user, etc.

In this case, the determination unit 337 determines the quality level to be applied to the second terminal 2 based on the usage state of the communication service used by the viewer identified by the information about the user stored in the storage unit 32. For example, if the viewer is using a communication service, the determination unit 337 determines the quality level to be applied to the second terminal 2 to be high, and if the viewer is not using a communication service, the determination unit 337 determines the quality level to be applied to the second terminal 2 to be low.

The determination unit 337 may determine a higher quality level to be applied to the second terminal 2 as the degree of usage of the communication service specified by the usage history of the viewer using the communication service (e.g., the amount of communication used by the viewer or the amount paid by the viewer, etc.) is higher, and may determine a lower quality level to be applied to the second terminal 2 as the degree of usage of the communication service is lower. For example, when the degree of usage of the communication service exceeds a usage threshold of a predetermined standard (e.g., a predetermined amount of communication or a predetermined amount), the determination unit 337 determines a high quality level to be applied to the second terminal 2, and when the degree of usage of the communication service does not exceed the usage threshold, the determination unit 337 determines a low quality level to be applied to the second terminal 2.

The determination unit 337 may determine a higher quality level to be applied to the second terminal 2 the higher the plan (e.g., a plan with a higher price) of the viewer using the communication service, and may determine a lower quality level to be applied to the second terminal 2 the lower the plan (e.g., a plan with a lower price). For example, when the viewer's plan is a plan higher than a predetermined standard plan threshold (e.g., a predetermined plan), the determination unit 337 determines a higher quality level to be applied to the second terminal 2, and when the viewer's plan is not a plan higher than the plan threshold, the determination unit 337 determines a lower quality level to be applied to the second terminal 2.

In a second step, the transmission unit 336 transmits the display information to the second terminal 2 based on the quality level determined by the determination unit 337. For example, if the quality level is the number of pixels of the display image of the avatar, the transmission unit 336 transmits the display information generated by the display information generation unit 335 based on the quality level determined to the second terminal 2. For example, the number of pixels is determined for each quality level, and the display information generation unit 335 generates a display image with the number of pixels corresponding to the quality level determined by the determination unit 337.

For example, if the quality level is the frame rate of the display image of the avatar, the transmission unit 336 transmits the display information generated by the display information generation unit 335 based on the quality level determined to the second terminal 2. For example, a frame rate is determined for each quality level, and the display information generation unit 335 generates the display image at a frame rate corresponding to the quality level determined by the determination unit 337. In this way, the information processing device 3 can present the avatar at a quality level according to the viewer.

The information processing device 3 may present the display image displayed on the display unit of the second terminal 2 on which the avatar is displayed to another second terminal 2 (hereinafter referred to as a "third terminal") different from the second terminal 2. The display image is an image in which a display image generated by the display information generating unit 335 in VR mode, or a display image generated by the display information generating unit 335 in AR mode, and an image captured by the imaging unit of the second terminal 2 are superimposed. Specifically, the information processing device 3 presents the display image of the second terminal 2 to the third terminal by executing the following three steps.

As a first step, the third terminal acquisition unit 338 acquires a third terminal display request from the third terminal to display the avatar displayed on the second terminal 2. The third terminal display request includes the ID of the viewer of the second terminal 2. For example, the avatar list screen further displays information (e.g., the viewer's name, etc.) indicating the viewer (the viewer of the second terminal 2) who is viewing the avatar, and when the viewer of the third terminal performs an operation to select the viewer of the second terminal 2, the third terminal transmits to the information processing device 3 a third terminal display request including the ID of the viewer of the second terminal 2 selected by the viewer of the third terminal.

As a second step, the second terminal acquisition unit 332 acquires the display image displayed based on the display information from the second terminal 2. Specifically, the second terminal acquisition unit 332 acquires the display image displayed based on the display information from the second terminal 2 corresponding to the viewer ID included in the third terminal display request acquired by the third terminal acquisition unit 338.

As a third step, the transmission unit 336 transmits the display image acquired by the second terminal acquisition unit 332 to the third terminal. In this way, the information processing device 3 can share the display image including the avatar displayed on a second terminal 2 with other viewers other than the viewer of the second terminal 2.

The information processing device 3 may present the display image displayed on the display unit of the second terminal 2 on which the avatar is displayed to a third terminal that satisfies a setting condition set by the viewer of the second terminal 2. The setting condition is, for example, whether or not sharing of the display screen displayed on the display unit of the second terminal 2 is permitted, the sharing range for sharing the display screen, etc. The sharing range is, for example, up to the friends of the viewer of the second terminal 2 (up to other viewers who are registered as friends of the viewer of the second terminal 2) in a case where the avatar presentation service has a function for registering friends, or up to other viewers in the nth hierarchical level (for example, up to friends (other viewers in the second hierarchical level) of the viewer of the second terminal 2 (other viewers in the first hierarchical level) in a case where the sharing range is set to up to other viewers in the second hierarchical level), etc.

In this case, the information processing device 3, for example, displays information indicating the viewer of the second terminal 2 on a list screen of avatars displayed on the third terminal when the third terminal satisfies the setting conditions of the viewer of the second terminal 2 displaying the avatar.

When the avatar presentation service is used for lessons, the instructor, who is also a viewer, can not only watch the performance of the performer P, but can also instruct the performer P on the performance. In this case, it is desirable for the instructor's instructional movements to be reflected in the avatar and fed back to the performer P.

The information processing device 3 may therefore control the movement of the avatar in response to the instructor's instructions. In this case, for example, the first terminal 1 has the function of the second terminal 2 (the function of displaying the avatar), and the second terminal 2 has the function of the first terminal 1 (the function of transmitting information indicating the instructor's movements). Specifically, the information processing device 3 controls the movement of the avatar in response to the instructor's instructions by executing the following two steps.

As a first step, the second terminal acquisition unit 332 further acquires action instruction information for causing the avatar to perform an action different from that of the performer P. For example, the display screen on which the avatar is displayed is provided with a switch button for switching between control of the avatar's action based on the action information of the first terminal 1 and control of the avatar's action based on the action information of the second terminal 2, and when the instructor presses the switch button on the second terminal 2, the second terminal 2 transmits the action instruction information to the information processing device 3. The second terminal 2 then transmits action information indicating the instructor's action to the information processing device 3.

In a second step, the operation control unit 334 controls the operation of the avatar generated by the avatar generation unit based on the operation instruction information. For example, when the operation instruction information indicates that the operation of the avatar is to be controlled based on the operation information of the second terminal 2, the operation control unit 334 controls the operation of the avatar based on the operation information acquired from the second terminal 2 by the second terminal acquisition unit 332. Thereafter, the transmission unit 336 transmits display information to the first terminal 1 and the second terminal 2 for displaying the avatar whose operation has been controlled by the operation control unit 334 based on the operation information of the second terminal 2.

[Processing of information processing device 3]
Next, a process flow of the information processing device 3 will be described. Fig. 5 is a flowchart showing the process flow of the information processing device 3. The first terminal acquisition unit 331 acquires video information corresponding to each frame image. In this case, this process starts when the second terminal acquisition unit 332 acquires a second terminal display request from each of the multiple second terminals 2 (S1).

The avatar generation unit 333 generates an avatar based on the second terminal display request each time the second terminal acquisition unit 332 acquires the second terminal display request (S2). The operation control unit 334 controls the operation of each avatar generated by the avatar generation unit 333 based on the latest operation information acquired by the first terminal acquisition unit 331 (S3).

The display information generation unit 335 generates, for each avatar generated by the avatar generation unit 333, display information for displaying the avatar whose operation is controlled by the operation control unit 334 (S4). Then, for each piece of display information generated by the display information generation unit 335, the transmission unit 336 transmits the display information to the second terminal 2 corresponding to the display information (S5).

[Effects of this embodiment]
As described above, each time the information processing device 3 acquires a second terminal display request from the second terminal 2, the information processing device 3 generates an avatar based on the second terminal display request, and transmits display information for displaying the avatar whose operation is controlled based on the operation information to the second terminal 2. In this manner, the information processing device 3 can present an avatar for each viewer. As a result, even if a viewer sets the display of an avatar for that viewer, the information processing device 3 can prevent the contents set by that viewer from being reflected in the display of avatars for other viewers. As a result, the information processing device 3 can provide a mechanism that enables each of multiple viewers to set the display of an avatar for each viewer.

Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1 First terminal 2 Second terminal 3 Information processing device 31 Communication unit 32 Storage unit 33 Control unit 331 First terminal acquisition unit 332 Second terminal acquisition unit 333 Avatar generation unit 334 Operation control unit 335 Display information generation unit 336 Transmission unit 337 Determination unit 338 Third terminal acquisition unit S Information processing system

Claims (8)

an acquisition unit for a first terminal that acquires first motion information indicating a motion of a first person captured by an imaging unit of the first terminal;
an acquisition unit for a second terminal that acquires, from each of a plurality of second terminals different from the first terminal, a second terminal display request for displaying an avatar corresponding to the first person, the second terminal display request including an ID of a viewer using the second terminal, position information indicating a position of a virtual viewpoint for viewing the avatar in a virtual space in which the avatar is placed and set by the viewer, and orientation information indicating a direction from the virtual viewpoint and set by the viewer;
an avatar generation unit that generates the avatar in association with an ID of the viewer included in the second terminal display request each time the second terminal acquisition unit acquires the second terminal display request;
a movement control unit that controls a movement of the avatar generated by the avatar generation unit based on the first movement information;
a display information generation unit that generates display information for displaying the avatar, the avatar whose movement is controlled by the movement control unit, when viewed from the virtual viewpoint, based on the position information and the orientation information;
a transmission unit that transmits the display information generated by the display information generation unit to the second terminal corresponding to an ID of the viewer associated with the avatar displayed by the display information, thereby transmitting the display information to each of the plurality of second terminals;
having
the second terminal acquisition unit further acquires second action information indicating an action of a second person who is captured by an imaging unit of the second terminal and is different from the first person, and action instruction information for switching from control of the action of the avatar based on the first action information to control of the action of the avatar based on the second action information;
when the second terminal acquisition unit acquires the operation instruction information, the operation control unit switches from control of the operation of the avatar based on the first operation information to control of the operation of the avatar based on the second operation information;
The transmission unit transmits the display information generated by the display information generation unit to the first terminal and the second terminal corresponding to the second person after the operation control unit switches to control of the operation of the avatar based on the second operation information.
Information processing device. the second terminal display request includes type selection information indicating a type of the avatar selected from a plurality of types of the avatar;
the avatar generation unit generates the avatar of a type indicated by the type selection information included in the second terminal display request.
The information processing device according to claim 1 . the first terminal acquisition unit further acquires body part setting information for setting a correspondence between a body part to be moved on the first person and a body part to be moved on the avatar;
the movement control unit applies a movement of the body part of the first person indicated by the first movement information to a body part of the avatar that is associated with the body part of the first person in the body part setting information;
The information processing device according to claim 1 . the second terminal display request includes mode selection information indicating a mode selected from a VR (Virtual Reality) mode in which the avatar is displayed in a virtual space and an AR (Augmented Reality) mode in which the avatar is displayed in a real space;
the display information generation unit generates the display information in which the avatar arranged in the virtual space is displayed when the mode selection information indicates that the VR mode has been selected, and generates the display information in which the avatar arranged in the real space is displayed when the mode selection information indicates that the AR mode has been selected;
The transmission unit transmits the display information generated by the display information generation unit to the second terminal.
The information processing device according to claim 1 . The information processing device further includes a determination unit that determines a quality level to be applied to the second terminal from among a plurality of quality levels, each of which has a different display quality of the avatar displayed on the second terminal, based on a selection level selected by a user who uses the second terminal, or based on a usage state of a communication service used by the user, and
The transmission unit transmits the display information to the second terminal based on the quality level determined by the determination unit.
The information processing device according to claim 1 . the information processing device further includes a third terminal acquisition unit configured to acquire, from a third terminal different from the first terminal and the second terminal, a third terminal display request for displaying the avatar displayed on the second terminal;
The second terminal acquisition unit acquires, from the second terminal, a display image that is displayed based on the display information;
The transmission unit transmits the display image acquired by the second terminal acquisition unit to the third terminal.
The information processing device according to claim 1 . The computer executes
acquiring first motion information indicating a motion of a first person captured by an imaging unit of a first terminal;
acquiring, from each of a plurality of second terminals different from the first terminal, a second terminal display request for displaying an avatar corresponding to the first person, the second terminal display request including an ID of a viewer using the second terminal, position information indicating a position of a virtual viewpoint for viewing the avatar in a virtual space in which the avatar is placed and set by the viewer, and orientation information indicating a direction from the virtual viewpoint and set by the viewer;
generating the avatar in association with an ID of the viewer included in the second terminal display request each time the second terminal display request is received;
controlling a movement of the generated avatar based on the first movement information;
generating display information for displaying the avatar whose motion has been controlled, when viewed from the virtual viewpoint, based on the position information and the orientation information;
transmitting the generated display information to the second terminal corresponding to the ID of the viewer associated with the avatar displayed by the display information, thereby transmitting the display information to each of the second terminals;
acquiring second action information indicating an action of a second person who is captured by an imaging unit of the second terminal and is different from the first person, and action instruction information for switching from control of the action of the avatar based on the first action information to control of the action of the avatar based on the second action information;
when the action instruction information is acquired, switching from control of the action of the avatar based on the first action information to control of the action of the avatar based on the second action information;
transmitting the display information generated after switching to control of the avatar's movement based on the second movement information to the first terminal and the second terminal corresponding to the second person;
An information processing method comprising the steps of: Computer,
an acquisition unit for a first terminal that acquires first motion information indicating a motion of a first person captured by an imaging unit of the first terminal;
an acquisition unit for a second terminal that acquires, from each of a plurality of second terminals different from the first terminal, a second terminal display request for displaying an avatar corresponding to the first person, the second terminal display request including an ID of a viewer using the second terminal, position information indicating a position of a virtual viewpoint for viewing the avatar in a virtual space in which the avatar is placed and set by the viewer, and orientation information indicating a direction from the virtual viewpoint and set by the viewer;
an avatar generation unit that generates the avatar in association with an ID of the viewer included in the second terminal display request each time the second terminal acquisition unit acquires the second terminal display request;
a movement control unit that controls a movement of the avatar generated by the avatar generation unit based on the first movement information;
a display information generating unit configured to generate first display information, the first display information being display information for displaying the avatar, whose operation is controlled by the operation control unit, as viewed from the virtual viewpoint, based on the position information and the orientation information; and a transmitting unit configured to transmit the display information generated by the display information generating unit to the second terminal corresponding to the ID of the viewer associated with the avatar displayed by the display information, thereby transmitting the display information to each of the plurality of second terminals.
Function as a
the second terminal acquisition unit further acquires second action information indicating an action of a second person who is captured by an imaging unit of the second terminal and is different from the first person, and action instruction information for switching from control of the action of the avatar based on the first action information to control of the action of the avatar based on the second action information;
when the second terminal acquisition unit acquires the operation instruction information, the operation control unit switches from control of the operation of the avatar based on the first operation information to control of the operation of the avatar based on the second operation information;
The transmission unit transmits the display information generated by the display information generation unit to the first terminal and the second terminal corresponding to the second person after the operation control unit switches to control of the operation of the avatar based on the second operation information.
program.

Images