JP7611579B2 - Computer program, server device, terminal device, and method - Google Patents

Description

The technology disclosed in this application relates to a computer program, a server device, a terminal device, and a method.

In recent years, technology that uses avatars played by broadcasters has been developing. This type of technology allows viewers to watch avatars played by broadcasters via a broadcasting service.

In relation to services using avatars, a service called "Custom Cast" is known as a service that uses technology to control the facial expressions and movements of an avatar object based on the movements of a performer or the like (Non-Patent Document 1). In this service, the performer pre-assigns one of a large number of prepared facial expressions or movements to each of a number of flick directions on the smartphone screen, and by flicking the smartphone screen in the direction corresponding to the desired facial expression or movement, the performer can cause the avatar object displayed in the video to express that facial expression or movement.

"Custom Cast", [online], Custom Cast Inc., [searched December 10, 2019], Internet (URL: https://customcast.jp/)

However, with the technology disclosed in Non-Patent Document 1, the performer must flick the smartphone screen while speaking, which makes it difficult for the performer to perform the flicking operation.

The above non-patent document 1 is incorporated herein in its entirety by reference.

Some embodiments disclosed in this application provide a computer program, a server device, a terminal device, and a method that can more appropriately represent an avatar.

A computer program according to one embodiment is executed by one or more processors to acquire information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and display the avatar to which the predetermined facial expression or pose has been applied.

A computer program according to one embodiment is executed by one or more processors to obtain information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, in a virtual space, determine whether the obtained information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generate information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmit the information for changing the predetermined facial expression or pose.

A server device according to one embodiment includes one or more processors, and the processor executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generate information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmit the information for changing the predetermined facial expression or pose.

A method according to one aspect is a method executed by one or more processors executing computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and displays the avatar to which the predetermined facial expression or pose has been applied.

A method according to one aspect is a method executed by one or more processors executing computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the predetermined facial expression or pose.

A viewer terminal device according to one embodiment includes one or more processors, and executes computer-readable instructions to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and display the avatar to which the predetermined facial expression or pose has been applied.

A broadcaster terminal device according to one embodiment includes one or more processors, and the processor executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generate information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmit the information for changing the predetermined facial expression or pose.

FIG. 1 is a block diagram showing an example of a configuration of a system according to an embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the server device 20 (terminal device 30) illustrated in FIG. FIG. 3 is a block diagram illustrating an example of the functions of a system according to an embodiment. FIG. 4 is an example of data held by the system according to an embodiment. FIG. 5 is an example of data held by a system according to an embodiment. FIG. 6 is an example of a flow of processing performed by a system according to an embodiment. FIG. 7 is an example of a flow of processing performed by a system according to an embodiment. FIG. 8 is an example of a flow of processing performed by a system according to an embodiment.

Various embodiments of the present invention will now be described with reference to the accompanying drawings. Please note that components depicted in one drawing may be omitted in another drawing for ease of explanation. Furthermore, please note that the accompanying drawings disclose an embodiment of the present invention, but are not necessarily drawn to scale. In addition, in this application document, the subject matter indicated by the term application may be what is called software or a program, and may be instructions for a computer that are combined to produce a result.

1. Example of a System FIG. 1 is a block diagram showing an example of a system configuration according to an embodiment. As shown in FIG. 1, the system 1 may include one or more server devices 20 connected to a communication network 10, and one or more terminal devices 30 connected to the communication network 10. Note that in FIG. 1, three server devices 20A to 20C are illustrated as examples of the server devices 20, and three terminal devices 30A to 30C are illustrated as examples of the terminal devices 30, but one or more server devices 20 other than these may be connected to the communication network 10 as the server devices 20, and one or more terminal devices 30 other than these may be connected to the communication network 10 as the terminal devices 30. Note that in the present application documents, the term "system" may include both a server and a terminal device, or may refer to only a server or only a terminal device. In other words, the system may be in any of the following forms: only a server, only a terminal device, or both a server and a terminal device.

The system may also be an information processing device on the cloud. The system may also constitute a virtual information processing device and be logically configured as a single information processing device. The owner and administrator of the system may be different persons.

The communication network 10 may be, but is not limited to, a mobile phone network, a wireless LAN, a landline telephone network, the Internet, an intranet, Ethernet (registered trademark), and/or a combination thereof. In the communication network 10, various data disclosed in the present application documents may be communicated.

The server device 20 executes a specific installed application to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar, in a virtual space, determines whether or not the acquired information satisfies a predetermined condition, and if it determines that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor, which has been generated based on the facial expression or pose related to the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition.
It may be possible to execute operations such as:

Alternatively, the terminal device 30 may execute an installed web browser or a specific application to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar, in a virtual space, determine whether or not the acquired information satisfies a predetermined condition, and if it determines that the predetermined condition is satisfied, generate information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, etc.

The terminal device 30 is any terminal device capable of performing such operations, and may be, but is not limited to, a smartphone, a tablet, a mobile phone (feature phone), and/or a personal computer.

2. Hardware Configuration of Each Device Next, an example of the hardware configuration of the server device 20 will be described.

2-1. Hardware configuration of server device 20 An example of the hardware configuration of the server device 20 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the server device 20 (terminal device 30) shown in Fig. 1 (note that in Fig. 2, the reference numerals in parentheses are those described in relation to each terminal device 30, as described later).

As shown in FIG. 2, the server device 20 may mainly include a computing device 21, a main memory device 22, and an input/output interface device 23. The server device 20 may further include an input device 24 and an auxiliary output device 26. These devices may be connected to each other by a data bus and/or a control bus.

The arithmetic unit 21 performs calculations using instructions and data stored in the main memory 22, and stores the results of the calculations in the main memory 22. Furthermore, the arithmetic unit 21 can control the input device 24, the auxiliary memory 25, the output device 26, etc. via the input/output interface device 23. The server device 20 may include one or more arithmetic units 21. The arithmetic unit 21 may include one or more central processing units (CPUs), microprocessors, and/or graphics processing units (GPUs).

The main memory device 22 has a storage function and stores commands and data received from the input device 24, the auxiliary memory device 25, and the communication network 10 (such as the server device 20) via the input/output interface device 23, as well as the results of calculations by the calculation device 21. The main memory device 22 can include, but is not limited to, RAM (random access memory), ROM (read only memory), and/or flash memory.

The auxiliary storage device 25 is a storage device. It may store instructions and data (computer programs) that constitute the specific application, web browser, etc., and these instructions and data (computer programs) may be loaded into the main storage device 22 via the input/output interface device 23 under the control of the computing device 21. The auxiliary storage device 25 may be, but is not limited to, a magnetic disk device and/or an optical disk device, a file server, etc.

The input device 24 is a device that inputs data from the outside, and may be a touch panel, a button, a keyboard, a mouse, and/or a sensor, etc.

The output device 26 may include, but is not limited to, a display device, a touch panel, and/or a printer device. The input device 24 and the output device 26 may also be integrated.

In such a hardware configuration, the calculation device 21 sequentially loads the instructions and data (computer program) constituting a specific application stored in the auxiliary storage device 25 into the main storage device 22, and by calculating the loaded instructions and data, it may be possible to control the output device 26 via the input/output interface device 23, or to transmit and receive various information to and from other devices (such as the server device 20 and other terminal devices 30) via the input/output interface device 23 and the communication network 10.

By having such a configuration and executing a specific application installed, the server device 20 may be able to perform some or all of the following operations (including various operations described in detail later): acquiring information related to one or more of the following in a virtual space: a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar; determining whether the acquired information satisfies a predetermined condition; and, if it is determined that the predetermined condition is satisfied, generating information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose corresponding to the predetermined condition (including various operations described in detail later). In addition, such operations may be performed by a user giving an instruction to a system of an example of the invention disclosed in the present application documents using the input device 24 or the input device 34 related to the terminal device 30 described later. In addition, when the program is executed on the calculation device 21, it may be displayed by the output device 26 of the server device 20 as a system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as a system used by the user.

2-2. Hardware Configuration of Terminal Device 30 An example of the hardware configuration of the terminal device 30 will be described with reference to Fig. 2. The hardware configuration of each terminal device 30 may be the same as that of each server device 20 described above. Therefore, the reference characters for the components of each terminal device 30 are shown in parentheses in Fig. 2.

As shown in FIG. 2, each terminal device 30 can mainly include a calculation device 31, a main memory device 32, an input/output interface device 33, an input device 34, an auxiliary memory device 35, and an output device 36. These devices are connected to each other by a data bus and/or a control bus.

The arithmetic device 31, the main memory device 32, the input/output interface device 33, the input device 34, the auxiliary memory device 35, and the output device 36 may be substantially the same as or different from the arithmetic device 21, the main memory device 22, the input/output interface device 23, the input device 24, the auxiliary memory device 25, and the output device 26 included in each of the above-mentioned server devices 20. Furthermore, the capacity and capabilities of the arithmetic device and the memory device may be different. The input device 34 and the output device 36 in particular will be described in detail below.

In such a hardware configuration, the calculation device 31 sequentially loads the instructions and data (computer program) constituting a specific application stored in the auxiliary storage device 35 into the main storage device 32, and by calculating the loaded instructions and data, it is possible to control the output device 36 via the input/output interface device 33, or to send and receive various information to and from other devices (such as each server device 20) via the input/output interface device 33 and the communication network 10.

The terminal device 30 may be equipped with such a configuration, and may perform some or all of the following operations (including various operations described in detail later): acquiring information related to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a facial color of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space; determining whether the acquired information satisfies a predetermined condition; and, if it is determined that the predetermined condition is satisfied, generating information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose corresponding to the predetermined condition. Also, by executing an installed web browser, it is possible to receive and display a web page from the server device 20 and perform the same operation. Also, such operations may be performed by a user giving an instruction to a system of an example of the invention disclosed in the present application documents using the input device 34. Also, when the program is executed on the computing device 31, it may be configured to display the information on the output device 36 of the terminal device 30 as a system used by the user.

In the present application, there may be two types of terminal devices, a "distributor terminal" and a "viewer terminal". These may be names given to the roles of the terminals, and a terminal device may be a viewer terminal, a distributor terminal, or a viewer terminal and a distributor terminal at the same time. A distributor terminal may be a terminal used by a distributor who appears as a subject such as an avatar in the distributed content. On the other hand, a viewer terminal may be a terminal used mainly by a viewer who watches the distributed content. Viewers may also have a certain influence on the distributed content by gifting gifts to subjects such as avatars. Similarly, viewers may have a certain influence on the distributed content in the form of voice, comments, etc. The distributed content may be a video, video, or animation, and it is sufficient that a subject such as an avatar played by the distributor appears in some of these. The above description is merely an example of application of one embodiment of the present invention, and each embodiment of the present invention may be applied to a virtual space in which subjects without such a premise appear.

As described above, the viewer terminal may be configured to allow voice input, comment input, gifting, and the like. For example, the user terminal may have an input device 34 such as a keyboard or touch panel that can input comments and acquire comments as text information. The user terminal may also have a function for generating certain comments using the computing device 34. These acquired comments may similarly be transmitted to the communication network 10 via the input/output interface device 23.

Note that the comments according to the present invention may be comments that are automatically generated by the system, in addition to comments that are input by a user terminal. Such comments may be sent to the user terminal and displayed on the user terminal.

The terminal may also include a microphone capable of acquiring sound as the input device 34. Human voice may be acquired by the microphone as the input device 34 and treated as sound, or may be acquired by the microphone as the input device 34 and then converted into text through voice recognition by the calculation device 31 and treated as a comment. The user terminal may generate sound by the calculation device 31. These acquired sounds or comments may be transmitted to the communication network 10 via the input/output interface device 23 in the same manner.

On the other hand, the broadcaster terminal may include a motion sensor capable of detecting human movement as the input device 34. The motion sensor may be attached to the broadcaster's body and have a function of detecting the movement of each part of the body. The parts of the body may include various parts such as the head, face, upper arms, forearms, hands, thighs, lower legs, feet, and torso, but are not limited to some or all of these, and may be attached in a position where the movement of the body can be detected. Such a sensor may track the facial expression and pose of the broadcaster and generate information. The motion sensor may have an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The acquired information may be transmitted to the communication network 10 via the input/output interface device 23 in the same manner.

The broadcaster terminal may also include a camera capable of capturing video as the input device 34. The camera may capture images of the broadcaster's head, face, eyes, part of the body, whole body, etc. The camera may be capable of detecting the direction of the face when capturing images of the head or face, of detecting facial expressions when capturing images of the face, of detecting the line of sight when capturing images of the eyes, and of detecting body movement together with or in place of the motion sensor described above when capturing images of part or the whole body. Information obtained about the face direction, line of sight, facial expressions, and body movement based on these imaging information may similarly be transmitted to the communication network 10 via the input/output interface device 23.

In addition, the output device 36 in the viewer terminal and the distributor terminal may be a display device or a sound generating device. The display device may be a variety of devices capable of displaying, such as, but not limited to, a liquid crystal display, a plasma display, an organic EL display, a micro LED display, and a head mounted display. The sound generating device may be a speaker, a headphone, an earphone, etc. The information output by the output device 36 may be information generated by the calculation device 31, or may be information received from the communication network 10.

3. Functions of Each Device Next, an example of functions of each of the server device 20 and the terminal device 30 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of functions of the system shown in Fig. 1. As shown in Fig. 3, the example system may have an acquisition unit that acquires information that can be used to generate an object, and a generation unit that generates an object using the acquired information.

2.1 Acquisition Unit The acquisition unit has a function of acquiring information that can be used to determine whether or not a certain condition, which will be described later, applies.

The information acquired by the acquisition unit may include information relating to one or more of the following in the virtual space: a specific action of the avatar performed by the distributor, the facial expression of the avatar, the complexion of the avatar, the voice of the avatar, or a specific situation that occurs in relation to the avatar (sometimes referred to as "specific action" in the present application documents).

The specific information acquired by the acquisition unit will be explained in the condition application unit. Note that FIG. 4 shows an example of part of the information acquired by these acquisition units.

In addition, if a gift is provided to a specific avatar, the acquisition unit may acquire information identifying the avatar to which the gift was provided.

The information processing device from which the acquisition unit acquires information may be the same as the information processing device having the function of the acquisition unit, or it may be different. The term "acquisition" may be a higher-level concept meaning receiving information from another information processing device, or may be a higher-level concept meaning reading information from a storage device, or may be a higher-level concept meaning receiving information from another program via a storage device.

The condition application unit has a function of determining whether the acquired information satisfies a predetermined condition, and when it is determined that the predetermined condition related to the acquired information is satisfied, generating information for changing the facial expression or pose of the avatar related to the distributor, which has been generated based on the facial expression or pose related to the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition.

Here, generating information for changing the facial expression or pose of the avatar of the distributor, which was generated based on the facial expression or pose of the distributor, to a predetermined facial expression or pose corresponding to the predetermined condition may mean generating only information for changing the facial expression of the avatar of the distributor, which was generated based on the facial expression of the distributor, to a predetermined facial expression corresponding to the predetermined condition when the predetermined condition is satisfied, generating only information for changing the pose of the avatar of the distributor, which was generated based on the pose of the distributor, to a predetermined pose corresponding to the predetermined condition, or generating information for changing the facial expression and pose of the avatar of the distributor, which was generated based on the facial expression and pose of the distributor, to a predetermined facial expression and pose corresponding to the predetermined condition. In the present application, the predetermined facial expression and/or pose specified in response to the predetermined condition may be referred to as a predetermined facial expression, etc., and the facial expression and/or pose may be referred to as a facial expression, etc.

The predetermined facial expression may be an expression displayed using predetermined parameters of facial expressions, or parameters of facial expressions based on the broadcaster's facial expression corrected based on a predetermined rule, or an expression displayed by setting a predetermined shape or movement, or a pose displayed using predetermined pose parameters, or parameters of poses based on the broadcaster's pose corrected based on a predetermined rule, or a pose displayed by setting a predetermined shape or movement. Here, the predetermined parameters of facial expressions refer to a plurality of parameters that affect the movement of each part of the face, and one facial expression may be displayed by setting each of the plurality of parameters to a predetermined value. Such one or more parameters may be parameters that affect the movement of each part of the face, and each parameter may be for a specific part of the face, or may be for the entire face (or a part of the face) but the shape of the entire face (or a part of the face) may differ for each parameter. The latter is a technique in which a plurality of objects with the same area but different shapes are prepared, and one facial expression is expressed by combining a part or a plurality of them. An example of the latter is blendshape, in which multiple different shapes of the whole face are prepared in advance and a single facial expression is generated by synthesizing the multiple shapes. In addition to the normal facial elements, the present technology may also provide a special facial expression by preparing a predetermined shape in advance. For example, if a shape showing vertical lines on the face is prepared in advance, such a vertical line on the face can be produced, and if an element for producing cross eyes is prepared in advance, such a cross eye can be produced. In this case, since the parameters are set in advance, the facial expression is reproduced regardless of the broadcaster's facial expression, and there is an advantage that an easy-to-understand facial expression can always be reproduced when appropriate parameters for the facial expression are selected. In this case, the predetermined facial expression may be set differently depending on the avatar. In other words, the predetermined facial expression applied when the same predetermined condition is satisfied for avatar A and avatar B may be different for avatar A and avatar B. In this case, it may be set for each avatar, or it may be set according to the type of avatar.

On the other hand, if the predetermined facial expression is a parameter of a facial expression based on the facial expression of the streamer, which is corrected based on a predetermined rule, the predetermined facial expression may be based on the facial expression of the streamer. For example, assume that the streamer has a surprised expression at some gift. In this case, the facial expression of the avatar is set to an expression corresponding to surprise by one or more parameters in response to the detection of the streamer's facial expression, but the value of the multiple parameters related to surprise is increased or decreased by a predetermined value as the predetermined facial expression corresponding to a predetermined condition, so that the surprised expression is emphasized. While the above-mentioned technology is absolute in that one or more parameters are determined as the predetermined facial expression, the present technology may use parameters relative to information obtained from the current facial expression of the streamer. Therefore, even when the same avatar played by the same streamer is subjected to the same predetermined conditions, the parameters of the facial expression based on the facial expression of the streamer at that time may be the same or different, so that the predetermined facial expression may be the facial expression according to such different parameters. Therefore, it is possible to provide more variation in facial expressions.

Facial expressions displayed by setting a predetermined shape or movement may be ones in which a predetermined shape, such as a mask, is itself the specified facial expression.

Note that while the above describes predetermined facial expressions, the same technology may be used for predetermined poses as well.

The predetermined facial expression, etc. may be an facial expression, etc. that is related to a predetermined action, etc. that is being performed. In this case, there is an advantage that the viewer can easily understand the reason why the predetermined facial expression, etc. is being expressed by associating it with the predetermined action, etc.

The predetermined facial expression, etc. may affect the parameters for all parts that make up the facial expression, etc., or may affect only some of the parameters. For example, the predetermined facial expression, etc. may only affect the parameters for the mouth, and the parameters for parts other than the mouth may be parameters that are linked to the broadcaster's facial expression and are parameters based on information corresponding to the broadcaster's facial movements from the broadcaster's terminal. The predetermined facial expression, etc. may also include the appearance of the avatar. For example, the predetermined facial expression, etc. may include a performance in which the avatar is charred.

The predetermined facial expression, etc. may also change in conjunction with the direction of the predetermined motion, etc. In the virtual space, the broadcaster may not know where the object, etc. described below appears. In this case, the broadcaster may not notice the location where the object, etc. appears, and as a result, the avatar related to the broadcaster may not be able to react to the object, etc. For example, the broadcaster may not notice when a ghost character appears or a basin falls from above. Therefore, there is an advantage that the predetermined facial expression, etc. related to the avatar changes in conjunction with the direction in which the predetermined motion, etc. is displayed, so that the viewers can become more interested in the avatar's behavior. Here, there may be various modes in which the facial expression, etc. changes in conjunction with the direction of the predetermined motion, etc. For example, the predetermined facial expression, etc. set may be directed toward the direction in which the ghost character has come into contact or the direction in which the zombie character is approaching. When the predetermined facial expression, etc. is directed, some parts of the avatar may change in conjunction with the avatar, or multiple parts of the avatar may change in conjunction with the avatar. For example, when a certain facial expression is shown, the entire body may be facing in that direction, only the head may be facing in that direction, or only a part of the face, such as the eyes, may be facing in that direction.

The predetermined pose may be a variety of poses. The pose may be only a part of the avatar's body, or the avatar's entire body. For example, it may be a pose of only the head, such as a tilted head pose, a pose of only the hands, including hands that are enlarged or red and swollen, a pose of only the upper body, such as a fist pump, or a pose of the entire body, such as a gun-holding pose or a jumping pose.

The specified condition may be information related to one or more of the following: a specified action of an avatar performed by the distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a specified situation occurring regarding the avatar.

The predetermined action of the avatar performed by the distributor may be a predetermined action that is perceived by the avatar through the five senses if the avatar is a living being. The predetermined action of the avatar may also include an action of looking at a predetermined object. For example, if the avatar is surprised at seeing something, a surprised expression may be set as the predetermined facial expression, etc. In this case, an object that naturally makes the avatar look surprised may be set in advance, and if such an object is on an extension of the line of sight, it may be determined that the avatar has seen such an object. In other words, if the predetermined action includes an action of looking at the predetermined object, the predetermined condition may include moving the line of sight of the avatar so that the predetermined object is in the line of sight of the avatar, or the predetermined object being present in the line of sight of the avatar. In the former case, if the avatar changes its line of sight by moving its own viewing direction, if a surprising object is located on an extension of the line of sight, a surprised expression may be set as the predetermined facial expression, etc. The latter may be the case when a specific object is present, or when an avatar is looking in a specific direction and a specific object appears and is displayed in that direction.

In the above, we have described objects that bring about a feeling of surprise in the avatar, but they are not limited to surprise and may be objects that bring about other emotions or sensations. For example, they may be objects that cause a certain emotion or sensation when a person sees them, such as objects that cause a feeling of fear, such as ghosts, objects that cause a feeling of like, such as steak, or objects that cause a feeling of dislike, such as natto. In addition, as for the predetermined facial expression when seeing such objects, a predetermined facial expression that indicates a face that a person generally makes when seeing such objects may be set. The above-mentioned objects that cause a feeling of fear, objects that cause a feeling of like, and objects that cause a feeling of dislike may correspond to a facial expression of horror, a smile, an expression of disgust, etc.

In the above, the object to be viewed is described as an object, but the object to be viewed may be another character or another avatar. That is, when the predetermined object is another avatar or character, the predetermined action of the avatar may include the line of sight of the avatar meeting the line of sight of the predetermined object. In this way, when the line of sight of another character or another avatar meets, a predetermined facial expression may be set. When the lines of sight of the avatars meet, a predetermined facial expression may be set only for one avatar, or a predetermined facial expression may be set for both avatars. In the latter case, for example, a facial expression of excitement in which the lines of sight meet and both avatars turn red may be considered. In addition, in the latter case where a predetermined facial expression is set for both avatars, the set facial expressions may be the same predetermined facial expression, or may be different predetermined facial expressions. The reason for the different predetermined facial expressions may be a predetermined facial expression according to the type to which the avatar belongs, a predetermined facial expression according to the preferences or characteristics of the avatar, or a predetermined facial expression selected probabilistically. As an example of a case where an avatar makes eye contact with another character, when the eye contact between the avatar and a ghost character is made, the avatar's face may turn pale as a predetermined facial expression for the avatar. Whether or not such a predetermined condition is satisfied may be determined by determining whether or not there is an intersection between the direction of the avatar's eye contact and an object such as a ghost character.

In the above, the object to be viewed has been described as an object, but the object to be viewed may also be some kind of event. The scale and subject of the event may vary and is not limited. For example, the event may be a scary situation such as a swarm of crickets charging at you, being swept away by a flooded river, or being hit by a strong wind, or it may be a painful situation such as another character or avatar falling or bleeding. Even in the case of such an event, the avatar who has viewed the event may be set to a corresponding predetermined facial expression, etc.

The predetermined action of the avatar may include an action of touching a predetermined target. And, when the predetermined action includes an action of touching the predetermined target, the predetermined condition may include a distance between a position related to the avatar and a position related to the predetermined target being within a predetermined distance. Here, the position related to the avatar may be any one of the position of the avatar in a virtual space, a part of the body that activates the predetermined facial expression, a predetermined part of the avatar, or the position of a collider of the avatar.

Here, the action of contacting refers to the manner in which the avatar makes contact, whether proactively or passively, but does not refer to the manner in which a third party makes contact. The manner in which a third party makes contact, whether proactively or passively, is included in the "predetermined situation" that arises with respect to the avatar. Note that "passive" includes, for example, cases in which contact occurs as a result of the objective movement of the avatar or third party, regardless of the will of the avatar or third party.

As an example of an avatar's contact action, for example, if the object the avatar touches is hot, the avatar may make a surprised expression, or a specified expression in which the part touched is swollen due to the heat. More specifically, for example, if the avatar touches something hot with its hand, the specified expression may be one in which the hand grows larger and becomes red and swollen.

The predetermined action of the avatar includes the eating and drinking action of a predetermined target. Here, the predetermined eating and drinking action may be predetermined as the eating and drinking action of the avatar, and may be detected by an ID indicating such eating and drinking action. The ID indicating such eating and drinking action may be an ID given to a specific movement predetermined as an eating and drinking action, or may be an ID given to a predetermined movement based on one or more motion data. In the present invention, the technical significance of incorporating the eating and drinking action as a predetermined action is that a predetermined facial expression, etc. is set by combining it with other information described later. For example, even if it is the same eating and drinking action (e.g., the action of eating meat), when a certain avatar A eats meat, a predetermined facial expression, etc. expressing a refined way of eating is set, while when another avatar B eats meat, a predetermined facial expression, etc. expressing a wild way of eating is set. Here, factors that may cause avatar A and avatar B to be different in appearance may include, for example, avatar A and avatar B being different types of avatars, or even if the avatars are the same type, predetermined facial expressions expressing different eating styles may be set according to the set tastes and preferences, etc. Such an avatar type may be the personality that was set when the avatar was set, or may be determined from the parts and clothing that the avatar is equipped with at the time that the specified conditions are determined.

The predetermined action of the avatar may include a predetermined gesture in response to a smell or odor. Examples of the predetermined gesture include a gesture of sniffing the nose, a gesture of moving the air in front of the nose from side to side with the hand, and the like. Such gestures may be detected, for example, by using an optical sensor, an inertial sensor, or the like to detect the movement of the nose or the movement of the right or left hand. That is, the predetermined condition may be detection of the predetermined gesture of the avatar. Examples of the corresponding predetermined facial expressions include a predetermined facial expression that creates a sense of ecstasy, a predetermined facial expression that shows agony, and the like. This has the advantage that the action performed by the broadcaster when he or she senses a certain odor or smell allows the viewer to more clearly understand the significance of the action. Note that by sharing specific gestures corresponding to such odors or odors in advance between the broadcaster and the system, the predetermined facial expressions corresponding to such gestures can be more effectively created.

The predetermined action of the avatar may include a predetermined gesture for listening to/hearing a sound. Possible predetermined gestures for listening to/hearing a sound are, for example, a gesture of bringing a hand up to an ear, or a gesture of moving one ear in a certain direction. Predetermined facial expressions corresponding to such gestures include, for example, a gesture of bringing a hand up to an ear, in which the hand becomes larger, and for a gesture of moving one ear in a certain direction, in which the ear becomes larger and moves.

The specified condition including the facial expression of the avatar played by the streamer may be, for example, a specified facial expression when the avatar's facial expression is one or more specific parameters. In addition, even if the specified facial expression is not always the case, the specified facial expression may be the case when other specified conditions are met. For example, even if a specified facial expression is not normally set as a facial expression of surprise for the streamer, the specified facial expression may be set when the streamer shows a surprised facial expression when a specific event occurs.

The facial color of the avatar played by the distributor may include changes in facial color. For example, if the facial color of the avatar becomes different from normal, the facial expression of the avatar may be set to a different predetermined facial expression to match the changed facial color.

The voice of the avatar played by the distributor may include a change in tone quality of the avatar's voice beyond a predetermined range, or a change in volume of the avatar's voice beyond a predetermined amount. For example, it is conceivable that the avatar's voice may have a different voice quality than usual due to nervousness, anger, etc. In the case of such a voice, a predetermined facial expression or the like corresponding to the voice may be set.

The predetermined situation occurring with respect to the avatar may include the occurrence of a predetermined sound in the virtual space. The predetermined sound may be, for example, a loud sound. In response to such a loud sound, the predetermined facial expression may be a surprised facial expression or a pose that expresses surprise with the whole body, such as jumping up.

The predetermined situation occurring with respect to the avatar may include a distance between a position related to a predetermined object and a position related to the avatar falling within a predetermined range. Here, the position related to the predetermined object may be, for example, a point that is the center of gravity of the predetermined object, a point that is the center of the predetermined object, a center of a circumscribing circle of the predetermined object, a center of an inscribing circle of the predetermined object, a center of a circumscribing circle on a display of the predetermined object, a center of an inscribing circle on a display of the predetermined object, or the like. The position related to the predetermined object may also be an intersection point of a tangent of a straight-line distance between a circle or polygon related to the predetermined object and a position related to the avatar with the circle or polygon. The position related to the predetermined object may be determined by setting a collider for the predetermined object and also for the avatar, and using the collider to perform contact determination. The position related to the avatar may be as described above. If the position related to the avatar is the position of a collider, it may be determined that the predetermined object and the avatar are in contact.

The specified object may be any of the various objects described above. For example, if a bucket falls from above and touches the head, the eyes may be set to be crossed out (X) as a specified facial expression, etc. Also, if lightning strikes (touches) the avatar, the entire body may be set to be charred as a specified facial expression, etc.

In addition, instead of the predetermined object, other things may be used, such as a character or another avatar. In the case of a character, for example, a predetermined situation occurring with respect to the avatar may include a situation in which the distance between the position of the zombie character and the position of the avatar is within a predetermined range. In this case, if the distance is within the predetermined distance, vertical lines may be added to the face of the avatar. In addition, the predetermined facial expression, etc. may change depending on the extent of the distance. For example, if the distance changes to a first distance and a second distance that is shorter than the first distance, the predetermined facial expression, etc. at the first distance may be different from the predetermined facial expression, etc. at the second distance. For example, the predetermined facial expression, etc. at the first distance may be 10 vertical lines in the facial expression, while the predetermined facial expression, etc. at the second distance may be 20 vertical lines in the facial expression and the width of the vertical lines is doubled. Conversely to the above, the predetermined condition may include a situation in which the position of the avatar and the position of the character or avatar are outside the range of the predetermined distance. In this case, a situation is shown in which the avatar or character moves relatively away from the avatar, but for example, a facial expression such as tears may be set as a specified facial expression for the avatar.

For example, a predetermined situation occurring with respect to an avatar may include a case where a predetermined relationship occurs between a first avatar and a second avatar. For example, when a first avatar strokes the head of a second avatar, the eyes of the second avatar may be set to hearts as a predetermined facial expression, etc. In this case, a collider may be set for each of the first avatar and the second avatar, and the collision may be judged.

The predetermined situation that occurs with respect to the avatar may include a predetermined combination of a predetermined object and a predetermined pose of the avatar. Here, the predetermined object may be an object, a predetermined situation, or a predetermined event. The object may include a gift. For example, when the avatar is equipped with a gift of a pistol as an object and takes a shooting pose, a brave expression may be set as the predetermined expression. Here, the shooting pose may be determined by setting a collider from the center position of the avatar and determining the collision of each position of the body such as the right hand, left hand, right foot, left foot, and head. Here, the positions of the right hand, left hand, right foot, left foot, head, etc. of each streamer may be detected by an optical sensor or an inertial sensor. In this case, the predetermined expression, etc. set may be suggested by the combination of the given gifts. For example, if the shield of a frog, the matter of a frog, and the cape of a frog are used, the face may look like a frog.

In addition, among the multiple conditions according to the present invention, multiple conditions may be satisfied. For example, the avatar and the ghost character make eye contact and the distance between the position of the avatar and the position of the ghost character is within a predetermined distance range. In such a case, if the predetermined expressions, etc. corresponding to the conditions of eye contact and distance within the range are different, processing may be defined when multiple predetermined conditions are satisfied. For example, if a first predetermined condition is satisfied and a second predetermined condition is satisfied within a predetermined time, a predetermined expression or pose corresponding to the first predetermined condition or the second predetermined condition that has a higher priority may be applied. Here, the priority may be associated with each predetermined condition and stored, or may be set as a rule when multiple predetermined conditions are satisfied. For example, when there are predetermined conditions A1, A2, A3, A4, etc., it is sufficient to be able to select a predetermined condition that is prioritized after comparing each predetermined condition. The advantage of using such a priority is that it allows for accurate processing even when multiple predetermined conditions are satisfied, especially when multiple predetermined conditions are satisfied at almost the same time. Furthermore, such priority may be expressed as probability. For example, when the probability is determined by a pseudorandom function or the like, and a first predetermined condition and a second predetermined condition are satisfied, if the pseudorandom function value is a predetermined value, a predetermined facial expression or the like corresponding to the first predetermined condition is applied, and if the value is another predetermined value, a predetermined facial expression or the like corresponding to the second predetermined condition is applied. In this case, the variety of presentations increases, which may further attract the interest of the viewer.

In addition, since the judgment of the multiple conditions that can be satisfied is made after waiting for the passage of a predetermined time, if the predetermined time is set to a certain time or more, there is a possibility that a time lag will occur between the satisfaction of the predetermined condition and the presentation of the predetermined facial expression, etc. Therefore, without setting such a priority, when the above-mentioned multiple conditions are satisfied, the predetermined facial expression, etc. corresponding to the predetermined condition that was satisfied first in time may be set. Here, the time precedence may mean that the processing unit judges the satisfaction of the predetermined condition first, even when the predetermined conditions are substantially satisfied almost simultaneously as a result of a network or information processing. In this case, there is an advantage that the predetermined facial expression, etc. corresponding to the predetermined condition that was satisfied first can be set without waiting for the passage of a predetermined time. In addition, in such a case, the predetermined facial expression, etc. corresponding to the predetermined condition that was satisfied later may or may not be set. If it is set, the predetermined facial expression, etc. corresponding to the predetermined condition that was satisfied first will be overwritten, but there is an advantage that the predetermined facial expression, etc. corresponding to the predetermined condition that was satisfied later can also be presented to the viewer. On the other hand, if it is not set, there is an advantage that the viewer can fully enjoy the predetermined facial expression, etc. that was satisfied first. In this way, when both the first and second predetermined conditions are satisfied, a predetermined facial expression, etc. may be set according to the timing when the first predetermined condition is satisfied and the timing when the second predetermined condition is satisfied. For example, when the time between the first timing when the first predetermined condition is satisfied and the second timing when the second predetermined condition is satisfied is shorter than a predetermined time, only the first predetermined facial expression, etc. corresponding to the first predetermined condition may be produced, and the second predetermined facial expression, etc. corresponding to the second predetermined condition may not be produced. In this case, there is an advantage that the viewer can enjoy the production of the first predetermined facial expression, etc. without changing it midway. On the other hand, when the time between the first timing and the second timing is longer than a predetermined time, a predetermined facial expression, etc. corresponding to the second predetermined condition may be set at the timing when the second predetermined condition is satisfied. In this case, there is an advantage that the viewer can enjoy both the production of the second predetermined facial expression, etc., following the first predetermined facial expression, etc.

The above-mentioned predetermined time for determining the time between the first timing and the second timing may be set according to the set of predetermined conditions that are satisfied. For example, it is assumed that the first predetermined facial expression, etc. is produced when the first predetermined condition is satisfied, the second predetermined facial expression, etc. is produced when the second predetermined condition is satisfied, and the third predetermined facial expression, etc. is produced when the third predetermined condition is satisfied. Also, it is assumed that the first predetermined condition is satisfied at the first timing, the second predetermined condition is satisfied at the second timing, and the third predetermined condition is satisfied at the third timing. At this time, it is assumed that if the time between the first timing and the second timing is shorter than the 12th predetermined time, the second predetermined facial expression, etc. is not produced, and if it is equal to or longer than the 12th predetermined time, the second predetermined facial expression, etc. is produced. Also, it is assumed that if the time between the first timing and the third timing is shorter than the 13th predetermined time, the third predetermined facial expression, etc. is not produced, and if it is equal to or longer than the 13th predetermined time, the third predetermined facial expression, etc. is produced. In this case, the twelfth and thirteenth predetermined times may be the same or different. If they are the same, it means that the presentation time of the first predetermined facial expression, etc. will be the same whether the second predetermined condition is satisfied or the third predetermined condition is satisfied after the first predetermined condition is satisfied. On the other hand, if they are different, the presentation time of the first predetermined facial expression, etc. will change between when the second predetermined condition is satisfied and when the third predetermined condition is satisfied after the first predetermined condition is satisfied. In this case, the variety of presentations will increase, which may further attract the viewer's interest.

In the above, the priority is associated with a specific condition, or an example of a rule for a specific condition that is satisfied is described. However, the priority may also be associated with an object, event, situation, etc. within the specific condition. For example, in the case where a gift is used under a specific condition, the gift and the priority may be associated.

The priority of an object, event, or situation (sometimes referred to as an object, etc. in the present application) related to a specified condition may be generated using the size at which the object, etc. is displayed, the location at which the object, etc. appears, the timing at which the object, etc. appears, etc.

For example, when it comes to the displayed size of an object, if a second object is displayed larger than a first object, a higher priority may be set for the second object than for the first object. In this case, the larger object will attract more attention from the viewer, so giving priority to it is considered to be in line with the viewer's wishes. On the other hand, the opposite may also be set. Setting it the other way around has the advantage of creating an element of surprise.

In addition to the size of the object, if the object has a price, the price may be used, or if the object has consumption points, the consumption points may be used. For example, if the price of the second object is higher than the price of the first object, a higher priority may be set for the second object than the first object. In this case, the higher the price, the more attention it attracts from the viewers, so it is considered that giving priority to the object is in line with the viewers' intentions. On the other hand, the opposite may be set. If the opposite is set, it has the advantage of creating an element of surprise. Similarly, for consumption points, if the consumption points of the second object are higher than the consumption points of the first object, the second object may be set to a higher priority than the first object. In this case, the higher the consumption points, the more attention it attracts from the viewers, so it is considered that giving priority to the object is in line with the viewers' intentions. On the other hand, the opposite may be set. If the opposite is set, it has the advantage of creating an element of surprise.

As for the location in the virtual space where an object etc. appears, for example, an object etc. that appears in the center of the virtual space may be set to have a higher priority than an object etc. that appears on the left or right edge of the virtual space. If the priority of an object etc. that appears in a conspicuous place in the virtual space is set high, the viewer can bring the object etc. that appears in the conspicuous place into view, which has the advantage that the viewer can more easily understand the meaning of a certain facial expression etc.

The priority of an object may be static information that is set in advance for the object, or may be dynamic information that is set only when the object is provided. The latter may be information that is set according to the position where the object appears. For example, the priority may be set according to the relationship between the position where the object appears in the virtual space and the position of the avatar. In this case, the distance itself may play the role of the priority, or the distance may be used as one piece of information for setting the priority. For example, in a case where the distance between the avatar and the position of the first object that appeared is a first distance, and the distance between the avatar and the position of the second object that appeared is a second distance, if the first distance is longer than the second distance, the priority of the first object may be set higher than the second object. In addition, if the first distance between the center of the screen and the position of the first object is shorter than the second distance between the center of the screen and the position of the second object, the priority of the first object may be set higher than the second object because the first object appears closer to the center of the screen. When the priority of items etc. is dynamically set, there is an advantage that gifts with specific expressions etc. can be selected more appropriately and dynamically.

Also, the type to which the avatar belongs may be used as information related to the avatar. For example, the type to which the avatar belongs, which is one of a plurality of types to which the avatar belongs, may be used as the predetermined condition. This may be such that, when certain information is acquired, a predetermined facial expression, etc. is obtained for a first avatar type, but a predetermined facial expression, etc. is not obtained for a second avatar type. For example, a certain type of avatar may be set to have a timid personality, and when the distance between the ghost character and the avatar is within the first distance range, the avatar is more likely to have a scared facial expression. Also, a certain type of avatar may be set to have a cheerful personality, and the condition for a facial expression including a smile as the predetermined facial expression, etc. may be set to be looser than other avatars. In this case, the avatar may be provided with a personality characteristic such as timid, cheerful, bullish, or competitive, and the satisfaction of the predetermined condition may be determined using the personality characteristic. Note that the personality may be set according to the type to which the avatar belongs, or may be set for each avatar separately from the type to which the avatar belongs.

The predetermined condition may include a probability. In other words, the predetermined condition may include a pseudorandom number function, etc., and a predetermined facial expression, etc. may be generated when the value of such a function satisfies the predetermined condition. When such a probability is included, the predetermined facial expression, etc. is not always generated, so it is possible to continuously attract the interest of the viewer. Also, for example, when the probability value is lower than a predetermined value, and a rare facial expression is generated, it is possible to attract the interest of the viewer in that the viewer can see a rare facial expression.

Fulfillment of the specified condition may also be determined using the timing of the display of individual objects, such as information on the start of the display of the object, information on a change in the display of the object, and/or information on the end of the display of the object. In this case, the facial expression and pose of the avatar change in response to the display of the object, such as the start, change, and end of the display of the object, which has the advantage that the viewer can easily understand the meaning of the change in the avatar's facial expression and pose in relation to the situation of the object.

In addition, information that an object has as an attribute may be used as the predetermined condition. The attributes of the object may or may not include characteristics that the object has, such as the type of the object, the level of the object, etc. In addition, predetermined conditions may be set so that the predetermined conditions are satisfied according to specific characteristics that the object has, and a corresponding predetermined facial expression, etc. may be set.

The specified conditions may also dictate the order or timing of the information obtained.

For example, with regard to a predetermined facial expression when a predetermined condition is satisfied, a predetermined facial expression A may be set if a bat character appears on the screen within a predetermined time after a ghost character appears on the screen, and a predetermined facial expression B may be set if a ghost character appears on the screen within a predetermined time after a bat character appears on the screen, and so on.

The condition application unit may set the facial expression, etc., specified by the satisfaction of a predetermined condition as the facial expression, etc., of the avatar to which the gift is provided. More specifically, the information indicating the facial expression, etc., specified by the satisfaction of a predetermined condition may be an ID indicating the specified facial expression, etc., or one or more parameters of the facial expression indicating the specified facial expression, etc., and the facial expression, etc. of the avatar may be set using the information indicating the specified facial expression, etc., so that the specified facial expression, etc., can be set as the facial expression of the avatar. After the information of the ID or parameters is specified or generated by the condition application unit, the information may be transmitted to a server or terminal for generating a video or animation, and may be used for a predetermined period of time. Here, the condition application unit may store information indicating the specified facial expression, etc., in association with each predetermined condition, and may be configured to specify information indicating the corresponding specified facial expression, etc., when each predetermined condition is satisfied. Note that after the specified facial expression, etc., applied by the satisfaction of a predetermined condition, is set for the avatar for a predetermined period of time, the original avatar's behavior may be restored. In other words, since the avatar is played by the distributor, after a certain period of time, the facial expression and pose of the avatar may be based on the information of the distributor's facial expression and pose. Here, based on the information of the distributor's facial expression and pose may mean that the avatar's movements are determined by information on the facial expression and pose of the distributor detected by an optical sensor or an inertial sensor.

In such processing, if the first information processing device on which the condition application unit is executed is the same as the second information processing device which performs processing including setting the facial expression, etc. of the avatar, the condition application unit may process the processing so as to set the facial expression, etc. of the avatar using the facial expression identified by the condition application unit upon satisfaction of a predetermined condition.

In addition, when a first information processing device on which the condition application unit is executed is different from a second information processing device which performs processing including setting the facial expression of the avatar, the first information processing device may transmit the facial expression, etc., identified by the condition application unit upon satisfaction of a predetermined condition to the second information processing device, and the second information processing device may process the identified facial expression, etc., to be set as the facial expression of the avatar.

Here, when the first information processing device is a server and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the server of the second information processing device. Also, when the first information processing device is a server and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the viewer terminal of the second information processing device. Also, when the first information processing device is a server and the second information processing device is a broadcaster terminal, information indicating the specified facial expression, etc. may be transmitted from the server of the first information processing device to the broadcaster terminal of the second information processing device.

Similarly, when the first information processing device is a viewer terminal and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the server of the second information processing device. Furthermore, when the first information processing device is a viewer terminal and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the viewer terminal of the second information processing device. Furthermore, when the first information processing device is a viewer terminal and the second information processing device is a distributor terminal, information indicating the specified facial expression, etc. may be transmitted from the viewer terminal of the first information processing device to the distributor terminal of the second information processing device.

Similarly, when the first information processing device is a broadcaster terminal and the second information processing device is a server, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the server of the second information processing device. Furthermore, when the first information processing device is a broadcaster terminal and the second information processing device is a broadcaster terminal, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the broadcaster terminal of the second information processing device. Furthermore, when the first information processing device is a broadcaster terminal and the second information processing device is a viewer terminal, information indicating the specified facial expression, etc. may be transmitted from the broadcaster terminal of the first information processing device to the viewer terminal of the second information processing device.

In the above, the condition application unit is executed in a single information processing device, but the condition application unit may be distributed and processed in multiple information processing devices.

The condition application unit may set the predetermined facial expression, etc., using the timing at which the predetermined facial expression, etc. is set. The timing may be, for example, as described above, the timing when the avatar meets the gaze of the avatar or character, the timing when the avatar comes into contact with an object, etc. In this case, information including information indicating the predetermined facial expression, etc., may be transmitted to an information processing device on which the information is displayed, and displayed. The timing deviation in this case may include a time lag in communication or processing.

On the other hand, the timing at which the specified facial expression, etc. is set after the specified conditions are met may be set separately. For example, the specified facial expression, etc. may be set two seconds after the specified conditions are set. As one example, the specified surprised facial expression may be set two seconds after a situation occurs that surprises the avatar, which is an intentionally delayed timing.

Figure 5 is an example showing the relationship between a specified condition and a specified facial expression, etc. By using the acquired information as the specified condition, the corresponding specified facial expression, etc. can be identified.

As described above, when a certain condition is satisfied, the function of the condition application unit makes it possible to realize a corresponding predetermined facial expression, etc., which has the advantage of allowing for variation in the facial expressions and poses of the avatar in the virtual space. Furthermore, even if the performer is able to specify the facial expression of the avatar himself/herself using a tablet or the like, when a wearable gift is put on by an operator in a studio broadcast, the performer may not notice that it has been put on, and in such a case, there is the advantage that the facial expression or pose corresponding to the gift can be applied to the avatar without the performer having to perform any operation. In other words, the information for creating a predetermined facial expression or pose may be generated without any operation other than providing the gift from the broadcaster.

In addition, as mentioned above, the function of the condition application unit makes it possible to automatically realize a corresponding specified facial expression, etc., when a certain condition is met, which has the advantage that viewers can enjoy the reaction that results in the avatar's facial expression changing without any operation from the viewer's terminal.

4. Embodiments
4-1. Embodiment 1
The first embodiment is an example in which the condition application unit operates in the viewer terminal. In this case, the acquisition unit may operate in the viewer terminal. This example will be described with reference to FIG. 6. Since the condition application unit operates in the viewer terminal, it may be possible to change the expression of the avatar for each viewer terminal by including information based on the viewer terminal as a predetermined condition.

Step 1
The acquisition unit acquires information on a predetermined action, etc. (step 001). Such information on the predetermined action, etc. may be information inputted in a viewer terminal on which the condition application unit operates, or information inputted in another viewer terminal or information generated in a server as a performance may be acquired by the viewer terminal on which the condition application unit operates via the server, etc.

Step 2
Next, the condition application unit determines a predetermined condition using at least information on the predetermined motion, etc., and if it determines that the predetermined condition is satisfied, identifies a facial expression, etc. corresponding to the satisfied predetermined condition (step 002). For this purpose, the condition application unit may have information indicating at least some of the predetermined conditions and a predetermined facial expression, etc. when such a condition is satisfied.

Step 3
The specified facial expression, etc. is set in the viewer terminal as the facial expression, etc. associated with the avatar used to satisfy the predetermined condition (step 003). This has the advantage that the viewer can watch the reaction of the avatar as the facial expression, etc. of the avatar changes in response to the predetermined action, etc.

Step 4
At each viewer terminal, after a predetermined facial expression of the avatar is displayed for a predetermined period of time, the facial expression of the avatar returns to the facial expression based on the facial expression of the distributor associated with the original avatar (step 004).

4-2. Embodiment 2
The second embodiment is an example in which the condition application unit operates in the server. In this case, the acquisition unit may operate in the server. Note that this example will be described with reference to FIG. 7 on the premise that animation is generated on a viewer terminal.

Step 1
The acquisition unit acquires information on a predetermined action or the like (step 001).

Step 2
Next, the condition application unit determines the predetermined condition using at least information on the predetermined movement, etc., and if it determines that the predetermined condition is satisfied, identifies the facial expression, etc. that corresponds to the satisfied predetermined condition (step 002).

Step 3
The server transmits information indicating the identified facial expression, etc. to each viewer terminal viewing the distribution of the avatar (step 003).

Step 4
In each viewer terminal, the facial expression of the avatar is set using the information indicating the facial expression, etc. acquired from the server (step 004). Here, in each viewer terminal, an animation may be generated using the set facial expression, etc. of the avatar.

Step 5
In each viewer terminal, the facial expression of the avatar using the information indicating the facial expression obtained from the server is displayed for a predetermined period of time, and then the facial expression is set to the facial expression based on the facial expression of the original distributor associated with the avatar (step 005). Note that when an animation is generated, the generated facial expression is displayed for a predetermined period of time in each animation, and thereafter, the facial expression obtained by tracking based on the facial expression of the distributor associated with the avatar may be used as the facial expression of the avatar.

4-3. Embodiment 3
The third embodiment is an example in which the condition application unit operates in the server. In this case, the acquisition unit may operate in the server. Note that this example will be described with reference to FIG. 8 on the premise that a video including an avatar's facial expression is generated on the server.

Step 1
The acquisition unit acquires information on a predetermined action or the like (step 001).

Step 2
Next, the condition application unit determines the predetermined condition using at least information on the predetermined movement, etc., and if it determines that the predetermined condition is satisfied, identifies the facial expression, etc. that corresponds to the satisfied predetermined condition (step 002).

Step 3
The server generates a video for the distribution of the avatar using information indicating the specified facial expression, etc. (step 003). Here, when the server generating the video for the distribution of the avatar and the server executing the condition application unit are the same server, the server executing the condition application unit may generate the video for the distribution of the avatar. On the other hand, when the server generating the video for the distribution of the avatar and the server executing the condition application unit are different servers, information indicating the specified facial expression, etc. may be transmitted from the server executing the condition application unit to the server generating the video, and the information indicating the specified facial expression, etc. may be used in the server generating the video. Note that when an animation is generated in the server, an animation for the distribution of the avatar may be generated together with or instead of the video, and in this case, the animation may be used together with or instead of the video hereafter.

Step 4
An image of an avatar generated using the specified facial expression based on information indicating the specified facial expression, etc. is transmitted to each viewer terminal (step 004).

Step 5
At each viewer terminal, an image using information indicating facial expressions, etc. obtained from the server is displayed for a predetermined period of time, and then an image of an avatar based on facial expressions, etc. based on the facial expressions, etc. of the distributor associated with the original avatar is displayed (step 005).

4-4. Various embodiments The computer program according to the first aspect is "a computer program, which, when executed by one or more processors, acquires information related to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring regarding the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and when it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition as the facial expression of the avatar, and displays the avatar to which the predetermined facial expression or pose is applied."

The computer program according to the second aspect is the first aspect described above, in which "the predetermined movement of the avatar is a predefined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The computer program according to the third aspect is the one according to the first or second aspect, in which "the specified action of the avatar includes an action of looking at a specified target."

The computer program according to the fourth aspect is any one of the first to third aspects above, in which "if the predetermined target is another avatar or character, the predetermined action of the avatar includes meeting the line of sight of the avatar with the line of sight of the predetermined target."

The computer program according to the fifth aspect is any one of the first to fourth aspects above, in which "if the predetermined action includes an action of looking at the predetermined target, the predetermined condition includes moving the line of sight of the avatar so that the predetermined target is in the line of sight of the avatar, or the predetermined target being present in the line of sight of the avatar."

The computer program according to the sixth aspect is any one of the first to fifth aspects, in which "the specified action of the avatar includes an action of contacting a specified object."

The computer program according to the seventh aspect is any one of the first to sixth aspects above, in which "if the predetermined action includes an action of contacting the predetermined object, the predetermined condition includes that the distance between the position of the avatar and the position of the predetermined object is within a predetermined distance."

The computer program according to the eighth aspect is any one of the first to seventh aspects above, in which "the position related to the avatar is either the position of the avatar in a virtual space, a part of the body that produces the predetermined facial expression, a predetermined part of the avatar, or the position of a collider for the avatar."

The computer program according to the ninth aspect is any one of the first to eighth aspects, in which "the predetermined action of the avatar includes an eating or drinking action of a predetermined target."

The computer program according to the tenth aspect is any one of the first to ninth aspects, in which "the specified action of the avatar includes a smell or a specified gesture in response to a smell."

The computer program according to the eleventh aspect is any one of the first to tenth aspects, in which "the predetermined action of the avatar includes a predetermined gesture for listening to or/listening to a sound."

The computer program according to the twelfth aspect is any one of the first to eleventh aspects, in which "the predetermined condition uses detection of the predetermined gesture of the avatar."

The computer program according to the thirteenth aspect is any one of the first to twelfth aspects above, in which "the predetermined situation occurring with respect to the avatar includes the occurrence of a predetermined sound in the virtual space."

The computer program according to the 14th aspect is any one of the first to 13th aspects, in which "the predetermined situation occurring with respect to the avatar includes the distance between the position of a predetermined object and the position of the avatar being within a predetermined range."

The computer program according to the fifteenth aspect is any one of the first to fourteenth aspects, in which "the predetermined situation occurring with respect to the avatar includes a predetermined combination of a predetermined object and a predetermined pose of the avatar."

The computer program according to the sixteenth aspect is any one of the first to fifteenth aspects above, in which "the facial color of the avatar includes a change in facial color."

The computer program according to the seventeenth aspect is any one of the first to sixteenth aspects above, "including that the tone quality of the avatar's voice changes by more than a predetermined range, or the volume of the avatar's voice changes by more than a predetermined amount."

The computer program according to the 18th aspect is any one of the above 1st to 17th aspects, which "applies a predetermined facial expression or pose corresponding to the first or second predetermined condition, whichever is set as the higher priority condition, when a first predetermined condition is satisfied and a second predetermined condition is satisfied within a predetermined time period."

The computer program according to the 19th aspect is any one of the first to 18th aspects above, in which "information for achieving the specified facial expression or pose is generated without obtaining information for operating the avatar from the distributor terminal."

The computer program according to the twentieth aspect is "a computer program which, when executed by one or more processors, acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the predetermined facial expression or pose."

The computer program according to the twenty-first aspect is the computer program according to the twenty-first aspect, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The server device according to the twenty-second aspect "has one or more processors, and the processor executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generate information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmit the information for changing the facial expression or pose."

The server device according to the twenty-third aspect is the twenty-second aspect, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The method according to the twenty-fourth aspect is "a method executed by one or more processors that execute computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and displays the avatar to which the predetermined facial expression or pose has been applied."

The method according to the 25th aspect is the method according to the 24th aspect, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The method according to the 26th aspect is "a method executed by one or more processors that execute computer-readable instructions, which acquires information relating to one or more of a predetermined movement of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the distributor that was generated based on the facial expression or pose of the distributor to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the facial expression or pose."

The method according to the 27th aspect is the method according to the 26th aspect, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The viewer terminal device according to the 28th aspect "has one or more processors, and the processor executes computer-readable instructions to acquire information relating to one or more of a predetermined action of an avatar performed by a distributor, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar, in a virtual space, determine whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition related to the acquired information is satisfied, generate information for changing the facial expression or pose of the avatar related to the distributor that was generated based on the facial expression or pose related to the distributor to a predetermined facial expression or pose corresponding to the predetermined condition, and display the avatar to which the predetermined facial expression or pose has been applied."

The viewer terminal device according to the 29th aspect is the viewer terminal device according to the 28th aspect, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The broadcaster terminal device according to the 30th aspect "has one or more processors, and the processor executes computer-readable instructions to acquire information relating to one or more of a predetermined movement of an avatar performed by the broadcaster, a facial expression of the avatar, a complexion of the avatar, a voice of the avatar, or a predetermined situation occurring in relation to the avatar in a virtual space, determines whether the acquired information satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, generates information for changing the facial expression or pose of the avatar of the broadcaster that was generated based on the facial expression or pose of the broadcaster to a predetermined facial expression or pose that corresponds to the predetermined condition, and transmits the information for changing the facial expression or pose."

The distributor terminal device according to the 31st aspect is the 30th aspect described above, in which "the predetermined movement of the avatar is a predetermined movement that is perceived by the avatar through the five senses if the avatar is a living being."

The system and information processing described in this application may be implemented by software, hardware, or a combination of these. Some or all of the system and information processing may be implemented as computer programs and executed by various computers. These computer programs may be stored in storage media. These programs may be stored in non-transient or temporary storage media.

It goes without saying that what is described in this application document is not limited to what is described in this application document, but can be applied to various examples within the scope of various technical ideas having various technical advantages and configurations described in this application document.

REFERENCE SIGNS LIST 1 System 10 Communication network 20 (20A to 20C) Server device 30 (30A to 30C) Terminal device 21 (31) Arithmetic device 22 (32) Main memory device 23 (33) Input/output interface 24 (34) Input device 25 (35) Auxiliary memory device 26 (36) Output device 41 Acquisition unit 42 Condition application unit

Claims (9)

one or more processors,
A first means for acquiring a facial expression of a distributor;
A second means for setting a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space;
and a third means for, when a situation that satisfies any of a plurality of conditions occurs for the avatar in the virtual space, replacing the facial expression of the avatar set by the second means with a facial expression of the avatar that corresponds to the satisfied condition, and after a predetermined period of time has elapsed since the facial expression was set, setting for the avatar a facial expression corresponding to the facial expression of the distributor obtained by the first means. The program according to claim 1, wherein the third means, when a situation that satisfies any one of a plurality of conditions and occurs without regard to the will of the avatar or a third party, sets the facial expression of the avatar corresponding to the satisfied condition to the avatar in place of the facial expression of the avatar set by the second means, and after a predetermined period of time has elapsed since the facial expression was set, sets the facial expression of the avatar corresponding to the facial expression of the distributor obtained by the first means to the avatar. 3. The program according to claim 1 , wherein a situation that satisfies a first condition of the plurality of conditions is that the avatar comes into contact with a predetermined object, character, or another avatar in the virtual space. a situation that satisfies a second condition among the plurality of conditions is that the avatar comes into contact with a first object, a first character, or a first avatar in the virtual space;
a situation that satisfies a third condition among the plurality of conditions is that the avatar comes into contact with a second object different from the first object, a second character different from the first character, or a second avatar different from the first avatar in the virtual space;
The program according to claim 1 , wherein a facial expression of the avatar corresponding to the second condition is different from a facial expression of the avatar corresponding to the third condition. The program according to claim 1 or 2, wherein a fourth condition among the plurality of conditions is satisfied when the distance between the position of the avatar and the position of a specified object, character or avatar in the virtual space is within or outside a specified range. a fifth condition among the plurality of conditions being satisfied when, in the virtual space, a distance between a position of the avatar and a position of a first object, a first character, or a first avatar is within a first range or outside a predetermined range;
a situation in which a sixth condition among the plurality of conditions is satisfied is that a distance between a position of the avatar and a position of a second object different from the first object, a second character different from the first character, or a second avatar different from the first avatar, in the virtual space, is within a second range different from the first range or outside a predetermined range;
The program according to claim 1 , wherein a facial expression of the avatar corresponding to the fifth condition is different from a facial expression of the avatar corresponding to the sixth condition. The seventh condition among the above conditions is satisfied in the following situation:
When a predetermined relationship occurs between the avatar and another avatar in the virtual space,
In the virtual space, a pose of the avatar and a predetermined object are combined in a predetermined manner; and
The program according to claim 1 , further comprising at least one of: generating a predetermined sound in the virtual space. A first means for acquiring a facial expression of a distributor;
A second means for setting a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space;
and a third means for, when a situation that satisfies any of a plurality of conditions occurs for the avatar in the virtual space, setting for the avatar a facial expression for the avatar that corresponds to the satisfied condition instead of the facial expression for the avatar set by the second means, and after a predetermined period of time has elapsed since the setting, setting for the avatar a facial expression corresponding to the facial expression of the distributor obtained by the first means. A step of a first means acquiring a facial expression of a distributor;
A step in which a second means sets a facial expression corresponding to the facial expression of the distributor acquired by the first means to an avatar corresponding to the distributor in a virtual space;
a third means, when a situation occurs in the virtual space for the avatar that satisfies any of a plurality of conditions, setting, in the avatar's facial expression set by the second means, a facial expression for the avatar that corresponds to the satisfied condition, and after a predetermined period of time has elapsed since the setting, setting, in the avatar, a facial expression corresponding to the facial expression of the distributor obtained by the first means.

Images