JP6945375B2 - Image generator and program - Google Patents

Description

The present invention relates to an image generator and a program.

Conventionally, there is a well-known technique for arranging a non-existent two-dimensional or three-dimensional idol character in a virtual space and operating the character as if it were a real idol. As a technique related to this, Patent Document 1 states that the privacy of a person who emits a voice is protected by operating a virtual character in accordance with the input of voice data without displaying the face of the person who emits the voice. , An image generation system for transmitting emotions to others, and the like are disclosed.

Japanese Unexamined Patent Publication No. 2003-2488837

However, in the technique described in Patent Document 1, basically, the movement (including facial expression) of the character displayed on the display screen is only controlled according to the emotion reflected in the voice data. Unless the person who emits the character (the person himself / herself) actively acts by voice, it is difficult to increase the variation of the movement (facial expression in Patent Document 1) of the character.

The present invention has been made to solve the above problems, and an object of the present invention is to appeal to the viewer the character-likeness without the performer performing a special performance, and to give the viewer a strong impression of likes and dislikes for the character. It is an object of the present invention to provide an image generator and a program which can be given.

To solve the above problems
(1) The image generator of the present invention is
An image generation device that reflects the movements of a performer existing in real space on the movements of a target character existing in an object space and generates an image of the target character operating in the object space.
Character attributes associated with each of the plurality of characters, emotional attributes defined by being associated with each of the plurality of words that the performer can present, and association with each of the plurality of states that the performer can take. A management means for managing a database that stores emotional attributes defined by the characters, a plurality of operation commands for controlling the movement of the target character, and the like.
A setting means for selecting the target character from the plurality of characters and setting the target character in the object space, and
An input receiving means for accepting input of performer data including the word presented by the performer, and
A word extraction processing means for extracting the word presented by the performer from the performer data, and
A first parameter generation processing means for specifying the extracted emotional attribute of the word based on the database and generating a first parameter corresponding to the emotional attribute.
A determination means for determining the state of the performer based on at least a part of the input performer data and reference data for determining the state, and
A second parameter generation processing means that specifies the determined emotional attribute of the state based on the database and generates a second parameter corresponding to the emotional attribute.
Among the plurality of operation commands, one or a plurality of operation commands to be used for controlling the operation of the target character are the first parameter, the second parameter, and the character attribute of the selected target character. , Operation command selection means to select based on given selection criteria,
An operation control means that controls the operation of the target character set in the object space based on the selected operation command, and
An image generation means for generating an image of the target character operating in the object space, and
An image output means for outputting the generated image of the target character, and
To be equipped.

Here, the "performer" includes a voice actor, an actor, an idol, a moderator, a speaker (including a rakugoka, a comic storyteller, etc.), and may include an ordinary person.

In addition, the "action of the performer" may include the act of presenting words by the performer, actions (including gestures, poses, etc.), facial expressions, etc., and the "action of the character" includes words spoken by the character. Actions (including gestures, poses, etc.), facial expressions, etc. may be included.

In addition, the "presentation by the performer" may include the performer's expression on at least a part of the body, the performer's intention (word or word), and the like, for example, text typing input, text. Flick input, voice input (speech) of words, input of words due to changes in lip shape, etc. may be included. Therefore, in the "presentation of words by the performer", the input of the word by the utterance of the performer 5, the input of the word by the typing of the performer 5, and the input of the word intended by the performer 5 in the shape of the lips and read by image analysis. , The input of a word intended by the performer 5 in a gesture (sign language, etc.) and read by image analysis is included.

Further, the "object space" may include a two-dimensional or three-dimensional virtual space in which an object including a character is arranged.

Further, the "character attribute" may include a character's gender, age, personality, hometown, and the like, and may include an emotional attribute described later.

In addition, the "plurality of words that the performer can present" may include words that reflect the emotions of the performer, such as "boring", "hungry", ... "like".

In addition, "emotional attributes" include negative (dark) and positive (cheerful) emotional tendencies, and emotion types such as "joy", "anger", "sorrow", "comfort", and "surprise". Etc. may be included.

In addition, the "state of the performer" may include the gender of the performer, the movement of the performer, and the state of the voice of the performer (fatigue level, etc.).

Further, the "operation command" may include a speech command, an action command, a facial expression command, a visual effect command, an amplitude increase / decrease command, and the like. The facial expression command may include a command for making the target character have a facial expression such as emotions, emotions, and surprises. The action command may include a jump gesture, a banzai gesture, a gesture of covering the face with a hand, an arm-folding gesture, a good hand gesture gesture, a poor hand gesture gesture, a guts pose, a fighting pose, and the like to make the target character. The visual effect command may include a command for adding a heart mark, a balloon display, a telop display, or the like to the image of the target character. The amplitude increase / decrease command may include a command for amplifying or suppressing the amplitude (amplitude of the time-varying waveform) of the movement (speech, action, facial expression, visual effect, etc.) of the target character. The utterance command may include a command for converting the ending of the expression of the target character, converting it into a dialect, and the like.

Further, the "performer data" may include data such as an image of the performer (including a moving image) and a voice of the performer.

Further, a given language model may be used for "word extraction".

In addition, in the "word extraction process", language analysis is performed to recognize words from the performer data, and language systems such as "te-ni-ha", "question form", and "assertion" are recognized (morphological analysis and morphological analysis). Processing to perform parsing) may be included.

In addition, the "first parameter" may include a numerical value (score, rank, level, stage, sign, etc.) indicating the negative degree or positive degree of emotion reflected in the word presented by the performer.

Further, the "generation of the first parameter" may be performed by taking into account not only the words included in the performer data but also the language system appearing before and after the words. The "linguistic system" here includes information such as the type of "teniwoha" located after the word, whether the flexion is interrogative, whether the flexion is definite, and so on. You may. In this case, even if the words included in the performer data are the same word, the first parameter may have a different numerical value depending on how the word is presented.

Further, the "at least a part of the performer data" may include voice amplitude data, voice frequency spectrum data, body data (body image data, etc.), facial expression data (face image data, etc.), and the like. ..

Further, the "reference data for determining the state" may include a given template, a given acoustic model, and the like.

Further, the "state determination" may include a determination by template matching.

In addition, the "second parameter" may include a numerical value (score, rank, level, stage, sign, etc.) indicating the negative degree or positive degree of the emotion reflected in the state of the performer.

According to the above configuration, even if the behavior of the performer is the same, the behavior of the target character can be changed depending on the attributes of the selected target character, so that the viewer viewing the image of the target character is presented by the performer. It is possible to perceive not only the words and emotions that appear in the state of the performer, but also the attributes of the target character, that is, the character-likeness of the target character, through the actions of the target character. On the other hand, on the performer side, it is possible to change the movement tendency of the target character simply by changing the character attribute of the target character without consciously changing the words presented by the performer or the state of the performer. There is an advantage.

For example, the operation command selection means can change the specific action of the target character having a specific character attribute to another specific action. Specifically, the operation command selection means selects an operation command that amplifies (over-actions) a joyful action (for example, jump) when the character attribute of the target character is "cheerful gal character". When the specific character attribute of the target character is "a male character who tends to get angry", it is possible to select an operation command that amplifies (overactions) a surprising action (for example, raising both hands).

Therefore, since the performer can appeal the character-likeness of the target character to the viewer without performing a special performance, it is possible to give the viewer a strong impression of good or bad for the target character.

(2) In the image generator of the present invention
The operation command selection means is
Before selecting the operation command, a process of correcting at least one of the first parameter and the second parameter may be executed according to the character attribute of the target character.

According to the above configuration, it is possible to use the same selection criteria as the existing selection criteria for selecting an operation command from the first parameter and the second parameter. Therefore, when an algorithm for selecting an operation command from the first parameter and the second parameter already exists, it can be effectively used.

(3) In the image generator of the present invention
The operation command selection means is
When the combination of the first parameter and the second parameter after correction satisfies a given condition, the operation command is selected based on the first parameter and the second parameter after correction. If the given condition is not satisfied, the operation command may be selected based on the first parameter and the second parameter before correction.

According to the above configuration, it is possible to control the timing at which the character attribute is reflected in the movement of the target character under a given condition. For example, it is possible to control that the character attribute is reflected when the combination is a specific combination and the character attribute is not reflected when the combination is not. Therefore, it is possible to increase the viewer's expectation that the operation of the target character 8A expresses the character-likeness of the target character. In this case, the viewer viewing the image of the target character can effectively perceive the uniqueness of the target character at an appropriate timing.

(4) In the image generator of the present invention
The character attribute includes an emotional attribute, and the character attribute includes an emotional attribute.
The image generator
A third parameter generation processing means for specifying an emotional attribute corresponding to the character attribute of the target character based on the database and generating a third parameter corresponding to the emotional attribute is further provided.
The operation command selection means is
The operation command may be selected based on the combination of the first parameter, the second parameter, and the third parameter, and the selection criteria related to the combination.

Here, the "third parameter" may include a numerical value (score, rank, level, stage, sign, etc.) indicating a negative degree or a positive degree, which is an emotional tendency peculiar to the character attribute.

According to the above configuration, the third parameter related to the emotion of the character attribute can be treated in the same manner as the first parameter and the second parameter. Therefore, when an algorithm for selecting an operation command from the first parameter and the second parameter already exists, the variation (order) of the parameters of the algorithm can be expanded to easily vary the emotional expression of the target character. It is possible to increase.

(5) In the image generator of the present invention
The plurality of operation commands include
Contains one or more specific operation commands associated with one or more specific character attributes.
The processing executed by the operation command selection means includes
A candidate setting process for setting a candidate for the operation command selected from the plurality of operation commands is included.
In the candidate setting process, when the character attribute of the target character is the specific character attribute, a process of adding the specific operation command to the candidate or a process of excluding from the candidate may be executed.

Here, the "specific character attribute" may include, for example, a specific animal character (such as a cat character) and a specific character (such as a nervous middle-aged man).

Further, the "specific operation command" may include a command for converting the flexion to "nya", a command for adding a balloon "nya", a command for converting the flexion to a dialect of a specific region, and the like.

According to the above configuration, it becomes easy to select a specific command suitable for a specific character attribute, so that the character-likeness of the target character can be more strongly appealed.

(6) In the image generator of the present invention
Further provided with a collection means for executing a collection process for collecting evaluation data performed by the viewer on the output image of the target character.
The operation command selection means is
Based on the plurality of the operation commands applied to the plurality of the images and the data of the plurality of evaluations related to the plurality of images, a process of weighting the plurality of the operation commands is executed to generate a new image. The operation command may be selected based on the weights of the plurality of operation commands.

Here, when the sound is not added to the image of the target character, the "viewer" can be paraphrased as the viewer of the image.

In addition, the "evaluation data" includes the number and value of gifts (clothes, clothing, dressing funds, throwing money, twists, etc.) from the viewer for the target character, and the target character by the vote made by the viewer. Data such as the number of votes obtained and the viewing rate of the image content including the image of the target character may be included.

In addition, in voting, the number of times the viewer presses the evaluation button (so-called like button) on his / her terminal device while viewing the image content, and the viewer from his / her terminal device to the operator while viewing the image content. The number of times the comment is transmitted to the user, the number of times the viewer introduces the image content to another person on the SNS, and the like may be included.

In addition, the direct or indirect operator of the image generation device may charge the viewer for presents, votes, and viewing of image contents.

According to the above configuration, since the evaluation data of the viewer can be reflected in the movement of the target character, it is easy for the movement popular among the viewers to be expressed, or a given condition is satisfied. It is also possible to express popular behaviors in some cases. If the above operations of the operation command selection means are sequentially performed during the output of the image, the popular operations will be reflected in real time. In that case, the viewer's awareness of participation in the live distribution will be increased. , The number of viewers can be improved.

(7) In the image generator of the present invention
Further, a feedback means may be further provided to execute an update process for updating the selection criteria based on the weights of the plurality of operation commands each time a given condition is satisfied.

Here, "every time a given condition is satisfied" may include "every time a predetermined time elapses" and "every time a change occurs in the weight".

According to the above configuration, the selection criteria can be changed in a direction in which the popular movement is likely to be expressed, so that, for example, the target character approaches the popular character each time the image generation is repeated. In addition, since the selection criteria themselves are updated, it is possible to converge the selection criteria to a universal selection criteria for improving the popularity of various target characters.

(8) In the image generator of the present invention
The plurality of operation commands include
An operation command for controlling the facial expression of the target character and
Operation commands that control the behavior of the target character and
An operation command that gives a visual effect to the target character,
Operation commands that control the speech of the target character and
At least one of may be included.

Here, the “facial expression of the target character” may include facial expressions of emotions, emotions, and surprises, and may include the shape of the mouth, the shape of the eyes, and the decoration.

Further, the "action of the target character" may include, for example, a jump gesture, a banzai gesture, a gesture of covering the face with a hand, an arm-folding gesture, a good hand gesture, a poor hand gesture, a guts pose, a fighting pose, and the like.

In addition, the "visual effect" may include the generation of a given mark (heart mark, star mark, anger mark, shadow, square red, mustache, kiss mark, etc.), balloon display, telop display, and the like.

According to the above configuration, since the emotion of the performer can be reflected in at least one of the facial expression, the action, the visual effect, and the speech of the target character, the emotion of the performer can be effectively amplified.

(9) In the image generator of the present invention
The image output means
Images of the target character that are sequentially generated during the performance of the performer may be sequentially output to the viewer of the image during the performance.

According to the above configuration, the image of the target character can be live-streamed (live broadcast). The live distribution (live broadcast) also includes so-called near live distribution (delayed distribution).

(10) The image generator of the present invention is
Further, a monitoring means may be further provided which sequentially outputs an image showing the balance of the first parameter, the second parameter, and the character attribute to the performer during the performance.

Here, the "image showing the balance" may include a graph such as a radar chart or a text display (such as "the effect of the character attribute is being expressed").

According to the above configuration, the performer can act while confirming the image and visually grasp the basis for selecting the operation command, so that it becomes easy to bring the movement of the target character closer to the movement desired by himself / herself. ..

(11) In the image generator of the present invention
The image output means
A delay output process for providing a delay time between the generation of the image of the target character and the output of the image to the viewer in the object space is executed, and the performer or the assistant inputs an operation within the delay time. When accepted, a process of adjusting the image output to the viewer may be executed.

According to the above configuration, since it is possible to distribute near live instead of live distribution, it is possible to raise the viewer's awareness of participation in a form similar to live distribution, and the action that the performer or assistant deems inappropriate can be performed. , Self-regulation is possible to prevent it before it is output to the viewer.

(12) In the image generator of the present invention
The operation command selection means is
If the combination of the first parameter, the second parameter, and the character attribute does not change over a predetermined time, a given operation command may be selected regardless of the selection criteria.

According to the above configuration, it is possible to make the target character perform some action even during the period when the performer is inactive.

(13) The program of the present invention is
A program that reflects the movements of a performer existing in real space on the movements of a target character existing in an object space, and causes a computer to function as an image generation device that generates an image of the target character operating in the object space.
Character attributes associated with each of the plurality of characters, emotional attributes defined by being associated with each of the plurality of words that the performer can present, and association with each of the plurality of states that the performer can take. A management means for managing a database that stores emotional attributes defined by the characters, a plurality of operation commands for controlling the movement of the target character, and the like.
A setting means for selecting the target character from the plurality of characters and setting the target character in the object space, and
An input receiving means for accepting input of performer data including the word presented by the performer, and
A word extraction processing means for extracting the word presented by the performer from the performer data, and
A first parameter generation processing means for specifying the extracted emotional attribute of the word based on the database and generating a first parameter corresponding to the emotional attribute.
A determination means for determining the state of the performer based on at least a part of the input performer data and reference data for determining the state, and
A second parameter generation processing means that specifies the determined emotional attribute of the state based on the database and generates a second parameter corresponding to the emotional attribute.
Among the plurality of operation commands, one or a plurality of operation commands to be used for controlling the operation of the target character are the first parameter, the second parameter, and the character attribute of the selected target character. , Operation command selection means to select based on given selection criteria,
An operation control means that controls the operation of the target character set in the object space based on the selected operation command, and
An image generation means for generating an image of the target character operating in the object space, and
An image output means for outputting the generated image of the target character, and
To make the computer work.

According to the above configuration, even if the behavior of the performer is the same, the behavior of the target character can be changed depending on the attributes of the selected target character, so that the viewer viewing the image of the target character is presented by the performer. It is possible to perceive not only the words and emotions that appear in the state of the performer, but also the attributes of the target character, that is, the character-likeness of the target character, through the actions of the target character. On the other hand, on the performer side, it is possible to change the movement tendency of the target character simply by changing the character attribute of the target character without consciously changing the words presented by the performer or the state of the performer. There is an advantage.

For example, the operation command selection means can change the specific action of the target character having a specific character attribute to another specific action. Specifically, the operation command selection means selects an operation command that amplifies (over-actions) a joyful action (for example, jump) when the character attribute of the target character is "cheerful gal character". When the specific character attribute of the target character is "a male character who tends to get angry", it is possible to select an operation command that amplifies (overactions) a surprising action (for example, raising both hands).

Therefore, since the performer can appeal the character-likeness of the target character to the viewer without performing a special performance, it is possible to give the viewer a strong impression of good or bad for the target character.

It is an example of the figure for demonstrating the content distribution system of one Embodiment. It is an example of the figure for demonstrating the image generation system of one Embodiment. There is an example of a figure for explaining the content displayed on the terminal device of one embodiment. It is a figure for demonstrating the confirmation information displayed on the monitor display part of one Embodiment. It is an example of the functional block diagram which shows the structure of the server apparatus in one Embodiment. It is an example of the functional block diagram which shows the structure of the image generation system in one Embodiment. It is an example of the figure explaining the character table stored in the database in one embodiment. It is an example of the figure explaining the word table stored in the database in one embodiment. It is an example of the figure explaining the state table stored in the database in one embodiment. It is an example of the figure explaining the command table stored in the database in one embodiment. It is an example of the correction table stored in the database in one embodiment. It is an example of the figure explaining the correction process in the 1st selection process. This is an example of a diagram illustrating a process of controlling the timing at which the character effect appears. It is an example of the figure explaining the correspondence table stored in the database. It is an example of the figure explaining the feature space in one embodiment. It is an example of the figure explaining the correspondence table stored in the database in one Embodiment. This is an example of a diagram for explaining the selection criteria (boundaries of ranges B1 to B6) before updating. This is an example of a diagram for explaining the selection criteria (boundaries of ranges B1 to B6) after the update. It is an example of the figure explaining the radar chart in one Embodiment. It is an example of the flowchart in one embodiment.

Hereinafter, embodiments will be described. The embodiments described below do not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present invention. In the present embodiment, a content distribution system for distributing an image in an object space in which a character to be viewed (viewed) (hereinafter, also referred to as “target character”) is arranged as content to a viewer will be described.

[1] Outline of Content Distribution System FIG. 1 is an example of a diagram for explaining a content distribution system. As shown in FIG. 1, the content distribution system includes an image generation system 20 that generates content (for example, image content including a moving image of a character) according to the performance (for example, free talk live) of a performer (for example, a voice actor) existing in real space. And the server device 11 that distributes the content generated by the image generation system 20 to the terminal devices 10A, 10B, ... Of the user (hereinafter referred to as "viewer") via a network such as the Internet (WAN) or LAN. included.

The image generation system 20 is installed in a studio in a real space, for example, and each of the terminal devices 10A, 10B, ... Used by individual viewers is a terminal used by the viewer at home or on the go. The server device 11 is arranged at the business office of the operator of the content distribution system. The terminal devices 10A, 10B, ... Are composed of an information processing device such as a tablet-type information terminal device, a personal computer, a smartphone, or a game machine, and can be connected to the server device 11 via a network.

The communication line between the terminal device 10A and the server device 11 and the communication line between the image generation system 20 and the server device 11 may be wired or wireless, respectively.

[1.2] Outline of Image Generation System FIG. 2 is an example of a diagram for explaining the image generation system 20. As shown in FIG. 2, the image generation system 20 includes a performer data acquisition unit 250 that acquires performer data from a performer 5 existing in the real space, and a target character existing in the object space (FIG. 3). An image generation device 20C that generates image content (FIG. 3) including a moving image of the target character 8A operating in the object space, which is reflected in the operation of reference numeral 8A), and confirmation information during generation of the image content (FIG. 3). A monitor display unit 290 for displaying (FIG. 4) is provided.

The performer data acquisition unit 250 acquires performer data including audio data and image data from the performer 5, including, for example, a sound collecting microphone 250A and an image pickup camera 250B.

The image generation device 20C is composed of, for example, an information processing device such as a tablet-type information terminal device, a personal computer, a smartphone, or a game machine, and can be connected to the server device 11 via a network such as the Internet (WAN) or LAN. Device. Further, the communication line between the image generation device 20C and the server device 11 may be wired or wireless.

The monitor display unit 290 may be used for confirming the performance of the performer 5 by the performer 5, or may be used for confirming the performance of the performer 5 by an assistant (such as a person who directs the performer 5). As shown in FIG. 4, various confirmation information is displayed on the monitor display unit 290.

[1.3] Overview of the server device The server device 11 in FIG. 1 collects viewer information 146 from the terminal devices 10A and 10B by using, for example, an API (application programming interface) or a predetermined platform. It is an information processing device that provides various data to the image generation system 20. The server device 11 provides data (streaming data) for distributing the image content generated by the image generation system 20 to the terminal devices 10A, 10B, .... Further, the server device 11 has a storage area (a storage unit 140 described later), and in conjunction with the terminal device 10 and the like, the server device 11 uses the recording area to provide billing information and action history information of each viewer (hereinafter, "" It is also possible to manage each information such as "action history information"). However, the server device 11 may store in a database (storage device, memory in a broad sense) connected via a network (intranet or the Internet) instead of the storage area. Further, the server device 11 may be composed of one (device, processor) or a plurality of (device, processor). Further, in the present embodiment, the image generation system 20 will be described as the main body that controls the operation of the target character 8A in the object space, but some functions of the image generation system 20 are mounted on the server device 11. You may.

[2] Configuration of Server Device FIG. 5 is an example of a functional block diagram showing the configuration of the server device 11. The server device 11 may have a configuration in which some of the components (each part) shown in FIG. 5 are omitted.

The server device 11 inputs data such as a server processing unit 100 that executes processing related to distribution of image contents, an input unit 120 for use by an administrator or other input, a display unit 130 that performs a predetermined display, and viewer information 146. It includes a storage unit 140 for storing, an information storage medium 180 for storing predetermined information, and a communication unit 196 for communicating with an image generation system 20 and others.

The input unit 120 is used by the administrator of the content distribution system for setting related to image content, other necessary settings, and data input. For example, the input unit 120 is composed of a mouse, a keyboard, and the like.

The display unit 130 displays an operation screen for the system administrator. For example, the display unit 130 of the present embodiment is configured by a liquid crystal display or the like.

The information storage medium 180 (a medium that can be read by a computer) stores programs, data, and the like, and its functions are an optical disk (CD, DVD), an optical magnetic disk (MO), a magnetic disk, a hard disk, and a magnetic tape. , Or a memory (ROM) or the like.

The communication unit 196 performs various controls for communicating with the outside (for example, a terminal, another server, or another network system), and its function is hardware such as various processors or ASICs for communication. It is composed of hardware and programs.

The storage unit 140 serves as a work area for the server processing unit 100, the communication unit 196, and the like, and its function is configured by a RAM (VRAM) or the like. The information stored in the storage unit 140 may be managed in a database.

The storage unit 140 includes a main storage unit 142 used as a work area. In addition, the storage unit 140 stores viewer information 146 and the like regarding the viewer. The viewer information 146 is associated with the viewer ID for each viewer.
(1) Viewer attribute information such as gender, age, and area of residence,
(2) Viewer's preference information such as favorite characters,
(3) Information on terminal devices such as smartphones and ID tags owned by viewers (for example, information indicating position coordinates (viewing area) at the time of viewing such as information that identifies the individual viewer).
(4) Information (game play information) regarding the play status and its history (including billing history) in a predetermined game,
(5) Billing information and billing history information from a predetermined online shopping site or actual sales store (including inside the event venue) related to the target character and the image content in which the target character is performing,
Each information such as may be stored for each viewer ID.

In addition, as described above, the billing history includes the billing history in the game and the billing related to the purchase of services and products, the billing date and time indicating the date and time when these billing was executed, the purchase of items and presents (described later). It includes the types of billing factors such as the purchase of, and the billing amount indicating the billed amount.

The server processing unit 100 performs various processes using the main storage unit 142 in the storage unit 140 as a work area. The functions of the server processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs.

The server processing unit 100 performs various processes of the present embodiment based on the program (data) stored in the information storage medium 180. That is, the information storage medium 180 stores a program for operating the computer as each part of the present embodiment (a program for causing the computer to execute the processing of each part).

For example, the server processing unit 100 (processor) has at least a communication control unit 101 and a viewer management unit 103 in order to control the entire server device 11 based on the program stored in the information storage medium 180. ing.

The communication control unit 101 performs a process of transmitting and receiving data via a network with the image generation system 20. That is, the communication control unit 101 receives the image content data from the image generation system 20, receives the viewer information 146 transmitted from the terminal device 10, and distributes the image content to the terminal device 10.

The viewer management unit 103 performs processing related to the registration procedure of each viewer (registration of attribute information, preference information, etc.) and storage management of data associated with the account (viewer ID). Specifically, the viewer management unit 103 controls the granting of an account for registering and managing each viewer information, and the viewer information of each viewer detected during the distribution of the image content (hereinafter, "detected viewer"). It is also called "information" and includes operation commands that the viewer has entered into his / her own terminal device.) Registration information management that registers and manages personal information such as attribute information and preference information related to the viewer, and billing linked to an account. It executes the setting and deposit / withdrawal control of a virtual account used as a source of payment for the consideration related to the execution of the element, the registration of the billing history related to the billing element by the viewer of the account, and the registration of the game history. In particular, the viewer management unit 103 stores and manages the detected viewer information by associating each information generated by the image generation system 20 with the viewer information account.

In addition, the viewer management unit 103 is linked with a game server device (not shown), a management server device that operates a mail-order site, or another server device (not shown) such as a credit card or a bank payment server device, to provide game play information and information. Registration of each information in the viewer information 146 may be executed by exchanging data such as billing history information.

[3] Configuration of Image Generation System FIG. 6 is an example of a functional block diagram showing the configuration of the image generation system 20. The portion surrounded by the dotted line frame in FIG. 6 is the image generator 20C. However, some or all of the elements other than the image generation device 20C may be mounted on the side of the image generation device 20C, and some elements of the image generation device 20C are configured by a device separate from the image generation device 20C. May be done.

The performer data acquisition unit 250 includes a sound collecting microphone 250A and an imaging camera 250B. However, the performer data acquisition unit 250 is equipped with various functions for detecting the state of the performer 5, such as an infrared sensor, a thermography, or a marker unit fixed to the performer 5 and its detection device (camera). May be good.

The input unit 260 is for the performer 5 or an assistant to input an operation command, and the function can be realized by a touch panel, a touch panel type display, a trackball, a keyboard, a mouse, or the like. When the input unit 260 is configured by a device that executes touch operation input such as a touch panel, the detection unit 260 can detect two-dimensional indicated position coordinates (x, y) on the screen on which the image is displayed. The part 262 is provided. For example, the input unit 260 includes a detection unit 262 capable of detecting two-dimensional contact position coordinates (x, y) in the contact detection area (touch panel). The monitor display unit (display) 290 for the operator may be a touch panel type display in which a liquid crystal display and a touch panel for detecting the contact position of the operator are laminated. That is, the monitor display unit 290 may function as the input unit 260.

The storage unit 270 serves as a work area for the processing unit 200, the communication unit 296, and the like, and its function can be realized by a RAM (VRAM) or the like. The storage unit 270 is related to the main storage unit 271 used as a work area, the buffer 272 that temporarily stores the frame data (audio data and image data) of the image content being generated, and the image content. It includes a content data storage unit 274 that stores various data to be stored, and a database 275 that stores various data necessary for generating image content.

A part of these may be omitted, or a part of the storage unit 140 of the server device 11 may be configured.

The information storage medium 280 (a medium that can be read by a computer) stores programs, data, and the like, and its functions are an optical disk (CD, DVD), an optical magnetic disk (MO), a magnetic disk, a hard disk, and a magnetic tape. Alternatively, it can be realized by a memory (ROM) or the like.

Further, the information storage medium 280 can store a program for operating the computer as each part of the present embodiment (a program for causing the computer to execute the processing of each part). As will be described later, the processing unit 200 performs various processes of the present embodiment based on the program (data) stored in the information storage medium 280.

The monitor display unit 290 outputs confirmation information (FIG. 4) during generation of image content and image content (FIG. 3) being generated, and its functions are CRT, LCD, touch panel display, HMD ( It can be realized by a head-mounted display), for example, a projection device composed of a screen and a projector. As described above, the monitor display unit 290 also functions as an input unit 260 in which the operator inputs various inputs by using the touch panel display. Here, as the touch panel, for example, a touch panel such as a resistive film type (4-wire type, 5-wire type), a capacitance method, an electromagnetic induction method, an ultrasonic surface acoustic wave method, or an infrared scanning method can be used.

The sound output unit 292 outputs the sound included in the performer data acquired from the performer 5 or the sound included in the generated image content, and the function can be realized by a speaker, headphones, or the like. The headphone worn by the performer 5 may output the voice of the performer 5 in real time as confirmation information.

The communication unit 296 performs various controls for communicating with an external device (for example, a host device or another terminal device), and its functions include hardware such as various processors or communication ASICs, and a program. It can be realized by such as.

The image generation device 20C receives and receives programs and data stored in the information storage medium of the server device 11 and the storage unit 270 for operating the computer as each part of the present embodiment via the network. The program or data may be stored in the information storage medium 280 or the storage unit 270. The case where the image generator 20C functions by receiving the program or data in this way can also be included in the scope of the present invention.

The processing unit 200 performs various processes using the storage unit 270 as a work area. The function of the processing unit 200 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), or a program.

In particular, the processing unit 200 of the present embodiment includes an object space setting unit 210, a monitor display control unit 213, a movement processing unit 215, a virtual camera control unit 216, a content management unit 219, a drawing unit 220, and a sound. Processing unit 230, database management unit 241, character setting unit 210A, performer data input reception unit 242, word extraction processing unit 243, first parameter generation processing unit 244, state determination unit 245, and second The parameter generation processing unit 246, the operation command selection unit 247, the operation control unit 248, the image generation unit 249, and the image output unit 240 are included. Further, a configuration in which some of these may be omitted may be used.

The object space setting unit 210 is a character object (target character), a movement path, a building, a tree, a pillar, a wall, a map (topography), and various other objects (sprites, billboards, polygons, free curved surfaces, or subdivisions). Performs a process of arranging and setting an object composed of primitive surfaces such as a surface in an object space (that is, a game space).

Specifically, the object space setting unit 210 determines the position and rotation angle (synonymous with orientation and direction) of the object (model object), and rotates the object (model object) at the position (X, Y) or (X, Y, Z). Place the object by an angle (rotation angle around the X, Y axis) or (rotation angle around the X, Y, Z axis).

The object space includes both a so-called virtual two-dimensional space and a virtual three-dimensional space. The two-dimensional space is, for example, a space in which objects are arranged in two-dimensional coordinates (X, Y), and the three-dimensional space is, for example, a space in which objects are arranged in three-dimensional coordinates (X, Y, Z). be.

Then, when the object space is a two-dimensional space, the objects are arranged based on the priority set for each of the plurality of objects. For example, it is possible to arrange objects in order from the object (sprite) that you want to appear to be on the back side, and to stack the objects that you want to appear to be on the front side.

Also, if an object with a large drawing size is placed below the image and an object with a small drawing size is placed above the image, the object space corresponding to the upper part of the screen can be made to appear to be on the back side. It is possible to make it appear that the object space corresponding to the lower part of is on the front side.

On the other hand, when the object space is a three-dimensional space, the objects are arranged in the world coordinate system.

The monitor display control unit 213 displays the image content (FIG. 3) being generated on the monitor display unit 290, and displays confirmation information (FIG. 4) for supporting the performer 5 and the assistant on the monitor display unit 290. ..

The movement processing unit 215 performs an object movement operation in the object space. That is, the movement processing unit 215 moves the moving object in the object space based on the operation command, the program (moving algorithm), various data (motion data), etc. selected by the method described later, or the moving body. Performs processing to control the movement (motion, animation) of an object.

Specifically, the movement processing unit 215 of the present embodiment includes movement information (movement direction, movement amount, movement speed, position, rotation angle, or acceleration) and motion information (position or rotation angle of each part object) of the object. ) Is sequentially obtained for each frame. Here, the frame is a unit of time for performing object movement processing, motion processing (simulation processing), and image generation processing. Then, in the present embodiment, the frame rate may be fixed or variable depending on the processing load.

The virtual camera control unit 216 generates an image that is visible from a given viewpoint and that appears to have depth. In this case, the virtual camera control unit 216 performs control processing of the virtual camera (viewpoint) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, processing (processing for controlling the viewpoint position and line-of-sight direction) for controlling the position (X, Y, Z) or rotation angle (rotation angle around the X, Y, Z axes) of the virtual camera is performed.

For example, when an object (for example, a character, a ball, a car) is photographed from behind by a virtual camera, the position or rotation angle of the virtual camera (virtual camera) so that the virtual camera follows a change in the position or rotation of the object. Orientation) is controlled.

In this case, the virtual camera can be controlled based on the information such as the position, rotation angle, and speed of the object obtained by the movement processing unit 215. Alternatively, the virtual camera may be controlled to be rotated by a predetermined rotation angle or moved by a predetermined movement path. Further, in this case, the virtual camera is controlled based on the virtual camera data for specifying the position (movement path) or the rotation angle of the virtual camera.

When there are a plurality of virtual cameras (viewpoints), the above control processing is performed for each virtual camera.

The content management unit 219 records in the content data storage unit 274 as content generation information the type and timing of the operation command adopted during the generation of the image content. For example, the content management unit 219 stores the operation commands applied to each section of the image content in the content data storage unit 274 for each section.

The drawing unit 220 performs drawing processing based on the results of various processes (each process related to distribution of image content) performed by the processing unit 200, thereby generating an image included in the image content.

Further, the image generated by the drawing unit 220 may be a so-called two-dimensional image or a so-called three-dimensional image, and the drawing unit 220 uses an image seen from a virtual camera in the object space as image content. Generate as a included image.

Here, when generating a two-dimensional image, the drawing unit 220 draws in order from the set low-priority objects, and when the objects overlap, the high-priority objects are overwritten and drawn. ..

Further, when generating a three-dimensional image, the drawing unit 220 first includes the vertex data (positional coordinates of the vertex, texture coordinates, color data, normal vector, α value, etc.) of each vertex of the object (model). Object data (model data) is input, and vertex processing is performed based on the vertex data included in the input object data. In addition, when performing the vertex processing, the vertex generation processing (tessellation, curved surface division, polygon division) for subdividing the polygon may be performed as necessary.

In the vertex processing, geometry processing such as vertex movement processing, coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective conversion, or light source processing is performed, and the object is processed based on the processing result. Change (update, adjust) the given vertex data for the constituent vertex group. Then, rasterization (scanning conversion) is performed based on the vertex data after the vertex processing, and the faces of the polygons (primitives) are associated with the pixels. Then, following the rasterization, pixel processing (fragment processing) for drawing the pixels (fragments constituting the display screen) constituting the image is performed.

In pixel processing, various processes such as texture reading (texture mapping), color data setting / change, translucent composition, antialiasing, etc. are performed to determine the final drawing color of the pixels that make up the image, and perspective conversion is performed. The drawing color of the object is output (drawn) to the buffer 272 (frame buffer, a buffer that can store image information in pixel units. VRAM, rendering target). That is, in the pixel processing, per pixel processing is performed in which image information (color, normal, brightness, α value, etc.) is set or changed in pixel units.

As a result, an image that can be seen from a virtual camera (given viewpoint) set in the object space is generated. When there are a plurality of virtual cameras (viewpoints), an image can be generated so that the image seen from each virtual camera can be displayed as a divided image on one screen.

The vertex processing and pixel processing performed by the drawing unit 220 are performed by hardware that enables programming of polygon (primitive) drawing processing by a shader program written in a shading language, so-called programmable shader (vertice shader or pixel shader). It may be realized. In the programmable shader, the processing in units of vertices and the processing in units of pixels can be programmed, so that the degree of freedom in the drawing processing content is high, and the expressive power can be greatly improved compared to the fixed drawing processing by hardware. .. Then, when drawing the object, the drawing unit 220 performs geometry processing, texture mapping, hidden surface erasing processing, α blending, and the like.

In the geometry processing, the object is subjected to processing such as coordinate transformation, clipping processing, perspective projection conversion, or light source calculation. Then, the object data after the geometry processing (after the perspective projection conversion) (position coordinates of the vertices of the object, texture coordinates, color data (luminance data), normal vector, α value, etc.) is stored in the storage unit 270.

In the texture mapping, a process of mapping the texture (texel value) stored in the texture storage unit of the storage unit 270 to the object is performed. Specifically, the texture (surface properties such as color (RGB) and α value) is read from the texture storage unit of the storage unit 270 using the texture coordinates and the like set (assigned) to the vertices of the object, and a two-dimensional image is obtained. Map the texture that is to the object. In this case, processing for associating pixels with texels and bilinear interpolation as texel interpolation are performed.

In this embodiment, when drawing an object, a process of mapping a given texture may be performed. In this case, the color distribution (texel pattern) of the mapped texture can be dynamically changed.

Further, in this case, textures having different color distributions (pixel patterns) may be dynamically generated, or a plurality of textures having different color distributions may be prepared in advance so that the textures to be used are dynamically switched. You may. Further, the color distribution of the texture may be changed for each object.

In the hidden surface erasing process, the hidden surface erasing process is performed by the Z buffer method (depth comparison method, Z test) using the Z buffer (depth buffer) in which the Z value (depth information) of the drawing pixel is stored. That is, when drawing the drawing pixel corresponding to the primitive of the object, the Z value stored in the Z buffer is referred to, and the Z value of the referenced Z buffer and the Z value of the drawing pixel of the primitive are referred to. In comparison, if the Z value in the drawing pixel is the Z value (for example, a small Z value) that is on the front side when viewed from the virtual camera, the drawing pixel is drawn and the Z value in the Z buffer is newly added. Update to a Z value.

In α blending (α synthesis), the drawing unit 220 performs a translucent synthesis process (usually α blending, addition α blending, subtraction α blending, etc.) based on the α value (A value). The α value is information that can be stored in association with each pixel (texel, dot), and is, for example, plus alpha information other than color information. The α value can be used as mask information, semi-transparency (equivalent to transparency and opacity), bump information, and the like.

The sound processing unit 230 performs sound processing based on the results of various processing performed by the processing unit 200, and generates various sounds when the target character such as BGM, sound effect, or voice performs a performance. It is output to the sound output unit 292.

The database management unit 241 manages the database 275. The management of the database 275 includes a character table (FIG. 7), a word table (FIG. 8), a status table (FIG. 9), a command table (FIG. 10), and a correction table (FIG. 11) stored in the database 275. Management of tables (FIGS. 14 and 16) and the like is included.

The character setting unit 210A of the object space setting unit 210 executes a process of selecting a target character from a plurality of characters and setting the target character in the object space based on the instruction of the performer 5 or the assistant.

The performer data input reception unit 242 receives input of performer data including the voice of the performer 5 during the talk live.

The word extraction processing unit 243 converts the voice into text based on the text conversion process for the voice included in the input performer data, and extracts the word uttered by the performer 5 from the text.

The first parameter generation processing unit 244 specifies the emotional attributes of the words extracted by the word extraction processing unit 243 based on the word table (FIG. 8) of the database 275, and generates the first parameter corresponding to the emotional attributes. ..

The state determination unit 245 determines the state (mainly the action here) of the performer 5 based on at least a part of the input performer data and the reference data for state determination.

The second parameter generation processing unit 246 specifies the emotional attribute of the state (behavior) determined by the state determination unit 245 based on the state table (FIG. 9) of the database 275, and sets the second parameter according to the emotional attribute. Generate.

The operation command selection unit 247 selects one or a plurality of operation commands to be used for controlling the operation of the target character 8A from the plurality of operation commands with the first parameter, the second parameter, and the selected target character. Select based on character attributes and given selection criteria.

The motion control unit 248 controls the motion of the target character 8A set in the object space based on the selected operation command.

The image generation unit 249 generates image data (FIG. 3) of the target character 8A operating in the object space.

The image output unit 240 outputs the image content including the generated image data of the target character 8A to the server device 11.

The database 275 includes a character table (FIG. 7), a word table (FIG. 8), a status table (FIG. 9), a command table (FIG. 10), a correction table (FIG. 11), a corresponding table (FIGS. 14 and 16), and the like. Remember.

[4] Outline of the method of the present embodiment [4.1] Outline Next, an outline of the method of the present embodiment (image content generation processing) will be described.

The image generation device 20C of the present embodiment reflects the movement of the performer 5 existing in the real space on the movement of the target character 8A existing in the object space, and the image of the target character 8A operating in the object space (FIG. 3). Generate data (hereinafter referred to as image data).

Here, "performer 5" includes voice actors, actors, idols, moderators, speakers (including rakugoka, comic storytellers, etc.), and includes ordinary people. However, "voice actor" is assumed here.

In addition, the "action of the performer 5" includes an act, an action, a facial expression, and the like in which the performer 5 presents words.

Further, the "presentation by the performer 5" may include that the performer 5 expresses with at least a part of the body, that the performer 5 intends (words or words), and the like, for example, in a text. Typing input, text flick input, word input (speech), word input due to lip shape change, etc. may be included. Therefore, in "presentation of words by performer 5", the input of words by the words of the performer 5, the input of words by typing of the performer 5, and the words that the performer 5 intended in the form of lips and read by image analysis. Includes input, input of words intended by the performer 5 in gestures (sign language, etc.) and read by image analysis. However, in the present embodiment, it is assumed that "speech by performer 5" is adopted as "presentation by performer 5".

In addition, the "action of the target character 8A" includes words, actions, facial expressions, and the like by the target character 8A. Here, it is assumed that the speech of the target character 8A is basically set to be the same as the speech of the performer 5, and the flexion of the speech of the target character 8A is controlled according to the operation command. Further, here, the timbre (also referred to as voice color or frequency spectrum) of the target character 8A is set to a voice color previously associated with the character attribute of the target character 8A, and the action and facial expression of the target character 8A are operation commands (described later). ) Shall be controlled. However, it is naturally possible to use the voice of the performer 5 as it is as the voice of the target character 8A without processing it.

Further, the "object space" includes a two-dimensional or three-dimensional virtual space in which an object including a character is arranged. In FIG. 3, a three-dimensional virtual space is assumed as the object space.

In the image generation device 20C of the present embodiment, even if the movement of the performer 5 is the same, the movement of the target character 8A can be changed depending on the attributes of the selected target character 8A, so that the image content of the target character 8A The viewer viewing (FIG. 3) not only perceives the words spoken by the performer 5 and the emotions appearing in the state (action) of the performer 5, but also the attributes of the target character 8A, that is, the characteristics of the target character 8A. It can be perceived through movement. On the other hand, on the side of the performer 5, the movement of the target character 8A can be performed simply by changing the character attribute of the target character 8A without consciously changing the words spoken by oneself or one's state (behavior). It has the advantage of being able to change trends. Therefore, since the performer can appeal the character-likeness to the viewer without performing a special performance, the viewer can be strongly impressed with the character's likes and dislikes.

[4.2] Information stored in the database Next, the character table (FIG. 7), the word table (FIG. 8), the status table (FIG. 9), and the command table (FIG. 9) stored in the database 275 (FIG. 6). 10) will be described in order. These tables are used when executing the image content generation process.

[4.2.1] Character table As shown in FIG. 7, the character table defines the character names of a plurality of characters, the character attributes associated with each of the character names, and the character attributes associated with the character attributes. A positive score obtained by quantifying the positive level is written in advance for each character ID.

In the example of FIG. 7, the character A of the character ID "C001" is associated with the character attribute "cheerful middle-aged man", and the character attribute "cheerful middle-aged man" has a positive score " 15 ”is associated with it.

Here, the positive score corresponding to the character attribute is a quantification of the emotional attribute included in the character attribute (the tendency of the emotion that the character attribute is easy to hold). That is, each emotional attribute is classified into two classes (two poles) in advance, "positive" and "negative", based on the impression generally recalled from the name of the emotional attribute, and is "positive". A positive value is assigned as a positive score to the emotional attribute belonging to, and a negative value is assigned as a positive score to the emotional attribute belonging to "negative". In addition, each emotional attribute is divided into levels according to the intensity of emotions generally recalled from the name of the emotional attribute, and the higher the emotional attribute, the larger the absolute value of the positive score is assigned. ..

Therefore, for example, the character attribute "cheerful girl" of the character ID "C007" is associated with a positive score "15" related to the emotional attribute "cheerful" included in the character attribute "cheerful girl". , The character attribute "slightly cheerful middle-aged man" of the character ID "C010" corresponds to the positive score "10" related to the emotional attribute "slightly cheerful" included in the character attribute "slightly cheerful middle-aged man". The character attribute "gloomy girl" of the character ID "C021" is associated with a positive score "-10" related to the emotional attribute "gloomy" included in the character attribute "gloomy girl". ing.

As described above, in the character table (FIG. 7) of the present embodiment, a character having a more intense emotion and a positive impression is associated with a higher positive score. In FIG. 7, expressions such as "cheerful", "gloomy", "calm", and "nervous" were used as emotional attributes, but "positive thinking", "positive", "negative", and "negative thinking" were used. , "Happiness," "anger," "sorrow," "comfort," "surprise," and the like. Regardless of which expression is used, it is desirable that a positive score be assigned based on the impression generally recalled from the expression.

[4.2.2] Word table As shown in FIG. 8, the word table includes a plurality of words that the performer 5 can utter, emotional attributes defined in association with each of the words, and the emotional attributes. The positive score defined in association with is written in advance for each word ID.

In the example of FIG. 8, the word “Yatane” with the word ID “W001” is associated with the emotional attribute “positive”, and the emotional attribute “positive” is associated with the positive score “15”. ing. Further, the word "hungry" of the word ID "W030" is associated with an emotional attribute of "negative", and the emotional attribute "negative" is associated with a positive score "-10". ..

Here, the positive score corresponding to the emotional attribute is a polarization and quantification of the emotional attribute. That is, each emotional attribute is classified into two classes (two poles) in advance, "positive" and "negative", based on the impression generally recalled from the name of the emotional attribute, and is "positive". A positive value is assigned as a positive score to the emotional attribute belonging to, and a negative value is assigned as a positive score to the emotional attribute belonging to "negative". In addition, each emotional attribute is divided into levels according to the intensity of emotions generally recalled from the name of the emotional attribute, and the higher the emotional attribute, the larger the absolute value of the positive score is assigned. .. The numerical values assigned in FIG. 8 are merely examples, and the numerical values can be appropriately changed.

Therefore, for example, the word "done" in the word ID "W001" is associated with a relatively high positive score "15", and is relative to the word "hungry" in the word ID "W030". A low positive score "-5" is associated with the target.

As described above, in the word table of the present embodiment (FIG. 8), words with more intense emotions and positive impressions are associated with higher positive scores. In FIG. 8, expressions such as "positive", "slightly positive", "flat", and "slightly negative" were used as emotional attributes, but "plus thinking", "positive", "negative", and "minus" were used. Expressions such as "thinking", "joy", "anger", "sorrow", "comfort", and "surprise" may be used. Regardless of which expression is used, it is desirable that a positive score be assigned based on the impression generally recalled from the expression.

[4.2.3] State table As shown in FIG. 9, in the state table, a plurality of possible states of the performer 5 (hereinafter, the state of the performer 5 pays particular attention to "action" among the actions). The emotional attribute defined in association with each of the plurality of states (actions) and the positive score defined in association with the emotional attribute are written in advance for each state ID.

In the example of FIG. 9, the state (action) "jump gesture" of the state ID "M001" is associated with the emotional attribute "comfort" generally recalled from the jump gesture, and the emotional attribute "comfort" is associated with the emotional attribute "comfort". , A positive score "15" is associated. In addition, the state (action) "Banzai gesture" of the state ID "M030" is associated with the emotional attribute "surprise" that is generally recalled from the banzai gesture, and the emotional attribute "surprise" has a positive score "surprise". -5 "is associated.

Here, the positive score corresponding to the emotional attribute is a quantification of the emotional attribute (different from emotions, emotions, and surprises). That is, each emotional attribute is classified into two classes (two poles) in advance, "positive" and "negative", based on the impression generally recalled from the emotional attribute (separate emotions, emotions, and surprises). A positive value is assigned as a positive score to the emotional attributes "comfort" and "joy" belonging to "positive", and a negative value as a positive score is assigned to the emotional attribute "surprise" belonging to "negative". Is assigned. In addition, each emotional attribute is divided into levels according to the intensity of emotions generally recalled from the emotional attribute (separate from emotions, emotions, and surprises), and the higher the emotional attribute, the more the absolute value of the positive score. Is assigned a large value. The numerical values assigned in FIG. 9 are merely examples, and the numerical values can be appropriately changed.

Therefore, a relatively high positive score "15" is associated with the state (behavior) "jump gesture" expressing positive emotions, and the state (behavior) "arm-folding gesture" expressing negative emotions is associated with. A relatively low positive score "0.3" is associated.

As described above, in the state table (FIG. 9) of the present embodiment, the higher the positive score is associated with the action of a more intense emotion and a positive impression. In FIG. 9, expressions such as "comfort", "joy", "sorrow", "anger", and "surprise" were used as emotional attributes, but "plus thinking", "positive", and "passive", Expressions such as "minus thinking", "positive", "slightly positive", "flat", "slightly negative", and "negative" may be used. Regardless of which expression is used, it is desirable that a positive score be assigned based on the impression generally recalled from the expression.

[4.2.4] Command Table As shown in FIG. 10, a plurality of operation commands for controlling the operation of the target character 8A are written in advance for each command ID in the command table. However, in FIG. 10, the operation commands are distinguished by the symbols a1, a2, ..., b1, b2, ..., C1, c2, ..., D1, d2, ... Instead of the command ID. Here, the operation commands include four types of facial expression commands a1, a2, ..., Action commands b1, b2, ..., Visual effect commands c1, c2, ..., And speech commands d1, d2, .... It shall be. Further, in the present embodiment, for each section during the talk live, one of the facial expression commands a1, a2, ..., any one of the gesture commands b1, b2, ..., And the visual effect commands c1, c2, ... It is assumed that any one and any one of the speech commands d1, d2, ... Are selected, and the operation of the target character 8A in the section is controlled according to the combination of the four selected operation commands. Specific examples of facial expression commands, action commands, visual effect commands, and speech commands will be described below in order.

[4.2.4.1] Facial expression command For example, the facial expression command includes a command a1 for making the facial expression of the target character 8A smile, a command a2 for making the facial expression of the target character 8A shy, and a target character 8A. The command a3 for making the facial expression expressionless, the command a4 for making the facial expression of the target character 8A angry, the command a5 for making the facial expression of the target character 8A surprised, and the like are included.

[4.2.4.2] Action command For example, the action command includes a command b1 that causes the target character 8A to make a jump gesture, a command b2 that causes the target character 8A to make a one-handed raising gesture, and a command that puts the target character 8A in a stationary state. b3, a command b4 for making the target character 8A make a gesture to cover the face with a hand, a command b5 for making the target character 8A make a banzai gesture, and the like are included.

[4.2.4.3] Visual effect command For example, the visual effect command includes a command c1 that generates a star mark (star object) in the object space, and a command c2 that generates a heart mark (heart object) in the object space. It includes commands c3, ..., Etc. that do not generate any object in the object space.

[4.2.4.4] Speaking command For example, a uttering command includes a command that converts the flexion or expression of a word uttered by the target character 8A into a dialect of a specific region, or a word uttered by the target character 8a. A command to convert the flexion of the target character to a specific flexion, a command to convert the flexion or expression of the word spoken by the target character 8a to the flexion of a female word, or the flexion or expression of the word spoken by the target character 8a to the flexion of a male word. Includes commands to convert to, ... and so on.

[4.3] Elements Related to Image Generation Processing Next, among the image generation devices 20C of FIG. 6, elements related to image content generation processing will be described in detail. Elements related to the image content generation process include, for example, a database management unit 241, a performer data input reception unit 242, a word extraction processing unit 243, a first parameter generation processing unit 244, a state determination unit 245, and a second parameter generation. The processing unit 246, the operation command selection unit 247, the operation control unit 248, the image generation unit 249, the image output unit 2491, and the character setting unit 210A.

[4.3.1] Database management unit The database management unit 241 assigns character attributes (character table in FIG. 7) associated with each of a plurality of characters and a plurality of words that the performer 5 can utter. The emotional attributes defined in association with each other (word table in FIG. 8) and the emotional attributes defined in association with each of the plurality of possible states (actions) of the performer 5 (state table in FIG. 9). It manages a database 275 that stores a plurality of operation commands (command table of FIG. 10) that control the operation of the target character 8A.

For example, the database management unit 241 can read the character table (FIG. 7), the word table (FIG. 8), the status table (FIG. 9), and the command table (FIG. 10) as needed during the talk live, or perform the talk live. Before and after the start, the contents of the character table (Fig. 7), word table (Fig. 8), status table (Fig. 9), and command table (Fig. 10) can be updated according to the request from the administrator of the content distribution system. Execute the process to be performed.

The database 275 also stores the reference data for status determination described later, the correspondence table (FIG. 14) described later, and the like in advance, and these data are also read out by the database management unit 241 as needed. ..

[4.3.2] Character setting unit The character setting unit 210A selects a target character from a plurality of characters and sets it in the object space based on the instruction of the performer 5 or the assistant. For example, the character setting unit 210A receives an operation input for designating a character from the performer 5 or an assistant (not shown) via the input unit 260 prior to the start of the talk live, and the performer 5 or the assistant based on the operation input. When the specified character is specified, the specified character is set in the object space as the target character 8A.

[4.3.3] Performer data input reception unit The performer data input reception unit 242 accepts input of performer data (voice data and image data) including the words spoken by the performer 5 during the talk live. For example, the performer data input reception unit 242 receives the performer data (audio data and image data) sampled by the performer data acquisition unit 250 at a predetermined cycle during the talk live in the same cycle as the sampling cycle, and receives the performer data (audio). Data and image data) are temporarily stored in the buffer 272. In the present embodiment, frame data (audio data and image data) of the corresponding section of the image content is generated based on the performer data (audio data and image data) of the section accumulated in the buffer 272.

The performer data input reception unit 242 stores the performer data of the entire section sampled by the performer data acquisition unit 250 during the talk live as a file of the performance that is the basis of the image content after the end of the talk live. It may be stored in part 274.

[4.3.4] Word extraction processing unit The word extraction processing unit 243 converts the voice into text based on the text conversion processing for the voice included in the input performer data, and the word spoken by the performer 5 is the relevant word. Extract from text.

Here, a given language model may be used for "conversion to text". In the "word extraction process", language analysis is performed to recognize words from the text, and language systems such as "teniwoha", "question form", and "assertion" are recognized (morphological analysis and syntax). Processing to analyze) may be included.

For example, the word extraction processing unit 243 converts the voice data into text data by referring to the voice data included in the performer data of the section stored in the buffer 272 and performing text conversion processing on the voice data. This text data includes not only text representing words but also text indicating "teniwoha". Then, the word extraction processing unit 243 extracts a word (for example, a noun or a verb) registered in the word table in advance from the text data.

[4.3.5] First Parameter Generation Processing Unit The first parameter generation processing unit 244 specifies the emotional attributes of the words extracted by the word extraction processing unit 243 based on the word table (FIG. 8) of the database 275. , Generates a first parameter according to the emotional attribute.

Here, the "first parameter" includes a numerical value (score, rank, level, stage, sign, etc.) indicating the negative degree or positive degree of the emotion reflected in the word spoken by the performer 5.

For example, the first parameter generation processing unit 244 refers to the word table (FIG. 8) according to the word extracted by the word extraction processing unit 243, and uses the positive score associated with the word as the first parameter. Generate.

Further, the "generation of the first parameter" may be performed in consideration of the language system. For example, the first parameter generation processing unit 244 refers to the word table (FIG. 8) according to one or more words extracted by the word extraction processing unit 243, and the emotion associated with the one or more words. When the attribute is recognized, a positive score corresponding to the most dominant (majority) emotion attribute among these emotion attributes is generated as the first parameter. As described above, if the word table (FIG. 8) is used, the first parameter can be generated at high speed because complicated arithmetic processing is not required.

Further, for example, when the text data described above includes the text "I want to cry", the first parameter generation processing unit 244 sets the positive score associated with the word "crying" in a word table (figure). While recognizing based on 8), the language system that appeared before and after the word "crying" (type of "teniwoha", "whether it is an interrogative form", "whether it is an assertive form"" The first parameter may be generated by applying a correction according to the information) to the positive score. In that case, even if the same word "crying" is included in the above-mentioned text data, the first parameter is determined depending on whether the word is "crying" or "want to cry". May be different numbers.

[4.3.6] State determination unit The state determination unit 245 determines the state (behavior) of the performer 5 based on at least a part of the input performer data and the reference data for the state determination.

Here, even if "at least a part of the performer data" includes voice amplitude data, voice frequency spectrum data, body data (body image data, etc.), facial expression data (face image data, etc.), etc. good.

Further, the "reference data for determining the state" includes a given template, a given acoustic model, and the like. However, the acoustic model as the reference data is not indispensable when only the behavior is judged as the state.

For example, the state determination unit 245 executes the following processes (i) to (iii).

(I) The state determination unit 245 refers to the image data of a plurality of frames included in the performer data of the section stored in the buffer 272, and the action of the performer 5 is reflected from the image data of the plurality of frames of the section. The process of extracting the characteristic pattern (time series pattern) is executed.

(Ii) The state determination unit 245 executes a process of calculating the correlation between the extracted feature pattern and the reference data for state determination (template for each action) for each action. The reference data (template for each action) for determining the state is stored in the database 275 in advance, for example.

(Iii) The state determination unit 245 determines that the action with the highest correlation is the action performed by the performer 5.

[4.3.7] Second parameter generation processing unit The second parameter generation processing unit 246 identifies the emotional attribute of the determined state (behavior) based on the state table (FIG. 9) of the database 275, and the emotion concerned. Generate a second parameter according to the attribute.

Here, the "second parameter" includes a numerical value (score, rank, level, stage, sign, etc.) indicating the negative degree or positive degree of the emotion reflected in the state (action) of the performer 5.

For example, the second parameter generation processing unit 246 refers to the state table (FIG. 9) according to the action determined by the state determination unit 245, and generates a positive score associated with the action as the second parameter. do. As described above, if the state table (FIG. 9) is used, the second parameter can be generated at high speed because complicated arithmetic processing is not required.

Further, for example, the second parameter generation processing unit 246 refers to the state table (FIG. 9) according to one or a plurality of actions determined by the state determination unit 2453, and is associated with the one or a plurality of actions. When the emotional attribute is recognized, a positive score corresponding to the most dominant (majority) emotional attribute among these emotional attributes is generated as a second parameter. As described above, if the state table (FIG. 9) is used, the second parameter can be generated at high speed because complicated arithmetic processing is not required.

[4.3.8] Operation command selection unit The operation command selection unit 247 selects one or more operation commands to be used for controlling the operation of the target character 8A from among a plurality of operation commands as the first parameter. The selection is made based on the second parameter, the character attribute of the selected target character, and the given selection criteria.

Here, the operation command selection unit 247 can also use the character attribute of the target character for correction of at least one of the first parameter and the second parameter (first selection process described later), or a third. It can also be used to generate parameters (second selection process and third selection process described later).

However, in the present embodiment, it is not essential that all of the first selection processing function, the second selection processing function, and the third selection processing function are mounted on the image generator 20C. , It is also possible to omit some of these functions. Details of the first selection process, the second selection process, and the third selection process will be described later.

[4.3.9] Motion control unit The motion control unit 248 controls the motion of the target character 8A (FIG. 3) set in the object space based on the selected operation command. For example, the operation commands selected by the operation command selection unit 247 are the expression command "wink", the action command "good gesture", the visual effect command "heart mark", and the "addition of" Nya "". In the case of a combination with a word command, the motion control unit 248 makes the target character 8A wink, makes a good gesture, generates a heart mark, and makes a statement of the target character 8A based on these operation commands. Executes the control to add "nya" to the end of the word.

[4.3.10] Image Generation Unit The image generation unit 249 generates image data (FIG. 3) of the target character 8A operating in the object space.

For example, the image generation unit 249 performs a process of generating image data with the above-mentioned virtual camera in the object space as a viewpoint based on the performer data of the section accumulated in the buffer 272 and writing the image data to the buffer 272 next to the buffer 272. It is executed within the period until the performer data of the section of is accumulated. Further, the image generation unit 249 repeatedly executes the process of generating and writing the image data of the section during the talk live. The image data of the entire section written in the buffer 272 during the period from the start to the end of the talk live (or the period corresponding thereto) is the image content.

In the present embodiment, the voice data of the voice emitted by the target character 8A is added to the image data generated by the image generation unit 249 and written to the buffer 272. The voice emitted by the target character 8a is basically the same as the voice emitted by the performer 5, but the voice emitted by the performer 5 is subjected to the conversion process of the ending based on the speech command and the character attribute of the target character 8a. Depending on the frequency conversion process (voice color conversion process), etc. may be performed. Hereinafter, the image data to which the audio data is added is referred to as "image data with audio".

[4.3.11] Image output unit The image output unit 240 outputs the generated image data (FIG. 3) of the target character 8A.

For example, the image output unit 240 sequentially transmits the image data with sound of each section sequentially stored in the buffer 272 to the server device 11. The server device 11 distributes (live distribution) image data with audio sequentially received from the image generation system 20 as streaming data to the terminal devices 10A, 10B, 10C, ... Of each viewer via the network.

The image output unit 240 may store the image data with sound of the entire section generated during the talk live in the content data storage unit 274 as an image content file. This file can be used for recording and distribution of image contents and the like.

[4.4] First Selection Process [4.4.1] Outline The first selection process for selecting an operation command will be described below.

In the first selection process
The operation command selection unit 247
Before selecting the operation command, a process of correcting at least one of the first parameter and the second parameter according to the character attribute of the target character 8A is executed.

According to the above configuration, it is possible to use the same selection criteria as the selection criteria for selecting the operation command from the first parameter and the second parameter. Therefore, when an algorithm for selecting an operation command from the first parameter and the second parameter already exists, it can be effectively used.

In addition, the operation command selection unit 247
When the combination (total value) of the first parameter and the second parameter after the correction satisfies the given condition, the operation command is selected based on the first parameter and the second parameter after the correction. If the given condition is not satisfied, the operation command is selected based on the first parameter and the second parameter before correction.

According to the above configuration, the timing at which the character attribute is reflected in the operation of the target character 8A can be controlled under a given condition. For example, it is possible to control that the character attribute is reflected when the combination is a specific combination and the character attribute is not reflected when the combination is not. Therefore, it is possible to increase the viewer's expectation that the operation of the target character 8A will have the appearance of the target character 8A. In this case, the viewer viewing the image content (FIG. 3) of the target character 8A can effectively perceive the uniqueness of the target character 8A at an appropriate timing.

[4.4.2] Correction table In order to realize the first selection process, the correction table (FIG. 11) is stored in the database 275 in advance. As shown in FIG. 11, the correction table is a table in which a plurality of correction coefficients are stored in advance. The plurality of correction coefficients are stored for each character attribute of the target character 8A and for each emotion attribute related to the state (behavior) of the performer 5. Here, the emotional attributes related to the state (behavior) of the performer 5 are represented by "joy", "anger", "sorrow", "comfort", and "surprise".

In the correction table of FIG. 11, when the character attribute of the target character is "nervous middle-aged male" and the emotional attribute related to the state (behavior) of the performer 5 is "easy", the correction coefficient "2". .0 ”is associated.

Further, in the correction table of FIG. 11, when the character attribute of the target character is "nervous middle-aged male" and the emotional attribute related to the state (behavior) of the performer 5 is "joy", the correction coefficient "1.2" is associated with it.

Further, in the correction table of FIG. 11, when the character attribute of the target character is "nervous middle-aged male" and the emotional attribute related to the state (behavior) of the performer 5 is "sorrow", the correction coefficient "1.2" is associated with it.

Further, in the correction table of FIG. 11, when the character attribute of the target character is "nervous middle-aged male" and the emotional attribute related to the state (behavior) of the performer 5 is "angry", the correction coefficient "0.3" is associated with it.

In this way, even if the character attributes are the same, the correction coefficient changes depending on the state (behavior) of the performer 5. This correction coefficient is used when correcting the first parameter and the second parameter.

[4.4.3] Correction processing In the image generator 20C of the present embodiment,
The operation command selection unit 247
Before selecting the operation command, the correction table (FIG. 11) is based on the character attribute of the target character 8A and the emotional attribute corresponding to the action performed by the performer 5 (the action determined by the state determination unit 245). To specify the correction coefficient corresponding to the combination of the character attribute and the emotional attribute. Then, the operation command selection unit 247 multiplies the specified correction coefficient by the total value of the first parameter and the second parameter (here, the sum of the first parameter and the second parameter). , Execute the process of correcting the total value.

FIG. 12 is a diagram illustrating a correction process using the above correction table (FIG. 11).

In the example shown in FIG. 12, the correction process is shown when the character attribute of the target character 8A is "nervous middle-aged male" and the word spoken by the performer 5 is "boring". When the character attribute of the target character 8A is "nervous middle-aged male", the correction coefficients "2.0" and "1. 2 ”,“ 1.2 ”,“ 0.3 ”, and“ 2.0 ”are used (hereinafter, these correction coefficients are referred to as“ correction coefficients for nervous middle-aged men ”).

And each stage of FIG. 12 shows an example of the case where the state (behavior) of the performer 5 is different from each other. The table in the first row (top row) of FIG. 12 is an example in which the action of the performer 5 is a "one-handed raising gesture", and the table in the second row is an example in which the action of the performer 5 is a "arm-folding gesture". In the third table, the action of the performer 5 is "gesture to cover the face with hands", and in the fourth table, the action of the performer 5 is "". It is an example when it is a "jump gesture", and the table in the fifth row is an example when the action of the performer 5 is a "banzai gesture".

As shown in the first row, when the state (behavior) of performer 5 is "one-handed raising gesture", the emotional attribute "joy" corresponding to the one-handed raising gesture among the correction coefficients for nervous middle-aged men. The correction coefficient "1.2" associated with is applied. In this case, the first parameter is "-5", the second parameter is "10", and the total value before correction is "5". Therefore, the correction coefficient "1.2" is multiplied to correct the value. The total value after that is "6".

As shown in the second row, when the state (behavior) of performer 5 is "arm-folding gesture", it corresponds to the emotional attribute "anger" corresponding to the arm-folding gesture among the correction coefficients for nervous middle-aged men. The attached correction coefficient "0.3" is applied. In this case, the first parameter is "-5", the second parameter is "-10", and the total value before correction is "-15", so the correction coefficient "0.3" is multiplied. , The corrected total value is "4.5".

As shown in the third row, when the state (action) of performer 5 is "gesture to cover the face with hands", it corresponds to the gesture to cover the face with hands among the correction coefficients for nervous middle-aged men. The correction coefficient "1.2" associated with the emotional attribute "sorrow" is applied. In this case, the first parameter is "-5", the second parameter is "1", and the total value before correction is "-4". Therefore, by multiplying by the correction coefficient "1.2", The corrected total value is "-4.8".

As shown in the 4th row, when the state (behavior) of performer 5 is "jump gesture", it corresponds to the emotional attribute "comfort" corresponding to the jump gesture among the correction coefficients for nervous middle-aged men. The attached correction coefficient "2" is applied. Since the first parameter is "-5", the second parameter is "15", and the total value before correction is "10", the total value after correction is obtained by multiplying by the correction coefficient "2". It becomes "20".

As shown in the 5th row, when the state (behavior) of performer 5 is "Banzai gesture", it corresponds to the emotional attribute "surprise" corresponding to Banzai gesture among the correction coefficients for nervous middle-aged men. The attached correction coefficient "-5" is applied. In this case, the first parameter is "-5", the second parameter is "-5", and the total value before correction is "-10". Therefore, the correction coefficient "2" is multiplied to correct the value. The total value after that is "-20".

[4.4.4] Judgment processing In addition, the operation command selection unit 247
It is determined whether or not the total value after correction satisfies a given condition, and if the given condition is satisfied, an operation command is selected based on the total value after correction, otherwise the total value before correction is selected. Select the operation command based on the value.

For example, the operation command selection unit 247 determines whether or not the corrected total value satisfies, for example, "a condition that the value deviates from the predetermined range B2 to B5 in FIG. 13", and if the condition is satisfied, the corrected total value is corrected. The operation command is selected based on the total value, or the operation command is selected based on the total value before correction.

In the example of FIG. 12, when the performer 5 makes a one-handed raising gesture, an arm-folding gesture, or a gesture of covering the face with a hand (first step, second step, and third step in FIG. 12), the corrected gesture is performed. Since the total value (6,4.5, -4.8) is within the predetermined range B2 to B5 in FIG. 13, the operation command is based on the total value (6,4.5, -4.8) before correction. Is selected. In this case, the target character 8A's uniqueness (character effect) does not appear in the operation of the target character 8A.

Further, when the performer 5 makes a jump gesture or a banzai gesture (4th and 5th steps in FIG. 6), the corrected total value (20, -20) deviates from the predetermined range B2 to B5 in FIG. Therefore, the operation command is selected based on the corrected total value (20, -20). In this case, the operation of the target character 8A expresses the character 8A-likeness (character effect).

[4.4.5] Selection of operation command The operation command selection unit 247 uses the corresponding table of FIG. 14 for selection of the operation command. The corresponding table of FIG. 14 is stored in the database 275 in advance. As shown in FIG. 14, the correspondence table is a table showing the correspondence between the ranges B1, B2, ..., B6 to which the total value belongs and the operation commands associated with each range.

For example, the operation command selection unit 247 refers to the corresponding table (FIG. 14) according to the value of the total value, and selects the operation command corresponding to the total value as the operation command to be used for the operation control of the target character 8A. ..

In the first selection process, the above-mentioned determination process is executed. Therefore, when the corrected total value is within the predetermined range B2 to B5 in FIG. 13, the correction coefficient corresponding to the character attribute of the target character 8B Does not contribute to the selection of the operation command, and when the total value after correction deviates from the predetermined range B2 to B5 in FIG. 13, the correction coefficient according to the character attribute of the target character 8B contributes to the selection of the operation command. ..

That is, in the first selection process, when the corrected total value deviates from the predetermined range B2 to B5 in FIG. 13, that is, when the influence of the character attribute is more than a certain level, the operation of the target character 8A is more like the target character 8A. If it is expressed, otherwise, the operation of the target character 8A does not express the character 8 likeness.

[4.4.6] Confirmation information In the monitor unit of the processing unit 200, when the total value after correction deviates from the predetermined range B2 to B5 in FIG. 13, the text "The effect of the character attribute is being expressed". A control process is executed in which the image is displayed on the monitor display unit 290, and if it does not come off, the text image "the effect of the character attribute is being expressed" is not displayed on the monitor display unit 290. That is, in the first selection process, the text image (upper part of FIG. 4) is displayed on the monitor display unit 290.

According to the above configuration, the performer 5 can perform a performance (talk live) while confirming the image, and can visually grasp the basis for selecting the operation command. Therefore, the movement of the target character 8A can be performed as desired by the performer 5. It becomes easy to get close to.

[4.5] Second Selection Process Next, a second selection process for selecting an operation command will be described.

In the second selection process, before selecting the operation command, a third parameter is generated based on the character attribute, and the operation command is generated according to the total value of the first parameter, the second parameter, and the third parameter. Perform the selection.

[4.5.1] Third Parameter Generation Process The character attribute of the character table (FIG. 7) stored in the database 275 in order to realize the second selection process includes an emotional attribute.

Further, the processing unit 200 of the image generation device 20C specifies the emotion attribute corresponding to the character attribute of the target character 8A based on the database 275, and generates a third parameter corresponding to the emotion attribute. A unit (an example of a third parameter generation processing means) is further provided.

Here, the "third parameter" includes a numerical value (score, rank, level, stage, sign, etc.) indicating a negative degree or a positive degree, which is an emotional tendency peculiar to the character attribute.

For example, in the present embodiment, as shown in FIG. 7, each character attribute stored in advance in the character table is classified into "cheerful", "slightly cheerful", "calm", "nervous", and "gloomy". The emotional attributes shown are included, and each character attribute is assigned a positive score according to the emotional attribute included in the character attribute.

Therefore, when the third parameter generation processing unit refers to the character table (FIG. 7) according to the character attribute of the target character 8A and recognizes the positive score associated with the character attribute, the third parameter generation processing unit determines the score. Generated as a parameter of. As described above, if the character table (FIG. 7) is used, complicated arithmetic processing is not required, so that the third parameter can be generated at high speed.

[4.5.2] Selection of operation command The operation command selection unit 247 determines the operation command based on the total value of the first parameter, the second parameter, and the third parameter, and the range as the selection criterion. Make a selection.

For example, the operation command selection unit 247 executes the following processes (i) to (iv).

(I) The operation command selection unit 247 calculates the sum of the first parameter, the second parameter, and the third parameter as the first total value, and calculates the sum of the first parameter and the second parameter as the first total value. Calculated as the total value of 2.

(Ii) The operation command selection unit 247 determines whether or not the first total value satisfies, for example, "a condition that the value deviates from the predetermined ranges B2 to B5 in FIG. 13". If the condition is satisfied, the process (iii) is executed, otherwise the process (iv) is executed.

(Iii) The operation command selection unit 247 sets up a corresponding table (FIG. 14) according to the range (here, B1 or B6) to which the first total value belongs in the ranges B1, B2, ..., B6 in FIG. With reference, the operation commands associated with the range (here, four types of operation commands) are selected as the operation commands to be used for the operation control of the target character. The correspondence table of FIG. 14 is a table showing the correspondence between the ranges B1, B2, ..., B6 and the operation commands associated with each range, and is stored in advance in, for example, the database 275.

(Iv) The operation command selection unit 247 sets up a corresponding table (FIG. 14) according to the range (here, B2 to B5) to which the second total value belongs in the ranges B2, B2, ..., B5 in FIG. With reference, the operation commands associated with the range (here, four types of operation commands) are selected as the operation commands to be used for the operation control of the target character. The correspondence table of FIG. 14 is a table showing the correspondence between the ranges B1, B2, ..., B6 and the operation commands associated with each range, and is stored in advance in, for example, the database 275.

The operation command selection unit 247 provides information for specifying the combination of the selected operation commands (for example, information for specifying the range Bi corresponding to the selected operation command) to the section to which the operation command is applied. It is stored in the content data storage unit 274 together with the image content in association with each other.

Further, in the above description, the operation command is selected based on the first total value and the second total value, but the operation command may be selected without using the second total value. In that case, the process of calculating the second total value (the latter stage of (i)) and the process of selecting the operation command according to the second total value (iv) can be omitted. In this case, the determination process (ii) of whether or not the first total value belongs to the predetermined ranges B2 to B5 is omitted.

[4.6] Third Selection Process Next, a third selection process for selecting an operation command will be described.

In the third selection process, before selecting the operation command, a third parameter is generated based on the character attribute, the combination of the first parameter, the second parameter, and the third parameter, and the selection related to the combination. Set operation commands based on the criteria. Here, the differences from the second selection process will be mainly described, and the description of the common points will be omitted.

The operation command selection unit 247 has a feature quantity vector (an example of a combination of parameters) having the first parameter, the second parameter, and the third parameter as vector components, and the feature quantity space in which the feature quantity vector exists. The operation command is selected based on the separation surface (boundary of areas A1, A2, ...) As a selection criterion set in advance.

For example, the operation command selection unit 247 executes the following processes (i) to (iii).

(I) As shown in FIG. 15, the operation command selection unit 247 defines a feature space in which the first parameter, the second parameter, and the third parameter are represented by different axes. A plurality of regions A1, A2, A3, ... Different from each other are set in this feature space. In FIG. 15, a plurality of regions A1, A2, ... Are interspersed, but in reality, it is assumed that they are densely arranged in the feature space without any gaps. Further, in FIG. 15, the contours of the regions A1, A2, ... Are represented by ellipsoids, but in reality, they may be other types of hyperplanes. In addition, these areas A1, A2,
The data (selection criteria) that defines the boundaries (separation planes) of A3, ... Are stored in the database 275 in advance.

(Ii) When the operation command selection unit 247 recognizes the tip coordinates (coordinates indicated by X in FIG. 15) of the feature quantity vector having the first parameter, the second parameter, and the third parameter as vector components, the database Based on the selection criteria (boundary of the area A) stored in advance in 275, the area to which the coordinates belong is determined from among the plurality of areas A1, A2, A3, ... Defined in the feature space. Hereinafter, it is assumed that the coordinates of the feature vector (coordinates indicated by X in FIG. 15) belong to the region A2.

(Iii) The operation command selection unit 247 refers to the corresponding table (FIG. 16) according to the determined area (here, area A2), and the operation command (here, area A2) associated with the area (here, area A2). Four types of operation commands a2, b2, c2, d2) are selected as operation commands used for operation control of the target character 8A. The correspondence table shown in FIG. 16 is a table showing the correspondence between each area A1, A2, ..., An in the feature space and the operation command associated with each area, and is stored in advance in, for example, the database 275. ing.

Further, the operation command selection unit 247 provides information for specifying the combination of the selected operation commands (for example, information for specifying the area Ai corresponding to the selected operation command) to the section to which the operation command is applied. It is stored in the content data storage unit 274 in association with each other.

According to the above configuration, as shown in FIG. 15, it is possible to treat the third parameter related to the emotion of the character attribute in the same manner as the first parameter and the second parameter. Therefore, when an algorithm for selecting an operation command from the first parameter and the second parameter already exists, the above configuration can be realized by expanding the number (order) of the parameters of the algorithm.

[4.7] Character-like enhancement processing Next, the character-like enhancement processing will be described.

In the image generator 20C of this embodiment
For the multiple operation commands mentioned above,
Contains one or more specific operation commands associated with one or more specific character attributes.
For the processing executed by the operation command selection unit 247,
Includes a candidate setting process that sets candidates for operation commands selected from multiple operation commands.
In the candidate setting process, when the character attribute of the target character 8A is a specific character attribute, a process of adding a specific operation command to the candidate or a process of excluding it from the candidate may be executed.

Here, the "specific character attribute" may include, for example, a specific animal character (such as a cat character) and a specific character (such as a nervous middle-aged man).

Further, the "specific operation command" may include a command for converting the flexion to "nya", a command for adding a balloon "nya", a command for converting the flexion to a dialect of a specific region, and the like.

Hereinafter, some examples of executing the candidate setting process by adjusting the corresponding table (FIG. 14) will be described in order. Here, the corresponding table of FIG. 14 will be described as an example, but the corresponding table of FIG. 16 can be adjusted in the same manner.

[4.7.1] Processing to replace some operation commands in the corresponding table For example, the operation command selection unit 247 determines whether or not the character attribute of the target character 8A is a specific character attribute "cat character". However, in the case of the specific character attribute "cat character", any of the speech commands (speaking commands d1, d3 to d6) corresponding to at least one of the ranges B1 to B6 of the corresponding table in FIG. 14 ) Is replaced with a specific speech command d2 that adds "nya" to the end of the speech of the target character 8A. Therefore, when the character attribute of the target character 8A is a specific character attribute "cat character", it becomes easy to select a specific speech command d2 that adds "nya" to the end of the statement of the target character 8A.

Further, for example, the operation command selection unit 247 determines whether or not the character attribute of the target character 8A is a "dog character", and if it is a "dog character", the range B1 to the corresponding table in FIG. A process of replacing a speech command (any of the speech commands d1 to d4 and d6) corresponding to at least one of B6 with a specific speech command d5 that adds "one" to the end of the speech of the target character 8A. To execute. Therefore, when the character attribute of the target character 8A is a specific character attribute "dog character", it becomes easy to select a specific speech command d5 that adds "one" to the end of the speech of the target character 8A.

[4.7.2] Processing to exclude some operation commands from the corresponding table For example, in the operation command selection unit 247, whether the character attribute of the target character 8A is a specific character attribute "calm middle-aged male character". If it is determined whether or not it is, and if it is the specific character attribute "calm middle-aged male character", a specific utterance that adds "nya" to the end of the remark of the target character 8A from the corresponding table in FIG. Execute the process of excluding at least one command d2. In this case, when the character attribute of the target character 8A is a specific character attribute "middle-aged male character", it is difficult to select a specific speech command d2 that adds "nya" to the end of the statement of the target character 8A. Become.

[4.7.3] Process of adding an operation command to the corresponding table For example, the operation command selection unit 247 determines whether or not the character attribute of the target character 8A is a specific character attribute "cheerful gal character". , In the case of the specific character attribute "cheerful gal character", a process of adding an amplitude increase / decrease command e1 for amplifying the amplitude of the action is executed in the corresponding table of FIG. For example, the operation command selection unit 247 executes a process of additionally associating the amplitude increase / decrease command e1 with at least one of the ranges B1 to B6 of the correspondence table of FIG. Therefore, when the character attribute of the target character 8A is a specific character attribute "cheerful gal character", the amplitude of the action of the target character 8A is likely to be amplified (the overaction is likely to occur).

[4.7.4] Processing to edit the corresponding table Further, the operation command selection unit 247 performs a process of replacing a part of the operation commands of the corresponding table of FIG. 14 and an operation of a part of the corresponding table of FIG. At least two of the process of excluding the command and the process of adding the operation command to the corresponding table may be combined and executed (editing process of the corresponding table). Therefore, when the target character 8A has a specific character attribute, the correspondence table of FIG. 14 is edited in a direction that facilitates selection of a specific command corresponding to the specific character attribute, so that the character of the target character 8A You can appeal the uniqueness more strongly.

[4.8] Lottery process Next, an example in which a lottery element is added to the operation command selection process will be described.

In the above description, the operation command selection unit 247 uniquely selects the operation command based on the corresponding table, but it is also possible to select the operation command by the lottery process based on the corresponding table.

Here, an example of transforming the second selection process into a lottery process will be described. However, it is also possible to modify the first selection process or the third selection process in the same manner.

For example, a plurality of operation command candidates are associated with each column of the corresponding table in FIG. In addition, the selection probability is defined for each of the candidates for the plurality of operation commands in each column.

In selecting the operation command, the operation command selection unit 247 specifies, for example, the range Bi to which the total values of the first parameter, the second parameter, and the third parameter belong.

Next, the operation command selection unit 247 identifies a plurality of candidates for the facial expression command corresponding to the range Bi and the selection probabilities defined in the plurality of candidates in the corresponding table (FIG. 14), and sets the selection probabilities to the selection probabilities. By executing the lottery process based on the method, a process of selecting one candidate from a plurality of facial expression command candidates is executed.

Further, the operation command selection unit 247 identifies a plurality of candidates for the action command corresponding to the range Bi and the selection probabilities defined in the plurality of candidates in the corresponding table (FIG. 14), and determines the selection probabilities. By executing the lottery process based on, the process of selecting one candidate from a plurality of candidates is executed.

Further, the operation command selection unit 247 identifies a plurality of candidates for the visual effect command corresponding to the range Bi and the selection probabilities defined in the plurality of candidates in the corresponding table (FIG. 14), and identifies the selection probabilities. By executing the lottery process based on the above, a process of selecting one candidate from a plurality of candidates is executed.

Further, the operation command selection unit 247 identifies a plurality of candidates for the speech command corresponding to the range Bi and the selection probabilities defined in the plurality of candidates in the correspondence table (FIG. 14), and identifies the selection probabilities. By executing the lottery process based on the above, a process of selecting one candidate from a plurality of candidates is executed.

[4.8.1] Probability adjustment process Further, the operation command selection unit 247 may adjust the selection probability before the lottery process.

For example, the operation command selection unit 247 determines whether or not the target character 8A is a specific character, and if it is a specific character, 1 or 1 corresponding to the specific character in the corresponding table of FIG. A process for increasing or decreasing the selection probability of a plurality of candidates is executed.

For example, the operation command selection unit 247 determines whether or not the character attribute of the target character 8A is a specific character attribute "cheerful gal character", and when the character attribute is the specific character attribute "cheerful gal character". Executes a process for improving the selection probability of a specific operation command (Banzai gesture action command, jump gesture action command, etc.) corresponding to a hilarious gal character. Since the selection probabilities of a plurality of candidates are defined as relative values to the last, the operation command selection unit 247 lowers the selection probabilities of other candidates instead of improving the selection probabilities of a certain candidate. May obtain the same effect.

Further, for example, the operation command selection unit 247 determines whether or not the character attribute of the target character 8A is a specific character attribute "cat character", and if it is the specific character attribute "cat character", the operation command selection unit 247 determines. A process for improving the selection probability of a specific operation command (a specific speech command that adds "Nyan" to the end of the speech of the target character 8A) associated with the cat character in advance is executed. Since the selection probabilities of a plurality of candidates are defined as relative values to the last, the operation command selection unit 247 lowers the selection probabilities of other candidates instead of improving the selection probabilities of a certain candidate. May obtain the same effect.

[4.9] Viewer Popularity Reflection Processing Next, a popularity reflection process that reflects the viewer's popularity (high evaluation) in the operation of the target character 8B will be described.

In the image generator 20C of the present embodiment, the processing unit 200 is
A collection unit (an example of collection means) for executing a collection process for collecting evaluation data performed by the viewer on the output image content (FIG. 3) of the target character 8A is further provided.
The operation command selection unit 247
A plurality of operation commands based on a plurality of operation commands applied to a plurality of image data with sound (FIG. 3) constituting an image content and a plurality of evaluation data related to a plurality of image data with sound (FIG. 3). Is executed, and the operation command for generating new image data with sound (FIG. 3) is selected based on the weights of the plurality of operation commands.

According to the above configuration, the evaluation data of the viewer can be reflected in the movement of the target character 8A, so that the movement popular among the viewers can be easily expressed, or a given condition is satisfied. It is also possible to express popular movements in such cases. If the above operations of the operation command selection unit 247 are sequentially performed during the output of the image data with sound (FIG. 3), the popular operations will be reflected in real time, so that the viewer's participation in the live distribution It is possible to raise awareness and increase the number of viewers.

Hereinafter, an example of executing the popularity reflection process in the second selection process will be described. However, the popularity reflection process can be similarly executed in the first selection process or the third selection process.

[4.9.1] Specific example of evaluation data Here, "evaluation data" includes presents from viewers for the target character (clothes, clothing, funds for dressing, throwing money, twisting, etc.). Data such as the number and value of the data, the number of votes of the target character obtained by the vote performed by the viewer, and the viewing rate of the image content including the image of the target character are included.

In addition, in voting, the number of times the viewer presses the evaluation button (so-called like button) on his / her terminal device while viewing the image content, and the viewer from his / her terminal device to the operator while viewing the image content. This includes the number of times the comment is sent to the user, the number of times the viewer introduces the image content to others via SNS, and the like.

In addition, the direct or indirect operator of the image generation device may charge the viewer for presents, votes, and viewing of image contents.

[4.9.2] Processing of the server device related to the collection of evaluation data The viewer management unit 103 of the server device 11 is a terminal device of the viewer via the communication control unit 101 and the network during the distribution of the image content. It communicates with each of 10A, 10B, ..., And accepts a request from a viewer for a present or a vote (hereinafter referred to as "present, etc.") for the target character 8A.

In addition, the viewer management unit 103 charges the viewer who requests the present or the like as necessary. The billing process is a process of deducting (settlement) the amount of currency according to the contents of the present or the like from the virtual account (described above) corresponding to the viewer ID of the viewer. It is assumed that the server device 11 provides a special event related to the target character 8A to the viewer who is the sender of the present at an appropriate timing such as after the distribution of the image content is completed.

Further, the viewer management unit 103 records the history of presents and the like (value of presents and the like, presence / absence of voting) during distribution of the image content in the viewer information 146. For example, during the distribution of image content, a predetermined monitoring period (for example, a 30-second throwing time) is set at the timing when the target character 8A performs a predetermined operation, and a present from the viewer is given during the monitoring period. Accept. Then, the received present or the like is recorded in the viewer information 146 for each viewer ID in association with the corresponding section of the image content. Hereinafter, it is assumed that information such as gifts is recorded for each section of the image content. The section of the image content is set to, for example, about 30 seconds.

Further, the viewer management unit 103 monitors the number of terminal devices 10A to which the image content is distributed, and the audience rating during the distribution of the image content (the number of the terminal devices 10 to which the image content is distributed may be itself. ) Is recorded in the viewer information 146 for each section of the image content, for example.

As a result of the above, in the viewer information 146, the evaluation data of the entire viewer for the image content (number and value of presents for each section, number of votes for each section, viewing rate for each section) is included for each section of the image content. Will be accumulated in.

[4.9.3] Collection processing of evaluation data by the image generator For example, every time the viewer management unit 103 of the server device 11 updates the viewer information 146, the evaluation data related to the update (in the latest section). The number and value of the associated presents, the number of votes associated with the latest section, and the audience rating associated with the latest section) are transmitted to the image generator 20C.

On the other hand, the collecting unit of the image generation device 20C is associated with the evaluation data received from the server device 11 (the number and value of presents associated with the latest section, the number of votes associated with the latest section, and the latest section. By sequentially accumulating the viewing rate) in, for example, the database 275, data on the evaluation of the entire viewer for the image content being distributed is collected.

[4.9.4] Weighting process by the image generator Then, the operation command selection unit 247 refers to the content data storage unit 274 and identifies the operation command applied to each of the delivered plurality of sections (FIG. 3). do.

Here, in the present embodiment, since a plurality of operation commands can be applied to each section of the image content as described above, the operation command selection unit 247 specifies a combination of operation commands applied to each section. do. Since the combination of operation commands applied to each section corresponds to any of the ranges B1 to B6 in FIG. 13, the combination of operation commands is represented here according to the range B1 to B6 in FIG.

In addition, the operation command selection unit 247 specifies evaluation data (value and number of presents, number of votes) related to a plurality of sections (FIG. 3) stored in the database 275. Then, the operation command selection unit 247 weights the combination of operation commands (ranges B1 to B6 in FIG. 13) applied to each section according to the evaluation data (value and number of presents, number of votes) of each section. Execute the process to be performed.

For example, the operation command selection unit 247 sets the weight of the range Bi corresponding to each section to be larger as the value of the present associated with the section is higher, and is set to be larger as the number of presents associated with the section is larger. However, the larger the number of votes associated with the section, the larger the setting.

FIG. 17 is a graph showing the weights given to each of the ranges B1 to B6 according to the operation commands of the sections generated during the period from the start of the talk live to the present time. In the example of FIG. 17, the case where the weight of the range B1 is the largest and the weight of the range B3 is the smallest is shown. In this case, it is shown that the combination of the operation commands corresponding to the range B1 was the most popular, and the combination of the operation commands corresponding to the range B3 was the least popular.

However, "popularity" here is not the popularity of the operation command itself, but the popularity of "what kind of operation command was selected in what case". In other words, "popularity" is popular when "selection of operation commands" is appropriate for the viewer (when it is as expected), and "selection of operation commands" is inappropriate for the viewer. Less popular (if not as expected).

[4.9.5] Update processing of selection criteria based on popularity vote The operation command selection unit 247 has expanded the size of a range having a relatively large weight (range B1 in FIG. 17) based on the graph of FIG. , The boundary (selection criterion) of the ranges B1 to B6 is updated in the direction in which the size of the range having a relatively small weight (range B3 in FIG. 17) is reduced.

For example, as shown in FIGS. 17 to 18, the operation command selection unit 247 has a boundary between ranges B1 to B6 so that each size of ranges B1 to B6 becomes a size corresponding to each weight of ranges B1 to B6. Adjust (selection criteria). In the examples of FIGS. 17 and 18, the size of the popular range B1 is increased and the size of the less popular range B3 is reduced.

According to the above update process, it becomes easy to select the operation commands that have been popular from the start of the image content to the present time, and it becomes difficult to select the operation commands that have been less popular.

[4.9.6] Feedback processing In the image generation device 20C of the present embodiment, the processing unit 200 is
A feedback unit (an example of feedback means) that executes an update process for updating selection criteria based on the weights of a plurality of operation commands each time a given condition is satisfied may be further provided.

For example, the feedback unit has the above-mentioned range every time a predetermined time elapses, each section of the image content is generated, each frame of the image content is generated, or every time the weight changes. The operation command selection unit 247 is made to execute the update process of the boundary of B.

For example, the feedback unit selects an operation command every time a predetermined time elapses, in which the size of the popular range B is increased and the size of the less popular range B is reduced as shown in FIGS. 17 to 18. Have unit 247 execute.

Therefore, for example, when the target character 8A has a relatively high popularity of "jumping action A while saying" hungry "" at the beginning of the talk live, A is likely to appear, and a similar "action A" is likely to occur. The action A'" that jumps while saying" troublesome "is also likely to occur, but if the action A'is significantly less popular than the action A, the action A'is less likely to occur.

According to the above configuration, the selection criteria (boundaries of ranges B1 to B6) can be sequentially changed in a direction in which popular movements are likely to occur, so that the target character 8A is a popular character that performs the movements expected by the viewer. It will gradually approach. Further, since the above update process updates the selection criteria themselves (boundaries of ranges B1 to B6), the selection criteria (boundaries of ranges B1 to B6) are used as universal selection criteria for improving the popularity of various target characters 8A. In principle, it is also possible to converge to.

[4.10] Variation of operation command As described above, in the image generator 20C of the present embodiment,
For multiple operation commands that are options for operation commands,
Operation commands (facial expression commands) that control the facial expressions of the target character 8A,
Operation commands (action commands) that control the behavior of the target character 8A,
Operation commands (visual effect commands) that give visual effects to the target character 8A,
Operation commands (speech commands) that control the speech of the target character 8A, and
At least one of is included.

According to the above configuration, the emotion of the performer 5 can be reflected in at least one of the facial expression, action, visual effect, and speech of the target character 8A.

[4.11] Real-time distribution processing As described above, in the image generator 20C of the present embodiment,
The image output unit 240 is performing the image data (FIG. 3) of the target character 8A sequentially generated during the performance of the performer 5 (during the talk live) to the viewer of the image data (FIG. 3) (during the talk live). Is output sequentially to.

For example, the image output unit 240 sequentially transmits the image data with sound (frame data of the image content) of each section generated sequentially during the talk live to the server device 11, and the server device 11 sequentially transmits the image output unit 11. The image data with sound (frame data of the image content) of each section received from 240 is sequentially converted into streaming data and distributed to the viewer's terminal devices 10A, 10BA, ....

According to the above configuration, the image output unit 240 can live-stream (live broadcast) the image data with sound (FIG. 3) of the target character 8A in cooperation with the server device 11.

In the present embodiment, the timing at which the image data with sound (frame data of the image content) is generated and the timing at which the image data with sound (frame data of the image content) is output from the terminal devices 10A, 10B, ... Even in the case of near live where a delay time (time lag) occurs between and, it shall be included in the above "live distribution".

[4.12] Confirmation information The processing unit 200 of the image generation device 20C of the present embodiment is
A monitor unit (an example of monitoring means) that executes a control process for sequentially outputting an image showing the balance of the first parameter, the second parameter, and the character attribute to the performer 5 during the performance (during talk live) is provided.

For example, when the second selection process or the third selection process is executed, the monitor unit of the processing unit 200 may use a radar chart showing the balance of the first parameter, the second parameter, and the third parameter. The graph image (FIG. 19) of the above is generated every time the image data with sound (frame data of the image content) of each section is generated, and the graph image is included in the above-mentioned confirmation information (FIG. 4) and monitored. The control process for outputting to the display unit 290 is executed.

Further, in the first selection process or the second selection process, the confirmation information (FIG. 4) includes, in addition to the graph image (FIG. 19) such as a radar chart, "the effect of the character attribute is being expressed" and the like. Text image of, and type information of the selected operation command may be included.

For example, when the first selection process is executed, the monitor unit of the processing unit 200 determines whether or not the corrected total value belongs to the "predetermined range B2 to B5 in FIG. 13" with audio for each section. It is judged every time image data (frame data of image content) is generated, and if it does not belong, a text image "The effect of the character attribute is being expressed" is displayed on the monitor display unit 290, and if it belongs, "Character" is displayed. The control process of not displaying the text image "the effect of the attribute is being expressed" on the monitor display unit 290 is executed.

For example, when the second selection process is executed, the monitor unit of the processing unit 200 determines whether or not the first total value belongs to the "predetermined range B2 to B5 in FIG. 13" with audio for each section. It is judged every time image data (frame data of image content) is generated, and if it does not belong, a text image "The effect of the character attribute is being expressed" is displayed on the monitor display unit 290, and if it belongs, "Character" is displayed. The control process of not displaying the text image "the effect of the attribute is being expressed" on the monitor display unit 290 is executed.

According to the above configuration, the performer 5 acts (talks live) while checking the image displayed on the monitor display unit 290, and the basis for selecting the operation command can be visually grasped. Therefore, the target character can be visually grasped. It becomes easy to bring the movement of 8A closer to the movement desired by oneself.

[4.13] Self-regulation processing Next, the self-regulation processing will be described.

In the image generator 20C of this embodiment
The image output unit 240
Delay output processing that provides a delay time from the generation of the image data with sound (FIG. 3) of the target character 8A to the output (delivery) of the image data with sound (FIG. 3) of the image content to the viewer in the object space. Is executed, and when the operation input is received from the performer 5 or the assistant within the delay time, the process of adjusting the image (FIG. 3) output to the viewer may be executed. In this case, it is possible to self-regulate that the movement that the performer 5 or the assistant deems inappropriate is blocked before being output to the viewer.

For example, the image output unit 240 has a speed (frame rate) at which image data with sound (frame data of image content) is accumulated in the buffer 272 and image data with sound (frame data of image content) from the buffer 272 to the server device 11. ) Is transmitted (frame rate), the timing at which the image data with sound (frame data of the image content) is accumulated in the buffer 272, and the image data with sound (frame data of the image content) are set. A delay time for a plurality of sections (for example, a delay time of 5 minutes) is provided before being transmitted to the server device 11. That is, in the buffer 272, the image data with sound of the latest section corresponding to the delay time is accumulated.

Then, the image output unit 240 determines whether or not a predetermined operation input (regulation instruction) has been received from the performer 5 or the assistant via the input unit 260 within the delay time, and if it has received the reception. A process of replacing the image data with sound (frame data of the image content) of the section stored in the buffer 272 at the time with another image data with sound (frame data of the image content) is executed.

Here, the "another image data with sound" is, for example, an image data including a text such as "inappropriate expression is being regulated" and a sound data including a buzzer such as a regulated sound (so-called beep sound). It is composed of combinations. However, the audio data may not be replaced, or the audio data may be replaced or not by the performer 5 or an assistant as necessary.

Further, for example, in the second selection process, the image generator 20C may execute the following replacement process.

That is, the operation command selection unit 247, the operation control unit 248, and the image generation unit 249 perform the image data with sound based on the above-mentioned first total value (the first image with sound reflecting the character attribute) during the talk live. The process of generating (data) and the process of generating image data with sound (second image data with sound that does not reflect character attributes) based on the second total value are executed in parallel, and a regulation instruction is input. In this case, the second image data with sound may be transmitted to the server device 11 as the image data with sound of the image content regardless of the magnitude of the first total value.

[4.14] Automatic selection of operation command The operation command selection unit 247 monitors the combination of the first parameter, the second parameter, and the character attribute (or the third parameter) during the talk live, and the relevant operation command is selected. If the combination does not change for a predetermined time (for example, 1 minute), a given operation command may be selected regardless of the selection criteria described above. The given operation command is an operation command for making the target character 8A perform a natural movement such as "tapping the outside of the left thigh with the left hand and tapping the outside of the right thigh with the right hand". ..

According to the above configuration, it is possible to make the target character perform some action even during the period when the performer 5 is paused during the talk live (for example, during a break, during hydration, during stool, etc.). ..

[4.15] Flow FIG. 20 is an example of a flowchart of processing by the image generator 20C. It is assumed that the flowchart of FIG. 20 starts before the start of the talk live and ends after the end of the talk live. Since the division of functions inside the image generation device 20C has already been described, the main body of the processing will be described here as the image generation device 20C. Further, here, it is assumed that the selection of the operation command is executed by the second selection process or the third selection process.

First, the image generation device 20C determines whether or not there is an operation input for character designation by the performer 5 or an assistant (S101), and if there is such an operation input (S101Y), the process proceeds to the next process (S102). The transition is performed, otherwise the determination process (S101) is repeated. The character-designated operation input by the performer 5 or the assistant is performed via the input unit 260.

Next, when the image generation device 20C specifies a character designated by the performer 5 or the assistant based on the operation input, the image generation device 20C sets the specified character as the target character 8A in the object space (S102). The appearance (face, hairstyle, clothes, body shape, belongings, etc.) of the target character 8A is set according to the character attributes associated with the target character 8A.

Next, the image generator 20C determines whether or not a talk live start instruction has been input from the performer 5 or an assistant (S103), and if a start instruction has been input (S103Y), the next process (S103Y). The process proceeds to S104), or the determination process (S103) is repeated. The input of the start instruction by the performer 5 or the assistant is performed via the input unit 260. Further, when the live start instruction is input, the image generation device 20C inputs the start instruction of the shooting and sound collection processing to the performer data acquisition unit 250. After that, the performer data acquisition unit 250 starts shooting and sound collection processing.

Next, the image generation device 20C determines whether or not the performer data of the unprocessed section is stored in the buffer 272 (S104), and if it is stored (S104Y), the image content of the target character (specifically). In the case where the process shifts to the generation process (S105 to S113) of the latest section audio data and image data constituting the image content, and the performer data of the unprocessed section is not stored in the buffer 272 (S104N). , The process proceeds to the end determination process (S115).

Next, the image generation device 20C reads the performer data of the unprocessed section stored in the buffer 272 and executes a text conversion process for the voice data included in the performer data to convert the voice data into text. (S105).

Next, the image generator 20C extracts one or a plurality of words included in the text by executing a word extraction process for the text (S106).

Next, the image generator 20C identifies the emotional attribute and the positive score associated with the extracted one or a plurality of words by referring to the word table (FIG. 8) according to the extracted one or a plurality of words. , The image generator 20C generates the first parameter based on the specified emotional attribute and positive score (S107). For example, when the number of extracted words is "1", the image generator 20C uses the positive score of the word as the first parameter. Further, for example, when the number of extracted words is 2 or more, the image generator 20C identifies the most majority emotional attribute among the emotional attributes of those words and associates them with the emotional attribute. The positive score is used as the first parameter.

Next, the image generation device 20C determines the state (behavior) of the performer 5 by performing pattern recognition processing on the image data included in the performer data (S108). The process of determining the state (behavior) is as described above.

Next, the image generation device 20C specifies the emotional attribute associated with the determined action based on the state table (FIG. 9), and generates a second parameter corresponding to the emotional attribute (S109). For example, the image generation device 20C uses the positive score associated with the emotion attribute in the state table (FIG. 9) as the second parameter.

Next, the image generation device 20C specifies the character attribute of the target character 8A based on the character table (FIG. 7), and generates a third parameter corresponding to the character attribute (S110). For example, the image generation device 20C uses a positive score associated with the character attribute in the character table (FIG. 7) as a third paraator.

Next, the image generator 20C selects an operation command based on the first parameter, the second parameter, and the third parameter (S111). The process of selecting an operation command based on the first parameter, the second parameter, and the third parameter is as described above.

Next, the image generation device 20C controls the operation of the target character 8A set in the object space based on the selected operation command (S112).

At the same time, the image generation device 20C generates the frame data of the latest section including the image data and the audio data of the target character 8A and stores it in the buffer 272, and the frame of the oldest section stored in the buffer 272. The data is transmitted to the server device 11 (S113).

Next, the image generator 20C determines whether or not the end instruction of the talk live has been input (S114), and if the end instruction is input (S114Y), ends the flow, or otherwise determines. Returning to the process (S104), the process returns to the generation process (S105 to S113) of the image content (specifically, the audio data and the image data of the latest section constituting the image content). However, if the performer data of the unprocessed section is stored in the buffer 272 at the time when the talk live end instruction is input, the image generator 20C targets the performer data before ending the flow. Steps S105 to S113 are executed.

In the present embodiment, since the talk live is assumed, the case where the image generator 20C continuously receives the performer data has been described, but the case where the performer data is intermittently received (the performer data acquisition unit 250 intermittently receives the performer data). In the case of being driven), the image generation device 20C generates and outputs an image of the target character (performer reflection image) in which the movement of the performer 5 is reflected during the period in which the performer data is received, and outputs the performer data. During the non-accepting period, an image of a target character or the like (preset image) that performs a predetermined operation may be generated and output.

[5] Modification example [5.1] Amplification process according to character attribute In the above embodiment or modification, the first parameter, the second parameter, and the character attribute (or the third parameter determined by the character attribute or An example of selecting an operation command according to the correction coefficient) has been described, but after selecting the operation command according to the first parameter and the second parameter, the character attribute (or the third parameter determined by the character attribute) or The amplitude of the movement of the target character performed under the selected operation command (speech amplitude, facial expression amplitude, action amplitude, visual effect amplitude, etc.) may be adjusted according to the correction coefficient). In order to realize this adjustment, the operation command selection unit 247 may select the amplitude increase / decrease command together with other operation commands.

For example, when the selected operation command is an action command of "jump gesture", the height and speed of the jump may be adjusted according to the character attribute. For example, the height when the character attribute is "girl" may be adjusted to be larger than the height when the character attribute is "middle-aged male" (or may be adjusted to be smaller). .. Alternatively, the height when the character attribute is "gal character" may be adjusted to be larger than the height when the character attribute is "neat character".

Further, when the selected operation command is a visual effect command of "heart mark", the number and size of the heart mark may be adjusted according to the character attribute. For example, the number and size of heart marks when the character attribute is "girl" may be adjusted to be larger (or smaller) than the number and size when the character attribute is "middle-aged male". May be).

[5.2] Distribution mode of image content In the system of the above embodiment, image content (video content with audio) is distributed in real time using a network, but a system provided to viewers by television broadcasting and an event venue. It is also possible to transform it into a system that provides viewers with a large screen for the audience who come to the venue.

[5.3] Mode of viewing image content In the above embodiment, the viewer is made to view the content by outputting the image content from the terminal device, but the terminal device is an HMD (wearable image). It may be a display device). VR experience (virtual reality experience) that makes people recognize as if they exist around each viewer in the virtual space The talk performance of the target character may be viewed in 3D. In this case, for example, a terminal device having an HMD unit and a control unit is provided for each viewer, and each terminal device provides an image and a sound to the HMD based on various data output from the control unit. do.

Each terminal device may be provided to a plurality of spectators by the same event venue and the content may be distributed at the same time, or the content may be distributed at different timings depending on different spaces.

[5.4] Other Modifications The present invention includes a configuration substantially the same as the configuration described in the embodiment (for example, a configuration having the same function, method and result, or a configuration having the same purpose and effect). .. The present invention also includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced. In addition, the present invention includes a configuration that exhibits the same effects as the configuration described in the embodiment or a configuration that can achieve the same object. Further, the present invention includes a configuration in which a known technique is added to the configuration described in the embodiment.

As described above, the embodiments of the present invention have been described in detail, but those skilled in the art will easily understand that many modifications that do not substantially deviate from the novel matters and effects of the present invention are possible. .. Therefore, all such modifications are included in the scope of the present invention.

11: Server device 12: Large display device 100: Server processing unit 101: Communication control unit 103: Viewer management unit 120: Input unit 130: Display unit 140: Storage unit 142: Main storage unit 146: Viewer information 180: Information Storage medium 196: Communication unit 200: Processing unit 210: Object space setting unit 211: Touch detection processing unit 215: Movement processing unit 216: Virtual camera control unit 219: Content management unit 220: Drawing unit 230: Sound processing unit 250: Performer Data acquisition unit 250A: Sound collecting microphone 250B: Imaging camera 260: Input unit 262: Detection unit 270: Storage unit 271: Main storage unit 272: Buffer 274: Content data storage unit 280: Information storage medium 290: Monitor display unit 292: Sound output unit 296: Communication unit 241: Database management unit 242: Performer data input reception unit 243: Word extraction processing unit 244: First parameter generation processing unit 245: Status determination unit 246: Second parameter generation processing unit 247: Operation command selection unit 248: Operation control unit 249: Image generation unit 240: Image output unit 210A: Character setting unit

Claims (13)

An image generation device that reflects the movements of a performer existing in real space on the movements of a target character existing in an object space and generates an image of the target character operating in the object space.
Character attributes associated with each of the plurality of characters, emotional attributes defined by being associated with each of the plurality of words that the performer can present, and association with each of the plurality of states that the performer can take. A management means for managing a database that stores emotional attributes defined by the characters, a plurality of operation commands for controlling the movement of the target character, and the like.
A setting means for selecting the target character from the plurality of characters and setting the target character in the object space, and
An input receiving means for accepting input of performer data including the word presented by the performer, and
A word extraction processing means for extracting the word presented by the performer from the performer data, and
A first parameter generation processing means for specifying the extracted emotional attribute of the word based on the database and generating a first parameter corresponding to the emotional attribute.
A determination means for determining the state of the performer based on at least the image or sound of the performer included in the input data of the performer and reference data for determining the state.
A second parameter generation processing means that specifies the determined emotional attribute of the state based on the database and generates a second parameter corresponding to the emotional attribute.
Among the plurality of operation commands, one or a plurality of operation commands to be used for controlling the operation of the target character are the first parameter, the second parameter, and the character attribute of the selected target character. , Operation command selection means to select based on given selection criteria,
An operation control means that controls the operation of the target character set in the object space based on the selected operation command, and
An image generation means for generating an image of the target character operating in the object space, and
An image output means for outputting the generated image of the target character, and
An image generator characterized by comprising. In the image generator according to claim 1,
The operation command selection means is
Before selecting the operation command, a process of correcting at least one of the first parameter and the second parameter according to the character attribute of the target character is executed.
An image generator characterized by this. In the image generator according to claim 2,
The operation command selection means is
When the combination of the first parameter and the second parameter after correction satisfies a given condition, the operation command is selected based on the first parameter and the second parameter after correction. If the given condition is not satisfied, the operation command is selected based on the first parameter and the second parameter before correction.
An image generator characterized by this. In the image generator according to claim 1 or 2.
The character attribute includes an emotional attribute, and the character attribute includes an emotional attribute.
The image generator
A third parameter generation processing means for specifying an emotional attribute corresponding to the character attribute of the target character based on the database and generating a third parameter corresponding to the emotional attribute is further provided.
The operation command selection means is
The operation command is selected based on the combination of the first parameter, the second parameter, and the third parameter, and the selection criteria related to the combination.
An image generator characterized by this. In the image generator according to any one of claims 1 to 4.
The plurality of operation commands include
Contains one or more specific operation commands associated with one or more specific character attributes.
The processing executed by the operation command selection means includes
A candidate setting process for setting a candidate for the operation command selected from the plurality of operation commands is included.
In the candidate setting process, when the character attribute of the target character is the specific character attribute, a process of adding the specific operation command to the candidate or a process of excluding from the candidate is executed.
An image generator characterized by this. In the image generator according to any one of claims 1 to 5,
Further provided with a collection means for executing a collection process for collecting evaluation data performed by the viewer on the output image of the target character.
The operation command selection means is
Based on the plurality of the operation commands applied to the plurality of the images and the data of the plurality of evaluations related to the plurality of images, a process of weighting the plurality of the operation commands is executed to generate a new image. The operation command is selected based on the weights of the plurality of operation commands.
An image generator characterized by this. In the image generator according to claim 6,
Further provided is a feedback means for executing an update process for updating the selection criteria based on the weights of the plurality of operation commands each time a given condition is satisfied.
An image generator characterized by this. In the image generator according to any one of claims 1 to 7.
The plurality of operation commands include
An operation command for controlling the facial expression of the target character and
Operation commands that control the behavior of the target character and
An operation command that gives a visual effect to the target character,
Operation commands that control the speech of the target character and
Includes at least one of
An image generator characterized by this. In the image generator according to any one of claims 1 to 8.
The image output means
Images of the target character that are sequentially generated during the performance of the performer are sequentially output to the viewer of the image during the performance.
An image generator characterized by this. In the image generator according to any one of claims 1 to 9.
A monitoring means for sequentially outputting an image showing the balance of the first parameter, the second parameter, and the character attribute to the performer during the performance is further provided.
An image generator characterized by this. In the image generator according to any one of claims 1 to 10.
The image output means
A delay output process for providing a delay time between the generation of the image of the target character and the output of the image to the viewer in the object space is executed, and the performer or the assistant inputs an operation within the delay time. When accepted, the process of adjusting the image output to the viewer is executed.
An image generator characterized by this. In the image generator according to any one of claims 1 to 11.
The operation command selection means is
If the combination of the first parameter, the second parameter, and the character attribute does not change over a predetermined time, a given operation command is selected regardless of the selection criteria.
An image generator characterized by this. A program that reflects the movements of a performer existing in real space on the movements of a target character existing in an object space, and causes a computer to function as an image generation device that generates an image of the target character operating in the object space.
Character attributes associated with each of the plurality of characters, emotional attributes defined by being associated with each of the plurality of words that the performer can present, and association with each of the plurality of states that the performer can take. A management means for managing a database that stores emotional attributes defined by the characters, a plurality of operation commands for controlling the movement of the target character, and the like.
A setting means for selecting the target character from the plurality of characters and setting the target character in the object space, and
An input receiving means for accepting input of performer data including the word presented by the performer, and
A word extraction processing means for extracting the word presented by the performer from the performer data, and
A first parameter generation processing means for specifying the extracted emotional attribute of the word based on the database and generating a first parameter corresponding to the emotional attribute.
A determination means for determining the state of the performer based on at least the image or sound of the performer included in the input data of the performer and reference data for determining the state.
A second parameter generation processing means that specifies the determined emotional attribute of the state based on the database and generates a second parameter corresponding to the emotional attribute.
Among the plurality of operation commands, one or a plurality of operation commands to be used for controlling the operation of the target character are the first parameter, the second parameter, and the character attribute of the selected target character. , Operation command selection means to select based on given selection criteria,
An operation control means that controls the operation of the target character set in the object space based on the selected operation command, and
An image generation means for generating an image of the target character operating in the object space, and
An image output means for outputting the generated image of the target character, and
A program characterized by the functioning of a computer.

Images