JP7620056B2 - Program and computer system - Google Patents

Description

The present invention relates to a program for enabling a computer to generate video.

In recent years, so-called "YouTubers" have become known, who edit gameplay videos and add other audio (for example, commentary, commentary, or audio from chats with friends) to create commentary videos and distribute them on video distribution sites. Related technology is also known (see, for example, Patent Document 1). In recent years, a form of video commentary that appears to be provided by fictional characters known as "Vtubers" instead of YouTubers has also become known. Hereinafter, such fictional characters will be referred to as "commentary characters."

JP 2017-182603 A

Now, generally speaking, for a YouTuber to distribute a live commentary video, they must go through the steps of preparing a gameplay video, editing the video, adding commentary audio (often the YouTuber's own voice), and uploading it to the distribution site. Furthermore, to have a commentary character appear in the live commentary video, it is necessary to set the character's movements so that it behaves as if it were a play-by-play announcer or a sports program personality, generate a video of the commentary character, and composite this into the gameplay video. Setting the character's movements and preparing the commentary character's video requires a fair amount of equipment and time, so many users who want to try being a VTuber are unable to get started.

In addition, the action settings for the commentary character and the preparation of videos for the commentary character also placed restrictions on the format of commentary video distribution. Specifically, in the case of live-style commentary video distribution in which the YouTuber plays the game as a player and provides commentary in real time, the YouTuber must simultaneously play three roles: playing the game, providing commentary or commentary, and operating the commentary character. For this reason, it was difficult to realize the act of distributing games in genres in which the player cannot stop playing for even a moment (for example, action games, shooting games, flight simulators, etc.) in a live format using a commentary character.

The present invention was conceived with the objective of providing new technology to assist users involved in video distribution using live commentary characters.

The first invention for solving the above-mentioned problems is a program for making a computer system function as a live video generation means (e.g., server system 1100, live video generation server 1102 in FIG. 1, server processing unit 200s in FIG. 9, live video generation control unit 228, image generation unit 292s, step S16 in FIG. 19) that generates a video of a live commentary character who provides live commentary on a game play video and a live commentary video that combines the game play video, an analysis means (e.g., server system 1100, live video generation server 1102 in FIG. 1, server processing unit 200s in FIG. 9, analysis unit 224, step S34 in FIG. 19) that analyzes the play situation of the game play, and a live commentary character control means (e.g., server system 1100, live video generation server 1102 in FIG. 1, server processing unit 200s in FIG. 9, commentary character control unit 230, step S60 and step S62 in FIG. 20) that controls the live commentary character based on the analysis results of the analysis means.

The "computer system" referred to here can refer to a single computer, or it can be a system consisting of multiple computers working together.

According to the first invention, the play situation of the gameplay displayed as a gameplay video can be automatically analyzed, and the commentary character can be automatically controlled based on the analysis results. Therefore, the user does not need to create each and every movement of the commentary character and prepare a video in which the commentary character is controlled. In addition, since the need for user operations related to the movement control of the commentary character can be significantly reduced, the user can easily generate a live-style commentary video using a commentary character by simply inputting commentary or commentary audio even while playing the game. A new technology can be provided to assist users in video distribution using a commentary character.

The second invention is a program of the first invention, which causes the computer system to function as a voice acquisition control means (e.g., server system 1100 in FIG. 1, live video generation server 1102, server processing unit 200s in FIG. 9, voice acquisition control unit 222, step S12 in FIG. 19) that controls the acquisition of a user's voice, and the live character control means has a voice response control means that performs voice response control to control the behavior of the live character based on the voice acquired by the voice acquisition control means.

According to the second invention, the behavior of the commentary character can be automatically controlled in response to the user's voice. For example, when a user's voice indicating surprise is acquired, it is possible to automatically control the commentary character to make a surprised movement. This adds a live feel to the commentary character's movements, making the commentary video more expressive than when voice response control is not performed. Also, from the user's perspective, being able to control the commentary character by voice makes it easier to create commentary videos and distribute live commentary videos.

The third invention is a second invention program, in which the play-by-play video generation means generates the play-by-play video by taking a video of the gameplay being performed by the user as the play video, and the audio acquisition control means controls the acquisition of the user's audio while playing the game.

According to the third invention, the user can input voice while playing and control the commentary character with that voice, making it easier to create a commentary video of the gameplay.

The fourth invention is a program according to the third invention, in which the voice response control means controls the action of the commentator character based on the voice acquired by the voice acquisition control means when the play situation satisfies a voice reception situation condition indicating that the voice response control is acceptable.

According to the fourth invention, voice input is possible only when the play situation satisfies the voice reception situation condition, and the commentary character can be made to act according to the voice input at that time. In other words, it is possible to appropriately determine the timing at which the commentary character's action can be controlled by voice according to the play situation.

The fifth invention is a program according to any one of the second to fourth inventions, in which the live video generation means generates the live video using the user's voice acquired by the voice acquisition control means as live voice.

According to the fifth invention, a live commentary video can be generated using the user's voice as the live commentary voice.

The sixth invention is a program of the first invention, which causes the computer system to function as a comment acquisition control means (e.g., the comment acquisition control unit 229 in FIG. 23) that controls the acquisition of user comments, and the live commentary character control means has a comment response control means (e.g., the comment response control unit 236 in FIG. 23) that performs comment response control to control the behavior of the live commentary character based on the comments acquired by the comment acquisition control means.

According to the sixth aspect of the invention, it becomes possible to control the commentary character based on comments entered by the user.

The seventh invention is the program of the sixth invention, in which the live video generation means generates the live video by taking a video of the gameplay being performed by the user as the play video, and the comment acquisition control means controls the acquisition of comments made by the user while playing the game.

According to the seventh invention, the user can input comments while playing and control the commentator character with those comments.

The eighth invention is a program according to the seventh invention, in which the comment response control means controls the action of the commentator character based on the comment acquired by the comment acquisition control means when the play situation satisfies a comment acceptance situation condition indicating that the comment response control is acceptable.

According to the eighth aspect of the invention, it is possible to appropriately determine the timing at which the commentator character's actions can be controlled using comments, depending on the playing situation.

The ninth invention is a program of any one of the first to eighth inventions for causing the computer system to function as a distribution control means (e.g., server system 1100, distribution server 1103 in FIG. 1, server processing unit 200s, distribution control unit 240 in FIG. 9, step S16 in FIG. 19) that controls the distribution of the live commentary video or controls the distribution of the live commentary video by uploading data of the live commentary video to a specified distribution system.

According to the ninth invention, it becomes possible to distribute live video.

The tenth invention is a program according to any one of the first to ninth inventions for causing the computer system to function as a layout setting means (e.g., the server system 1100 in FIG. 1, the live video generation server 1102, the server processing unit 200s in FIG. 9, the layout setting unit 226, step S18 in FIG. 19) for setting the layout of the gameplay video and the live commentary character in the live commentary video.

According to the tenth invention, it becomes possible to set the relative positions and relative sizes of the gameplay video and the commentary character in the commentary video, i.e., the screen layout.

An eleventh invention is a program according to any one of the first to tenth inventions, in which the commentary character control means has a motion control pattern selection control means (e.g., the server system 1100 in FIG. 1, the commentary video generation server 1102, the server processing unit 200s in FIG. 9, the motion control pattern selection control unit 234, step S58 in FIG. 20) that presents a selection candidate for the motion control pattern of the commentary character to a user and controls the motion of the commentary character based on the motion control pattern of the selected selection candidate.

According to the eleventh invention, the user can select from among the selection candidates what he/she thinks is appropriate as an action for the commentary character. For example, even if the commentary character should make a surprised action, the nuance of the surprised action will differ depending on whether it is a genuine surprised action or a formal surprised action. And since the personality of the commentary character changes depending on which of these actions is made, the user can manage the detailed personality settings of the commentary character.

The twelfth invention is a program according to any one of the first to eleventh inventions, which causes the computer system to function as a motion control pattern setting means (e.g., server system 1100 in FIG. 1, live video generation server 1102, server processing unit 200s in FIG. 9, motion control pattern setting unit 220, step S8 in FIG. 19) that sets a motion control pattern of the commentary character based on a user's operational input, and the commentary character control means controls the motion of the commentary character based on the motion control pattern set by the motion control pattern setting means.

According to the twelfth aspect of the invention, the user can set the motion control pattern of the commentary character. Therefore, it is possible to make the commentary character perform original motions that are not included in the existing motion control patterns prepared by the manufacturer or administrator.

A thirteenth invention is a program according to any one of the first to twelfth inventions, in which the commentary character control means changes and controls the appearance of the commentary character.

According to the thirteenth aspect of the invention, the appearance of the commentary character can be changed along with the change in the movement control of the commentary character. For example, it is possible to change the appearance of the commentary character by making the character act as if surprised and by making the character's face turn pale.

A fourteenth invention is a program according to any one of the first to thirteenth inventions, in which the analysis means performs image analysis on the gameplay video and analyzes the gameplay situation based on the results of the image analysis.

According to the fourteenth aspect of the present invention, the gameplay situation can be analyzed by performing image analysis on the gameplay video.

The fifteenth invention is a program according to any one of the first to fourteenth inventions, in which the analysis means analyzes operation input information for the game play and analyzes the play situation based on the analysis results.

According to the fifteenth aspect of the present invention, the playing situation can be analyzed based on the operation input information of the game play.

The sixteenth invention is the program described in any one of claims 1 to 15, in which the play-by-play video generation means generates the play-by-play video by changing the playback speed of the video of the play-by-play character and/or the gameplay video based on the analysis results of the analysis means.

There is a video directing technique in which impressive or shocking scenes are played in slow motion to more effectively convey the impression or impact. There is also a directing technique in which scenes that are necessary for the passage of time or explaining a situation but are not important from the viewer's perspective are fast-forwarded to maintain the tempo of the story.
According to the sixteenth aspect of the present invention, such video expression techniques can be automatically applied to live commentary videos, making it possible to generate live commentary videos with rich dramatic effects.

The seventeenth invention is a computer system including a video of a commentary character who provides commentary on a gameplay video, a commentary video generation means for generating a commentary video by combining the gameplay video, an analysis means for analyzing the gameplay situation of the game, and a commentary character control means for controlling the commentary character based on the analysis results of the analysis means.

According to the seventeenth aspect, a computer system can be realized that can achieve the same effect as the first aspect.

FIG. 1 is a diagram showing an example of the configuration of a live video generation and distribution system. FIG. 4 is a front view showing a configuration example of a user terminal. FIG. 1 is a diagram for explaining an overview of live video generation. FIG. 13 is a diagram for explaining analysis of a play situation. 11A and 11B are diagrams for explaining the operation control of a commentator character based on analysis results. 13 is a diagram for explaining how to handle a case in which multiple action control patterns of commentary characters are assigned to one play situation. FIG. 11 is a diagram for explaining user settings of an operation control pattern. 11A and 11B are diagrams for explaining control of changing the playback speed of a gameplay video when a play-by-play video is created by adding commentary to an existing gameplay video. FIG. 2 is a functional block diagram showing an example of the functional configuration of the server system according to the first embodiment. 5A and 5B are diagrams showing examples of programs and data stored in a server storage unit in the first embodiment. FIG. 4 is a diagram showing an example of a data configuration of motion control pattern definition data. 5 is a diagram showing an example of the data structure of playback speed change pattern definition data; FIG. 4 is a diagram showing an example of the data configuration of game management data. FIG. 4 is a diagram showing an example of the data configuration of live management data. FIG. 4 is a diagram showing an example of a data configuration of user management data. FIG. 13 is a diagram showing an example of the data configuration of play video registration data. FIG. 4 is a diagram showing an example of the data structure of live video registration data. FIG. 4 is a functional block diagram showing an example of a functional configuration of a user terminal. 5 is a flowchart for explaining a process flow in the server system of the first embodiment. 20 is a flowchart continuing from FIG. 19 . FIG. 11 is a functional block diagram showing an example of a functional configuration of a user terminal according to the second embodiment. 13A and 13B are diagrams showing examples of programs and data stored in a terminal storage unit according to the second embodiment. FIG. 11 is a functional block diagram showing an example of the functional configuration of a server system according to a modified example of the first embodiment.

An example of an embodiment of the present invention will be described below, but it goes without saying that the forms to which the present invention can be applied are not limited to the following embodiment.

First Embodiment
FIG. 1 is a diagram showing an example of the configuration of a live video creation and distribution system according to the present embodiment.
The live video generation and distribution system 1000 is a computer system that provides a service of generating live video and distributing live video, and includes a server system 1100 and a plurality of user terminals 1500 (1500a, 1500b, ...) that are connected to each other via a network 9 so that data communication is possible.

The network 9 refers to a communication path that allows data communication. In other words, the network 9 includes a LAN (Local Area Network) such as a dedicated line (dedicated cable) for direct connection or Ethernet (registered trademark), as well as a communication network such as a telephone communication network, a cable network, or the Internet, and the communication method can be either wired or wireless.

The server system 1100 is a computer system having a game server 1101, a live video production server 1102, and a distribution server 1103.

The game server 1101 is a server system and computer system that manages the progress of a game to allow an online game to be played on a user terminal 1500 (1500a) used by a user 2 (2a) (hereafter referred to as the "player") who is a registered user of the live video generation and distribution system 1000 and is a player of the game. The game server 1101 has, for example, a keyboard 1106, a touch panel 1108, and storage 1140, and the main device is equipped with a control board 1150.

The control board 1150 is equipped with various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM, and ROM, and a communication device 1153. Note that a part or all of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a SoC (System on a Chip).

The game server 1101 realizes a user management function related to user registration and the like, and a function of providing data necessary for a registered user, user 2 (2a), to play the game on the user terminal 1500 (1500a), by the control board 1150 performing calculations based on a predetermined program and data. Note that in this embodiment, the user 2 (2a) is described as one person, but there may be multiple users.

The live video generation server 1102 can be realized by the same hardware as the game server 1101, and is a computer system that handles the processing related to the generation of live video. In this embodiment, it acquires gameplay video data from the game server 1101, and generates a live video by synthesizing a video of a live commentary character set by a user who is a registered user of the live video generation and distribution system 1000 and who will be the commentator (hereinafter referred to as the "commentary user") with the gameplay video. In this embodiment, an example is given of live-format live video distribution in which user 2 (2a) doubles as the commentator while playing the game, so in this embodiment, user 2 (2a) is also the commentary user.

The distribution server 1103 can be realized by the same hardware as the game server 1101, and is responsible for accepting the provision (so-called submission) of video data (in this embodiment, gameplay videos and live video), managing the video data, and distributing the video data. In this embodiment, the video data of the live video generated by the live video generation server 1102 is sequentially acquired, and distributed to the user terminals 1500 (1500c, 1500d, ...) of one or more viewers 3 (3a, 3b, ...).

In FIG. 1, each is depicted as a separate server device, but a single server device may be configured to realize the three server functions. In that case, server system 1100 may be configured to include multiple blade servers each sharing each function, connected to each other via an internal bus for data communication. Furthermore, the location of the hardware that constitutes server system 1100 does not matter. A configuration in which multiple independent servers installed in separate locations communicate data over network 9 to function as server system 1100 as a whole may also be used.

The user terminal 1500 (1500a, 1500c, ...) is a computer system that is used individually by the user 2 (2a, 2c, ...) to play games, generate live videos, and watch gameplay videos and live videos, and is an electronic device (electronic device) that can access the server system 1100 via the network 9. Functionally speaking, the user terminal 1500 is a game device, a live video editing device, and a video viewing device. The user terminal 1500 in this embodiment is a device known as a smartphone, but it may also be a portable game device, a tablet computer, a personal computer, etc.

FIG. 2 is a front view showing an example of the configuration of a user terminal 1500 according to this embodiment.
The user terminal 1500 includes a directional input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a contact position input device, a speaker 1510, an internal battery 1509, a microphone 1512, an image sensor unit 1520, a control board 1550, and a memory card reader 1542 that can read and write data from a memory card 1540 that is a computer-readable storage medium. In addition, a power button, a volume control button, etc. that are not shown are provided. In addition, an IC card reader that can read and write data contactlessly from an IC card type credit card or prepaid card that can be used to pay for playing games, etc. may be provided.

The control board 1550 is
1) Various microprocessors such as the CPU 1551, GPU, DSP, etc.
2) Various IC memories 1552 such as VRAM, RAM, ROM, etc.
3) A wireless communication module 1553 for wireless communication with a mobile phone base station or a wireless LAN base station connected to the network 9;
4) Equipped with an interface circuit 1557, etc.

The interface circuit 1557 includes a driver circuit for the touch panel 1506, a circuit for receiving signals from the directional input keys 1502 and button switches 1504, an output amplifier circuit for outputting audio signals to the speaker 1510, an input signal generation circuit for generating audio signals collected by the microphone 1512, a circuit for inputting image data of an image captured by the image sensor unit 1520, and a signal input/output circuit for the memory card reader 1542.

These elements mounted on the control board 1550 are electrically connected via a bus circuit or the like, and are connected so that data can be read and written and signals can be sent and received. Note that part or all of the control board 1550 may be configured using an ASIC, FPGA, or SoC. The control board 1550 stores programs and various data for realizing the functions of the user terminal of the game of this embodiment in IC memory 1552.

In this embodiment, the user terminal 1500 is configured to download programs and various setting data from the server system 1100, but it may also be configured to read them from a storage medium such as a memory card 1540 obtained separately.

FIG. 3 is a diagram for explaining an overview of generation of a live video in this embodiment.
In this embodiment, the user 2 (2a) serves both as a game player and a commentary user using the commentary character 7. The user 2 (2a) then performs the task of live-streaming a commentary video 12 that combines a gameplay video 10 with a video of the commentary character 7 commenting on the gameplay (video of the commentary character 7).

On the user terminal 1500 (1500a) of user 2 (2a), the game is executed by communicating with the game server 1101, and the game screen is displayed. The continuous live display of this game screen becomes the gameplay video 10 in this embodiment. Of course, if the distribution format is not the live format, the game screen can be separately saved in video format and used as the gameplay video 10.

The game genre is not important. The example in FIG. 3 shows an action game in which a player character 4 operated by a user 2 (2a) as a player fights an enemy character 6 operated by another user or a computer. In the game of this embodiment, for example, when an enemy is encountered on the battlefield, a battle with a time limit starts, and the side that is defeated within the time limit or the side that has suffered the most damage when the time limit has elapsed becomes the loser of that round. The side that first wins a predetermined number of times becomes the winner of the battle.

In this embodiment, the game server 1101 controls the progress of the game and:
1) Recording the history of operation input received from the user terminal 1500 (1500a);
2) Recording the history of various parameter values for controlling the progress of the game (hereinafter referred to as progress control parameter values);
3) Recording video data of the gameplay video 10;
4) Recording of voice data of various user voices uttered by the user 2 (2a) during game play collected by the microphone 1512 of the user terminal 1500 (1500a); and
Do the following.

The live video production server 1102 stores the registration information of the user 2 (2a) as a live user in association with the setting information of the live character 7 prepared in advance.

When a specific live video generation operation is input on the user terminal 1500 (1500a), the live video generation server 1102 starts acquiring various data from the game server 1101, including the gameplay video 10 (in this embodiment, a live game screen), the history of operation input information, the history of progress control parameter values, and user voice. Hereinafter, these will be collectively referred to as "play information."

Then, the live video production server 1102 analyzes the gameplay situation based on the play information. The "play situation" here refers to the progress of the game, and corresponds to the content of the gameplay that is the subject of live commentary. For example, "The player is in the lead," "The player's character is close to being defeated with one more hit," "A rare item has been obtained," "It's a close match," "Less than one minute left," etc.

Then, based on the analysis results of the play situation, the live video production server 1102 searches for a movement control pattern that defines a movement corresponding to the analysis results at that time from among the movement control patterns of the live character 7 that have been prepared in advance, and controls the movement of the live character 7 using the movement control pattern that has been searched for.

The state of the commentary character 7 in action (video of the commentary character 7) is synthesized with the gameplay video 10 to form the screen of the commentary video 12. In addition, the game sound included in the gameplay video 10 is synthesized with the user's voice, which becomes the commentary sound, to add sound and generate the commentary video 12.
Data of the live video 12 is sequentially transmitted from the live video generation server 1102 to the distribution server 1103, and the distribution server 1103 distributes the live video 12 in a live format.

In this embodiment, the live commentary video 12 is distributed in a live format, so that the control of the gameplay progress, the saving of the gameplay video 10, the control of the movements of the commentary character 7, the generation of the commentary video 12, and the control of distribution of the commentary video 12 are all performed simultaneously in parallel.

Previously, to distribute a live gameplay video, it was necessary to perform each step individually: recording the gameplay video, editing the video, adding commentary audio, and uploading it to a distribution site. Also, to have a commentary character appear in the live video, the commentator had to set the character's movements as if the character were a live announcer or a sports program personality, create a video of the commentary, and then composite the video.

However, in this embodiment, the commentary character 7 is automatically controlled, so the user 2 (2a), who is both the player and the commentary user, only needs to operate the game while making a commentary or explanation sound. It becomes possible to easily distribute a live commentary video 12 as a VTuber without having to go through the numerous steps of the conventional process.

In addition, user 2 (2a), who is both a player and a commentary user, can operate the game while emitting commentary or narration, making it easy to distribute commentary videos in a live format, especially for games in genres where the gameplay cannot be stopped even for a moment (e.g., action games, shooting games, flight simulators, etc.).

FIG. 4 is a diagram for explaining the analysis of the play situation in this embodiment.
In this embodiment, the play situation is analyzed using four methods.
As a first technique, the live video generation server 1102 performs image analysis of the gameplay video 10. For example, the image analysis dictionary data 512 defines a dictionary image to be compared with a captured image of the gameplay video 10 and a play situation to be recognized when the dictionary image matches. The live video generation server 1102 executes a process of determining image matching between the captured image and the dictionary image, and regards the play situation indicated by the image analysis dictionary data 512 of the matching dictionary image as the analysis result of the technique.

Of course, the image analysis method is not limited to this and can be selected appropriately. For example, to recognize the state of the player character 4 by image analysis, image analysis dictionary data 512 is prepared for analyzing the hit point gauge of the player character 4. Then, if it is recognized that the hit points are lower than a reference value, the play situation of "player character in trouble" may be determined as the analysis result of the method. Also, for example, the capture image may be searched for an image of the performance display when the enemy character 6 is defeated, and if a corresponding image is included, the play situation of "enemy defeated" may be determined as the analysis result of the method.

As a second method, the live video production server 1102 compares the progress control parameter values with the dictionary data 514 for progress control parameter values, and determines the playing situation that is pre-associated with the matching dictionary data 514 for progress control parameter values as the analysis result of the method. For example, when a recognition criterion is met that indicates that the value by which the number of wins of the player exceeds the number of wins of the opponent has reached a predetermined disadvantage criterion value, the playing situation of "the player is at a disadvantage" is determined as the analysis result of the method.

As a third method, the operation input information history of the user 2 (2a)'s gameplay is compared with the dictionary data for operation input information 516, and the play situation that is pre-associated with the matching dictionary data is taken as the analysis result of the method. For example, if the operation input information regarding the enemy character 6 is "executing a special move," the play situation associated therewith, such as "enemy executes a special move" or "player is in danger," may be taken as the analysis result of the method.

As a fourth technique, the live video generation server 1102 compares the user's voice with the voice recognition dictionary data 518, and determines the playing situation that is pre-associated with the matching dictionary data as the analysis result of the technique. For example, when keywords such as "bad," "bad," or "failed" are recognized by voice, the playing situation associated with them, such as "the player made a mistake" or "is the player in trouble!?", may be determined as the analysis result of the technique.

The fourth technique may be applied only when the play situation satisfies a specified voice reception situation condition. The "voice reception situation condition" referred to here is, for example, when an event occurs. The "event" referred to here may be the acquisition of a rare item, the defeat of the player character 4, the defeat of the enemy character 6, the volume of the user's voice exceeding a reference value, etc., and may be set as appropriate. Of course, there is no need to limit the application of the fourth technique based on the voice reception situation condition.

FIG. 5 is a diagram for explaining the operation control of the commentator character based on the analysis result.
The live video generation server 1102 stores a plurality of pieces of movement control pattern definition data 520 for each type of movement in advance. Each piece of movement control pattern definition data 520 includes a type of movement 521, application requirements 522 indicating the conditions under which the movement of the data definition data should be applied to the movement control of the live character, motion data 523 for realizing the movement, effect data 524 for realizing a performance display executed together with the movement, and appearance change data 525 for realizing an appearance change applied together with the movement. Of course, data other than these may also be included as appropriate.

The live video generation server 1102 searches for motion control pattern definition data 520 having application requirements 522 that match the analysis results. Then, it controls the motion of the live commentary character 7 according to the motion data 523 set in the searched motion control pattern definition data 520, performs a performance display according to the effect data 524, and changes the appearance of the live commentary character 7 according to the appearance change data 525.

In the example of FIG. 5, the analysis result indicates that the player character 4 is in a situation where "one more hit is needed to defeat it." The motion control pattern definition data 520 includes definition data in which application requirements 522 are set in correspondence with such a situation where "one more hit is needed to defeat it." In accordance with the motion data 523, the commentator character 7 makes a motion of clutching his head in panic, and a sweat object that comically indicates panic is added and displayed in accordance with effect data 524, and the appearance is changed to one with white eyes and an open mouth in shock in accordance with appearance change data 525.

By setting in the action control pattern definition data 520 detailed play situations predicted to occur under the game rules of the target game title and defining appropriate actions for the commentary character 7, the commentary character 7 will automatically take appropriate actions to comment on the play situation unfolding in the gameplay video 10. User 2 (2a), who is both the player and the commentary user, controls the actions of the commentary character 7 by inputting game play operations with his or her own hands and providing commentary and explanations with his or her own voice, so that a high-quality commentary video 12 can be easily created without user 2 (2a) having to input operations just to control the actions of the commentary character 7 every time.

FIG. 6 is a diagram for explaining how to handle a case in which a plurality of action control patterns of the commentator character 7 are assigned to one play situation.
Even in the same play situation, there are cases where a plurality of different actions are performed. For example, when the commentary character 7 performs a surprised action in response to a certain play situation, the surprised action may be a genuine surprised action or a formal surprised action for other purposes such as to elicit laughter. Of course, there may be other different surprised actions. Depending on the surprised action performed at that time, the character's characterization changes, and the overall flow of the presentation of the commentary video 12 is determined. This is an important choice for the user 2 (2a) who is the commentary user.

In this embodiment, when multiple definition data that match the current play situation are found, the live video production server 1102 presents them to the user 2 (2a) as selection candidates and accepts a selection operation. Then, the live video production server 1102 controls the movement of the live character 7 according to the selected definition data.

In the example of FIG. 6, two selection candidates 30 (30a, 30b) are searched for as movement candidates, and the live video production server 1102 overlays an action monitor image 32 showing the current movement of the live commentary character 7 and selection operation icons 33 (33a, 33b) corresponding to the selection candidates 30 (30a, 30b) on the game screen W6 (or a live operation screen not shown) on the user terminal 1500 (1500a) of the live commentary user. Then, the movement control pattern corresponding to one of the selection operation icons 33 touched by the user 2 (2a) is applied as the movement of the live commentary character 7.

Thus, in this embodiment, even if multiple action control patterns are assigned to one play situation, the user 2 (2a) can easily make important choices related to the commentary character 7 and commentary video 12 with just a few operations.

In addition, in a configuration in which the user terminal 1500 is equipped with a three-axis acceleration sensor, instead of selecting by touching the selection operation icon 33, the selection may be input by shaking or tilting the user terminal 1500. In this case, fingertip operations, which are important for game play, do not need to be used to select the action control pattern, which is preferable because it improves usability.

FIG. 7 is a diagram for explaining user settings of the operation control pattern.
The movement control patterns of the commentator character 7 are basically prepared by the manufacturer or administrator of the server system 1100, but can also be created and registered by the user himself.

When a user setting operation for a specific motion control pattern is detected on the user terminal 1500, the live video production server 1102 displays a motion control pattern setting screen W7 on the user terminal 1500. On this setting screen, it is possible to input an operation to call up a function for setting a new motion control pattern or editing an existing pattern.

In the new setting, a plurality of setting method selection icons 40 (40a, 40b, . . . ) are displayed.
In this embodiment, user 2 can choose to load a motion file that he or she has created separately, activate a motion capture function to create the basis for motion data, or create a new motion by combining pre-prepared motion elements.

Figure 7 shows an example of creating motion data material by activating the motion capture function. User 2 (2a) activates the motion capture function on user terminal 1500 (1500a) and fixes it in front of a green screen 44 or the like on a tripod 42. User terminal 1500 is oriented so that the camera of image sensor unit 1520 faces the green screen 44.

User 2 (2a) stands in front of the green screen 44, performs the action he or she wishes to set toward the user terminal 1500 (1500a), and has the user terminal 1500 (1500a) take a picture.

When the user terminal 1500 (1500a) transmits the captured video to the live video generation server 1102, the live video generation server 1102 recognizes the joint structure of the human body from the captured video, and based on the recognized movement of the human body's joints, creates motion data 46 to enable the live character 7 to reproduce the same movement, and transmits this to the user terminal 1500 (1500a). The user terminal 1500 (1500a) then displays a motion editing screen W8 that accepts editing operations for the received motion data 46.

User 2 (2a) cuts out unnecessary movement parts of the motion through editing operations on the motion editing screen W8 and sets an appropriate data name. When the user terminal 1500 detects a specified upload operation, it transmits the edited motion data 46 together with the user account to the live video production server 1102. The live video production server 1102 then registers and stores the uploaded movement control pattern definition data 520 in association with the received user account.

By using such a user setting service for movement control patterns, user 2 (2a) can impart originality to the commentator character 7 in terms of movement.

The process of generating motion data from captured images may be configured to be performed by the user terminal 1500. The motion data 46 may also be generated in other ways (for example, by using 3DCG creation software).

Up to this point, the explanation has been given on the premise of live-format distribution of the commentary video 12 in this embodiment, but in the commentary video generation and distribution system 1000, the user 2 (2a) can input commentary and explanation audio to an already recorded and registered gameplay video 10 (hereinafter referred to as an existing gameplay video 10), while automatically controlling the commentary character 7 to create the commentary video 12.

FIG. 8 is a diagram for explaining a technique for changing and controlling the playback speed of a gameplay video 10 when creating a play-by-play video 12 by adding commentary to an existing gameplay video 10. In FIG.
When creating a commentary video 12 using an existing gameplay video 10, if a keyword that changes a specified playback speed is detected in voice recognition of the user's voice, the commentary video generation server 1102 changes the playback speed of the gameplay video 10 that is the subject of commentary.

Specifically, as shown in FIG. 8(1), in a play situation where a production technique of showing the scene slowly to enhance the impression is effective, if a keyword predicted to be uttered by the user (for example, the examples in FIG. 8(1) are "Get a rare (item)!" or "Defeat the boss!") is detected by user voice recognition, the playback speed of the existing gameplay video 10 to be commented on is changed to slow playback for a certain period of time or until a specified release condition is met. However, the movements of the commentator character 7 are maintained at standard speed.

Also, as shown in FIG. 8 (2), if a voice keyword (for example, the examples in FIG. 8 (2) are "Let's skip through it quickly" and "Let's let it flow") is detected in user voice recognition, indicating that the play situation is one in which a production technique of fast-forwarding scenes that are necessary for explaining the passage of time and the situation but that tend to be boring for the viewer of the video to maintain the tempo of the progress is effective, the playback speed of the existing gameplay video 10 to be commented on is changed to fast-forward playback (meaning playback at a speed greater than or equal to 1x) for a certain period of time or until a specified release condition is met. However, the movements of the commentator character 7 are maintained at standard speed.

In this way, according to this embodiment, video expression techniques that make use of slow-motion and fast-forward playback can be automatically applied to the live commentary video 12, making it possible to generate live commentary videos with rich dramatic effects.

The change in playback speed may include reverse playback and repeated playback using the reverse playback, as appropriate. In addition, the speed of the movements of the commentary character 7 may be changed or the playback speed of the video of the commentary character 7 may be changed in accordance with the change in the playback speed of the gameplay video 10. The change in playback speed may also be applied to the distribution of the live commentary video 12 in a similar manner, provided that a replay video can be generated on the spot.

FIG. 9 is a functional block diagram showing an example of the functional configuration of the server system 1100.
The server system 1100 includes an operation input unit 100s, a server processing unit 200s, a sound output unit 390s, an image display unit 392s, a communication unit 394s, and a server storage unit 500s.

The operation input unit 100s is a means for inputting various operations for managing the server. This corresponds to the keyboard 1106 in FIG. 1.

The server processing unit 200s is realized by electronic components such as processors that are arithmetic circuits, such as a CPU, GPU, ASIC, FPGA, etc., as well as IC memory, and controls the input and output of data between each functional unit, including the operation input unit 100s and the server storage unit 500s. It performs various types of arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100s, data received from the user terminal 1500, etc., and performs integrated control of the operation of the server system 1100.

The server processing unit 200s includes a user management unit 210, a game management unit 212, a motion control pattern setting unit 220, an audio acquisition control unit 222, an analysis unit 224, a layout setting unit 226, a live video generation control unit 228, a live character control unit 230, a distribution control unit 240, a timing unit 280s, a sound generation unit 290s, an image generation unit 292s, and a communication control unit 294s. Of course, other functional units may also be included as appropriate.

In this embodiment, the user management unit 210 and the game management unit 212 correspond to the functions of the game server 1101. In addition, the action control pattern setting unit 220, the voice acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video production control unit 228, and the live character control unit 230 correspond to the functions of the live video production server 1102. In addition, the distribution control unit 240 corresponds to the function of the distribution server 1103. Of course, depending on the hardware configuration of the server system 1100, these may be realized by a single server device, and the division of functions and hardware is not limited to this.

The user management unit 210 handles the processing related to the user registration procedure and stores and manages the information of each user linked to the user account.

The game management unit 212 performs various processes related to game execution, such as controlling the game progress.

The movement control pattern setting unit 220 sets the movement control pattern of the commentator character 7 based on the user's operational input.

The voice acquisition control unit 222 controls the acquisition of the user's voice while playing the game.

The analysis unit 224 analyzes the gameplay situation. Specifically, the analysis unit 224 performs image analysis on the gameplay video 10, and analyzes the gameplay situation based on the image analysis results. The analysis unit 224 can also analyze operation input information of the gameplay, and analyze the gameplay situation based on the analysis results.

The layout setting unit 226 sets the layout of the gameplay video 10 and the commentary character 7 in the commentary video 12. The "layout" here refers to the setting of how the gameplay video 10 and the commentary character 7 are displayed, such as the relative position and relative size of the gameplay video 10 and the commentary character 7 within the screen display of the commentary video 12. Of course, the elements displayed within the game screen of the commentary video 12 can include display elements other than the gameplay video 10 and the commentary character 7, and in that case the settings for those display elements can also be included in the layout settings.

The commentary video generation control unit 228 controls the generation of commentary video 12, which is a combination of the video of the commentary character 7 providing commentary on the gameplay video 10 of the game play and the gameplay video 10. The commentary video generation control unit 228 functions as a commentary video generating means together with an image generating unit 292s, which will be described later. Specifically, the commentary video generation control unit 228 generates commentary video 12 by treating the video of the gameplay being performed by the user as the gameplay video 10 to be commented on, and using the user's voice acquired by the voice acquisition control unit 222 as commentary voice.

In addition, the commentary video generation control unit 228 can generate a commentary video by changing the playback speed of the video of the commentary character 7 and/or the gameplay video 10 based on the analysis results of the analysis unit 224.

The commentary character control unit 230 controls the commentary character 7 based on the analysis results of the analysis unit 224. Specifically, it controls the movement of the commentary character 7, controls changes to the appearance, and controls the display of effects that are displayed in conjunction with the movement control of the commentary character 7. The commentary character control unit 230 also has a voice response control unit 232 and a movement control pattern selection control unit 234.

When the play situation satisfies a voice reception situation condition indicating that the voice response control is acceptable, the voice response control unit 232 performs voice response control to control the behavior of the commentator character 7 based on the voice acquired by the voice acquisition control unit 222.

The movement control pattern selection control unit 234 presents the user with selection options for the movement control pattern of the commentator character 7, and controls the movement of the commentator character 7 based on the selected selection option for the movement control pattern.

The distribution control unit 240 controls the distribution of the live video 12.

The timing unit 280s uses the system clock to measure the current date and time, time limit, etc.

The sound generation unit 290s is realized by executing an IC or software that generates or decodes audio data, and generates or decodes audio data such as operation sounds and background music related to the system management and video distribution of the server system 1100. Then, audio signals related to system management are output to the sound output unit 390s.

The sound output unit 390s emits an audio signal. In the example of FIG. 1, this corresponds to a speaker (not shown) provided on the main device or touch panel 1108.

The image generation unit 292s generates images, synthesizes images, and outputs image signals to display them on the image display unit 392s. In this embodiment, it is responsible for part of the function of generating images related to system management of the server system 1100, game images (or data for displaying game images on the user terminal 1500), play-by-play videos 12, etc.

The communication control unit 294s executes data processing related to data communication and realizes data exchange with external devices via the communication unit 394s.

The communication unit 394s connects to the network 9 to realize communication. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc. In the example of FIG. 1, it corresponds to the communication device 1153.

The server storage unit 500s stores programs and various data for implementing various functions for the server processing unit 200s to comprehensively control the server system 1100. It is also used as a working area for the server processing unit 200s, and temporarily stores the results of calculations executed by the server processing unit 200s according to various programs. This function is implemented, for example, by IC memory such as RAM or ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs or DVDs, online storage, etc. In the example of Figure 1, this corresponds to storage media such as IC memory 1152 and hard disks mounted on the main unit, and storage 1140.

FIG. 10 is a diagram showing examples of programs and data stored in the server storage unit 500s in this embodiment. The server storage unit 500s in this embodiment stores a server program 501, a distribution client program 503, game initial setting data 510, image analysis dictionary data 512, dictionary data for progress control parameter values 514, dictionary data for operation input information 516, voice recognition dictionary data 518, motion control pattern definition data 520, and playback speed change pattern definition data 540.

The server storage unit 500s also stores game management data 550, commentary management data 570, user management data 600, playthrough video registration data 700, commentary video registration data 730, and current date and time 800 as data that is generated and managed sequentially. The server storage unit 500s can also store other programs and data (e.g. timers, counters, various flags, etc.) as appropriate.

The server program 501 is a program that is read and executed by the server processing unit 200s to realize the functions of the user management unit 210 to the distribution control unit 240. In this embodiment, the server system 1100 is composed of three hardware elements: a game server 1101, a live video production server 1102, and a distribution server 1103, and therefore stores server programs for realizing the functions assumed by each of these elements.

The distribution client program 503 is an original client program provided to the user terminal 1500. In this embodiment, one distribution client program 503 includes a distribution game client program 503a for executing an online game on the user terminal 1500, and a distribution commentary client program 503b for causing the user terminal 1500 to function as a terminal for generating commentary actions. These may be distributed separately to the user terminal 1500.

The game initial setting data 510 includes various initial setting data required to run an online game. In the example of FIG. 10, only one game initial setting data 510 is illustrated, but if the server system 1100 provides multiple game titles, the game initial setting data 510 is stored for each game title.

The image analysis dictionary data 512, the dictionary data for progress control parameter values 514, the dictionary data for operation input information 516, and the voice recognition dictionary data 518 are dictionary data that the analysis unit 224 refers to in order to analyze the play situation (see FIG. 4).

Movement control pattern definition data 520 is prepared for each movement control pattern and stores various data that defines the pattern. For example, as shown in FIG. 11, one piece of movement control pattern definition data 520 includes a unique movement type 521, application requirements 522, motion data 523, effect data 524, appearance change data 525, configurator account 526, and permitted range of use 527 (see FIG. 5).

The application requirement 522 is described by combining one condition or multiple conditions with AND or OR.
1) A character type condition 522a, which is a condition related to the type of the commentary character 7;
2) Game title condition 522b, which is a condition related to the game title being played in the gameplay video 10;
3) Image analysis result conditions 522c, which are conditions regarding analysis results based on image analysis;
4) progress control conditions 522d, which are conditions regarding the analysis results based on the progress control parameter values;
5) Operation input conditions 522e, which are conditions for the analysis results based on operation input information;
6) A speech recognition result condition 522f, which is a condition regarding an analysis result based on speech recognition;
Of course, other conditions may also be included as appropriate.

The configurator account 526 indicates the person who configured the definition data. Basically, this is a value indicating the manufacturer or administrator of the server system 1100, but for settings made by a user, the user's user account is set.

The permitted range of use 527 indicates the range within which the definition data is permitted to be used. For example, if it is set to unlimited, anyone can use the definition data at any time. If a specific user account is set, the users who can use it are limited. The setting of the permitted range of use 527 is included as one of the setting items in the user settings of the operation control pattern.

Returning to FIG. 10, the playback speed change pattern definition data 540 is prepared for each pattern of changing the playback speed of the gameplay video 10 displayed in the live video 12, and stores various related data. As shown in FIG. 12, for example, one playback speed change pattern definition data 540 includes a unique playback speed change pattern ID 541, application requirements 542, application playback speed 543, and application end conditions 544. Of course, other data may also be included as appropriate.

The application requirements 542 indicate the conditions that must be met for the definition data to be applied. Similar to the application requirements 522 of the motion control pattern definition data 520, the application requirements 542 are described by combining a single condition or multiple conditions with AND or OR.

The application end condition 544 specifies the condition for ending the change in playback speed and returning to the standard speed. For example, it is possible to set the elapsed time from when the definition data is applied and the playback speed change begins, the detection of a specified operation input, the detection of a specified keyword by voice recognition, etc.

Returning to FIG. 10, game management data 550 is prepared for each game played on the user terminal 1500, and stores various data for managing the execution of the game. One piece of game management data 550 includes, for example, as shown in FIG. 13, a unique game management ID 551, a player account 552, terminal access information 553, a latest progress control parameter value 554, and play information 560. Of course, other data may also be included as appropriate.

The gameplay information 560 includes gameplay video data 561, progress control parameter value history data 562 that stores the latest progress control parameter values 554 at each time in chronological order, operation input history data 563 that stores operation input information in chronological order, and user voice recording data 564.

Returning to FIG. 10, commentary management data 570 is prepared for each generation of commentary video 12, and stores various data related to the generation. As shown in FIG. 14, for example, one commentary management data 570 includes commentary target ID 571 indicating the target of commentary, applied layout type 572 indicating the current screen layout of commentary video 12, commentary character control data 573, commentary video data 574, and analysis result history data 580. Of course, data other than these can also be included as appropriate.

The commentary target ID 571 is set to the game management ID 551 (see FIG. 13) of the gameplay to be commented on or the gameplay video ID 701 (see FIG. 16) that is uniquely set to an existing gameplay video 10.

The applied layout type 572 is set to a predetermined initial setting when the live broadcast starts, but is changed in response to input of a predetermined layout change operation.

The commentary character control data 573 stores various data necessary for controlling the movements and display of the commentary character 7. For example, it stores control data for motion data, control data for performance display, control data for textures and color tables for the exterior, position information, posture information, various flags and counters, etc.

Analysis result history data 580 stores the analysis results of the play situation in chronological order. In this embodiment, four different methods are used to analyze the play situation, and the corresponding data includes image analysis result history data 581, analysis result history data based on progress control parameter values 582, analysis result history data based on operation input information 583, and voice recognition result history data 584.

Returning to FIG. 10, user management data 600 is prepared for each registered user of the live video generation and distribution system 1000, and stores various data related to that user. As shown in FIG. 15, for example, one user management data 600 stores a user account 601, terminal access information 602, live character setting data 603, character type 604 indicating the type of live character 7, and self-set action type list 605. Of course, data other than these may also be stored as appropriate.

The commentary character setting data 603 is created for each commentary character 7 used by the user, and stores data such as the character model, standard texture data, a list of added exterior decoration items, a color table, etc.

The self-configured operation type list 605 is a list of operation types 521 (see FIG. 11) of the operation control pattern definition data 520 that the user has configured using the user setting function of the operation control pattern.

Returning to FIG. 10, gameplay video registration data 700 is prepared for each existing gameplay video 10. As shown in FIG. 16, for example, one gameplay video registration data 700 stores a unique gameplay video ID 701, a providing user account 702 indicating the poster/provider of the gameplay video, and gameplay information 710 of the gameplay when the gameplay video was recorded. Of course, data other than these may also be stored as appropriate.

The play information 710 is a copy of the play information 560 (see FIG. 13) at the time of registration, and includes play video data 711, progress control parameter value history data 712, operation input history data 713, and user voice recording data 714.

Returning to FIG. 10, the live video registration data 730 is, so to speak, live video registration data that is created for each live video 12 that is generated. The registered live video 12 becomes the subject of distribution at any time in response to requests from the viewer 3. For example, as shown in FIG. 17, one live video registration data 730 stores a unique live video ID 731, a providing user account 732, and live video data 733. Of course, data other than these may also be stored as appropriate.

FIG. 18 is a functional block diagram showing an example of the functional configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes an operation input unit 100, a voice input unit 102, a device processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, and a device storage unit 500.

The operation input unit 100 outputs operation input signals corresponding to various operation inputs made by the player to the device processing unit 200. For example, this can be realized by a push switch, joystick, touch pad, track ball, acceleration sensor, gyro, CCD module, etc. Examples of this include the directional input key 1502, button switch 1504, touch panel 1506, and image sensor unit 1520 in FIG. 2.

The audio input unit 102 collects audio generated by the user (user audio) and environmental sounds, and outputs audio signals to the device processing unit 200. In the example of FIG. 2, this corresponds to the microphone 1210.

The device processing unit 200 is realized by electronic components such as a microprocessor such as a CPU or GPU, or an IC memory, and controls data input/output between each functional unit including the operation input unit 100 and the device storage unit 500. It controls the operation of the user terminal 1500 by executing various calculation processes based on predetermined programs and data, operation input signals from the operation input unit 100, and various data received from the server system 1100. This corresponds to the control board 1550 in Figure 2. The device processing unit 200 in this embodiment includes a user terminal calculation unit 260, a timer unit 280, a sound generation unit 290, and a communication control unit 294.

The user terminal calculation unit 260 includes an operation signal transmission control unit 261 and an image display control unit 262.

The operation signal transmission control unit 261 executes processing for transmitting various data and requests to the server system 1100 in response to operations performed on the operation input unit 100. In this embodiment, it also generates voice data of the user's voice collected by the voice input unit 102 and controls external transmission.

The image display control unit 262 performs control for displaying game screens and the like based on various data received from the server system 1100. In this embodiment, the server system 1100 is configured to generate game space images (images of the game space captured by a virtual camera; for example, 3DCG images, etc.), but it is also possible to generate game space images in the user terminal 1500. In that case, the image display control unit 262 includes control of objects arranged in a virtual three-dimensional space for generating the 3DCG, for example.

In addition, in relation to the generation of the commentary video 12, in this embodiment, the server system 1100 generates a video of the commentary character 7 and synthesizes it with the gameplay video 10 to generate the commentary video 12, but if the generation and synthesis processes are performed on the user terminal 1500 side, the image display control unit 262 will be responsible for some or all of these.

Then, the image display control unit 262 outputs the image signal to the image display unit 392 .
The image display unit 392 displays various game images and various screens related to the generation of the play-by-play video 12 based on the image signal input from the image display control unit 262. For example, the image display unit 392 can be realized by an image display device such as a flat panel display, a projector, or a head-mounted display. In this embodiment, the touch panel 1506 in FIG. 2 corresponds to this.

The sound generation unit 290 is realized by, for example, a digital signal processor (DSP), a processor such as a voice synthesis IC, an audio codec capable of playing audio files, etc., and generates sound signals for sound effects and background music related to the game, various operation sounds, and operation sounds and sound effects related to the generation of the play-by-play video 12, and outputs these to the sound output unit 390.

The sound output unit 390 is realized by a device that outputs sound effects, background music, etc. based on the sound signal input from the sound generation unit 290. This corresponds to the speaker 1510 in FIG. 2.

The communication control unit 294 executes data processing related to data communication, and realizes data exchange with external devices via the communication unit 394. The communication unit 394 realizes communication by connecting to the network 9. For example, this is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack and a control circuit of a wired communication cable, etc., and corresponds to the wireless communication module 1553 in FIG. 2.

The terminal storage unit 500 stores various data and programs for implementing various functions for the terminal processing unit 200 to comprehensively control the user terminal 1500. It is also used as a working area for the terminal processing unit 200, and temporarily stores the results of calculations executed by the terminal processing unit 200 according to various programs and input data input from the operation input unit 100. These functions are implemented, for example, by IC memory such as RAM or ROM, magnetic disks such as hard disks, and optical disks such as CD-ROMs and DVDs. This corresponds to the IC memory 1552 and memory card 1540 mounted on the control board 1550 in Figure 2. A configuration using online storage is also possible.

In this embodiment, the terminal storage unit 500 stores a client program 502. Of course, other programs and data can also be stored as appropriate.

The client program 502 is application software that the device processing unit 200 reads and executes to realize the functions of the user terminal computing unit 260. In this embodiment, the client program 502 is a copy of the distribution client program 503 (see FIG. 10) provided by the server system 1100. The client program 502 may be a dedicated client program, or may be configured with a web browser program and a plug-in that realizes interactive image display.

Next, the operation of the live video creation and distribution system 1000 of this embodiment will be described.
19 and 20 are flowcharts for explaining the flow of processing in the server system 1100. The flow of processing explained here is realized by the server processing unit 200s of the server system 1100 executing the server program 501. It is assumed that the client program 502 is being executed in the user terminal 1500, the game can be played, and a function related to the generation of an execution video is available.

As shown in FIG. 19, the server system 1100 first accepts user settings for the operation control pattern (step S8; see FIG. 7).

Next, the server system 1100 creates and initializes live commentary management data 570 (step S10; see FIG. 14). The live commentary target ID 571 is set to a predetermined value meaning live distribution, and the applied layout type 572 is set to a predetermined initial value. Live commentary character control data 573, live commentary video data 574, and analysis result history data 580 are not stored at this stage.

Next, the server system 1100 starts acquiring user voice data collected by the user terminal 1500 (step S12). Specifically, the live video production server 1102 starts generating and transmitting to the user terminal 1500 voice data of the user voice collected by the microphone 1512. The user terminal 1500 starts generating the user's voice data and starts controlling the sequential transmission to the live video production server 1102.

Next, the server system 1100 starts controlling the commentary character 7 (step S14), starts generating and recording the commentary video 12, and if the commentary video 12 is to be distributed in a live format, starts the distribution here (step S16).

At step S14, no motion control is performed based on the motion control pattern definition data 520, so the commentary character 7 is controlled to take a paused motion in a standby state. Then, from step S14 onwards, the commentary character control data 573 in the commentary management data 570 is updated one by one, and from step S16 onwards, the commentary video data 574 is also updated one by one. At this point, game play has not started, so the gameplay video 10, which is a live video, is not yet displayed in the commentary video 12. In terms of the flow of the commentary video 12, this is the time period in which user 2 (2a), who is the commentary user, greets the viewers and gives an overview of the commentary video 12 that is about to begin.

Next, the server system 1100 starts layout change control when it detects an input of a layout change operation for the live video 12 on the user terminal 1500 (step S18). After that, the user 2 (2a) can change the screen layout as desired even during the live video.

Next, when the server system 1100 detects input of a predetermined game start operation on the user terminal 1500, it performs preparation control for game play (step S30). Specifically, the game server 1101 creates and initializes game management data 550 (see FIG. 13). It accepts the selection and setting of the player character 4 and the selection of the play stage, constructs a game space in a virtual three-dimensional space, and arranges the player character 4 and enemy character 6 objects. It also initializes the latest progress control parameter values 554.

Also, at this stage, the server system 1100 sets the gameplay video 10 as the commentary target. That is, the game server 1101 provides the game management ID 551 to the commentary video generation server 1102, and the game management ID 551 is set as the commentary target ID 571 (see FIG. 14).

After completing the preparations, the server system 1100 starts controlling the game progress and recording the play information 560 (step S32), and starts analyzing the play situation (step S34). Specifically, the game server 1101 starts transmitting the play information 560 to the live video production server 1102. The live video production server 1102 starts receiving the play information 560. The live video production server 1102 then starts sequentially analyzing the play situation based on the received play information 560, and accumulates the analysis results in the analysis result history data 580 (see FIG. 14).

In the case of distribution of a commentary video 12 that is not in a live format, for example, if the commentary subject is an existing gameplay video 10, step S30 is to accept the selection setting of the existing gameplay video 10 to be commented on, and step S32 is to start playing the existing gameplay video 10 and start referring to the gameplay information.

If the analysis results of the play situation, which are determined one by one, satisfy the application requirements 542 of any of the playback speed change pattern definition data 540 (see FIG. 12) (YES in step S40), the server system 1100 starts changing the playback speed in accordance with that definition data (step S42). The started playback speed change is continued until the application end condition 544 of that definition data is satisfied.

Moving on to FIG. 20, if the analysis results of the most recent play situation at that time satisfy the voice reception situation conditions (YES in step S50) and there is movement control pattern definition data 520 (see FIG. 11) for which the application requirements 522 are satisfied (YES in step S52), the server system 1100 sets the movement control pattern of the corresponding definition data as a selection candidate (step S54).

If there are multiple selection candidates (multiple in step S56), the server system 1100 displays selection operation icons 33 for each selection candidate on the user terminal 1500 and presents them to the user (step S58; see FIG. 6). Then, the server system 1100 starts controlling the movement of the commentator character 7 with the movement control pattern of the movement control pattern definition data 520 of the selection candidate selected by the user (step S60).

If there is only one selected candidate (single in step S56), the server system 1100 starts controlling the movement of the commentator character 7 using the movement control pattern of the movement control pattern definition data 520 that is the selected candidate (step S62).

Then, when the motion control starts, the server system 1100 starts changing the appearance associated with the motion control pattern (step S64) and starts applying the effect (step S66).

When the reproduction of the movement control pattern that has been applied is completed, the server system 1100 controls the commentator character 7 to assume a pause in a standby state. In addition, when the reproduction of the movement control pattern is completed, the appearance change is completed and the appearance is returned to the standard appearance, and the application of the effect is also completed.

The server system 1100 repeats steps S40 to S66 until a predetermined operation to end the live commentary is detected on the user terminal 1500 (NO in step S70). If an operation to end the live commentary is detected (YES in step S70), the server system 1100 performs a new registration process for the current live commentary video 12, and if the live commentary video 12 is being distributed in a live format, ends the distribution here (step S74), and ends the series of processes.

As described above, according to this embodiment, a new user assistance technology for video distribution using a commentary character can be provided. In other words, the play situation of the gameplay displayed as a gameplay video can be automatically analyzed, and the commentary character can be automatically controlled based on the analysis results. Therefore, the user does not need to create the commentary character's movements one by one and prepare a video in which the commentary character is controlled. In addition, since the need for user operations related to the movement control of the commentary character can be significantly reduced, the user can easily generate a live-style commentary video using the commentary character 7 by simply inputting the sound of the commentary or explanation while playing the game.

Second Embodiment
Next, a second embodiment to which the present invention is applied will be described. In the following, differences from the first embodiment will be mainly described, and the same components as those in the first embodiment will be given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted.

This embodiment differs from the first embodiment in that, among the processes performed by the server system 1100 in the first embodiment, the processes related to game management and the generation of live video, except for distribution, are performed by the user terminal 1500.

Figure 21 is a functional block diagram showing an example of the functional configuration of a user terminal 1500B in this embodiment. Compared to the user terminal 1500 of the first embodiment, the user terminal 1500B has a game management unit 212B equivalent to the game management unit 212 of the first embodiment instead of the user terminal calculation unit 260, and executes various processes for controlling the execution of the game in the user terminal 1500B. The user terminal 1500B also has a motion control pattern setting unit 220, a voice acquisition control unit 222, an analysis unit 224, a layout setting unit 226, a live video generation control unit 228, a live character control unit 230, and a distribution control unit 240B.

The distribution control unit 240B controls the distribution of the live video 12 by uploading the generated live video 12 data to a predetermined distribution system (in this embodiment, the distribution server 1103).

Figure 22 is a diagram showing an example of programs and data stored in the terminal storage unit 500 in this embodiment. The terminal storage unit 500 in this embodiment stores a terminal program 504. The terminal program 504 is a program for causing the terminal processing unit 200 to realize the functions of the game management unit 212B, the action control pattern setting unit 220, the voice acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video generation control unit 228, and the live character control unit 230. The original of the terminal program 504 may be stored in the server system 1100 as a distribution terminal program and downloaded as necessary.

The terminal storage unit 500 of this embodiment also stores game initial setting data 510, image analysis dictionary data 512, dictionary data for progress control parameter values 514, dictionary data for operation input information 516, voice recognition dictionary data 518, motion control pattern definition data 520, playback speed change pattern definition data 540, game management data 550, and commentary management data 570. The originals of these various data may be stored in the server system 1100 and downloaded as necessary.

The process flow executed by the user terminal 1500B in this embodiment is basically the same as that executed by the server system 1100 in the first embodiment (see Figures 19 to 20). In accordance with the fact that the processes related to game management and play-by-play video generation are executed by the user terminal 1500B, the execution subject of each step may be appropriately replaced with the user terminal 1500B.

When the user terminal 1500B executes a process that requires referencing or changing the user management data 600, it requests the server system 1100 to provide or verify the necessary data as appropriate. Alternatively, it requests the process itself and handles it.

[Modifications]
Although the embodiment to which the present invention is applied has been described above, the forms to which the present invention can be applied are not limited to the above-described forms, and constituent elements can be added, omitted, or modified as appropriate.

(Variation 1)
In the first embodiment, an example is given in which processing related to game management and generation of commentary video is executed by server system 1100, and in the second embodiment, an example is given in which processing related to game management and generation of commentary video is executed by user terminal 1500B, but the entity executing processing related to game management and generation of commentary video is not limited to the above embodiments.

For example, it is possible to configure the server system 1100 to handle the processes related to game management, and the user terminal 1500 to handle the processes related to the generation of live video. In other words, it is possible to appropriately set which hardware will execute the game management unit 212, the action control pattern setting unit 220, the audio acquisition control unit 222, the analysis unit 224, the layout setting unit 226, the live video generation control unit 228, and the live character control unit 230.

(Variation 2)
In addition, in the above embodiment, with regard to the analysis of the play situation, a movement control pattern to be applied to the commentator character 7 is selected based on all of the analysis results of each of the four analysis methods, but it is also possible to configure the system to select four analysis results and use them to select a movement control pattern.

For example, the live video production server 1102 may determine the final analysis result of the playing situation at each moment based on the analysis results of each of these four methods. Specifically, when an analysis result is output using one of the methods, that analysis result may be set as the final analysis result, and when multiple methods have output analysis results, the analysis result using the method with the highest adoption priority, which is preset for each method, may be set as the final analysis result. Of course, the final analysis result may be selected by majority vote.

(Variation 3)
Moreover, an element for controlling a commentary character based on comments input by the player 2a during play can be added to the above embodiment. Specifically, for example, if the first embodiment is used as a base, as shown in Fig. 23, a comment acquisition control unit 229 that performs control for acquiring comments from the player, user 2 (2a), is added, and a comment response control unit 236 that performs comment response control for controlling the behavior of the commentary character 7 based on the comments acquired by the comment acquisition control unit 222 is added to the commentary character control unit 230.

If the operation input unit 110 includes a keyboard, the comment acquisition control unit 229 acquires a comment by text input from the keyboard. Alternatively, a text input unit (such as a software keyboard) may be displayed within the game screen to acquire a comment. Alternatively, the voice acquired by the voice acquisition control unit 222 may be used as the voice comment as is, or text obtained by performing voice recognition processing may be used as the text comment.

The comment response control unit 236 may also use the fourth analysis method by the analysis unit 224. That is, the comment response control unit 236 may compare the voice of the voice comment acquired by the voice acquisition control unit 222 (user voice) with the voice recognition dictionary data 518, and use the play situation that is pre-associated with the matching dictionary data as the analysis result in this method, and control the movement of the commentary character 7 based on this analysis result, thereby achieving comment response control. When acquired as a text comment, the comment response control may be achieved by regarding the acquired text as words obtained as a result of voice recognition, and selecting a movement control pattern based on the voice recognition result condition 522f (see FIG. 11) to control the movement of the commentary character 7.

Furthermore, a comment acceptance status condition indicating that the play situation is one in which comment response control is acceptable may be set by reusing the voice acceptance status condition in the fourth method for analyzing the play situation, and when the comment acceptance status condition is satisfied, the action of the commentator character may be controlled based on the comment acquired by the comment acquisition control unit 229.

Of course, even if the second embodiment is used as a base, an element can be added in which the commentary character is controlled based on comments entered by the player 2a while playing.

2...User 7...Commentary character 10...Play video 12...Commentary video 30...Selection candidate 100...Operation input unit 102...Audio input unit 200s...Server processing unit 212...Game management unit 220...Movement control pattern setting unit 222...Audio acquisition control unit 224...Analysis unit 226...Layout setting unit 228...Movement control video generation control unit 229...Comment acquisition control unit 230...Commentary character control unit 232...Audio response control unit 234...Movement control pattern selection control unit 236...Comment response control unit 240...Distribution control unit 500s...Server storage unit 501...Server program 502...Client program 510...Game initial setting data 512...Image analysis dictionary data 514...Dictionary data for progress control parameter values 516...Dictionary data for operation input information 518...Voice recognition dictionary data 520...Movement control pattern definition data 522...Application requirements 523: Motion data 525: Appearance change data 540: Playback speed change pattern definition data 550: Game management data 560: Play information 561: Play video data 562: Progress control parameter value history data 563: Operation input history data 564: User voice recording data 570: Live commentary management data 571: Live commentary target ID
572: Applied layout type 573: Live commentary character control data 574: Live commentary video data 580: Analysis result history data 581: Image analysis result history data 582: Progress control parameter value reference analysis result history data 583: Operation input information reference analysis result history data 584: Voice recognition result history data 603: Live commentary character setting data 1000: Live commentary video generation and distribution system 1100: Server system 1101: Game server 1102: Live commentary video generation server 1103: Distribution server 1500: User terminal W6: Game screen W7: Action control pattern setting screen W8: Motion editing screen

Claims (8)

A computer system
a commentary video generating means for generating a commentary video by combining a video of a commentary character who provides commentary on the game play video of the game, and the game play video;
a layout setting means for setting a layout of the gameplay video and the commentary character in the commentary video;
An analysis means for analyzing a playing situation of the game play;
commentary character control means for controlling the commentary character to automatically perform a preset action in correspondence with a play situation predicted based on an analysis result of the analysis means;
A program to function as a A computer system
a commentary video generating means for generating a commentary video by combining a video of a commentary character that provides commentary on a gameplay video of a game in which a commentary user and an opponent user compete against each other by operating each other's characters, and the gameplay video;
an analysis means for analyzing a playing situation of the game by analyzing operation input information of the opponent user;
a commentary character control means for controlling the commentary character based on an analysis result of the analysis means;
A program to function as a A distribution control means for controlling the distribution of the live video or controlling the distribution of the live video by uploading data of the live video to a predetermined distribution system;
3. The program according to claim 1 or 2, for causing the computer system to function as a a layout setting means for setting a layout of the gameplay video and the commentary character in the commentary video;
3. The program according to claim 2 , for causing the computer system to function as follows: The commentary character control means changes and controls the appearance of the commentary character.
The program according to any one of claims 1 to 4. the play-by-play video generation means changes a playback speed of the play-by-play video based on the analysis result of the analysis means to generate the play-by-play video;
The program according to any one of claims 1 to 5. a commentary video generating means for generating a commentary video by combining a video of a commentary character who comments on a game play video of a game, and the game play video;
a layout setting means for setting a layout of the play-by-play video and the play-by-play character in the play-by-play video;
An analysis means for analyzing a playing situation of the game play;
commentary character control means for controlling the commentary character to automatically perform a preset action corresponding to a play situation predicted based on an analysis result of the analysis means;
A computer system comprising: a commentary video generating means for generating a commentary video by combining a video of a commentary character that provides commentary on a gameplay video of a game in which a commentary user and an opponent user compete against each other by operating each other's characters, and the gameplay video;
an analysis means for analyzing a playing situation of the game by analyzing operation input information of the opponent user;
a commentary character control means for controlling the commentary character based on an analysis result of the analysis means;
A computer system comprising:

Images