JP7460059B2 - A video distribution system for live streaming videos including animations of character objects generated based on the movements of actors - Google Patents

Description

The disclosure herein relates to a video distribution system that live-distributes a video including animation of a character object generated based on the movement of an actor.

2. Description of the Related Art A video distribution system is known that generates animation of a character object based on the movement of an actor and live-distributes a video including the animation of the character object. Such a video distribution system is disclosed in, for example, Japanese Patent Application Publication No. 2015-184689 (Patent Document 1).

JP 2015-184689 A

During live streaming of videos, unforeseen circumstances may occur, such as malfunctions in the equipment used to generate animations. In the conventional video distribution system described above, live distribution is performed only by actors and camera operators. When an actor or camera operator deals with an unexpected situation, there is a risk that the movement of the actor or camera operator will be reflected in the movement of the character object, or that the camera work will be inappropriate. As a result, there is a possibility that an undesirable effect will appear on the quality of the distributed video.

The objective of the present disclosure is to provide a technical improvement that solves or alleviates at least some of the problems of the conventional technology described above. One of the more specific objectives of the present disclosure is to provide a video distribution system that makes it easier to deal with unforeseen circumstances that occur during live distribution.

A video distribution system according to one embodiment includes a distribution server device configured to live-distribute a video including an animation of a character object generated based on the movement of an actor to a first display device and a first client device used by a first user, and a supporter computer configured to display first additional information on the video displayed on the first display device while preventing the first additional information from being displayed on the first client device. The first display device may be used by the actor. The first display device may display an object selectable by the actor.

In one embodiment, the supporter computer is configured to display second additional information on the video displayed on the first display device and on the video displayed on the first client device based on a second operation input.

In one embodiment, the supporter computer is configured to generate a modified character object by modifying at least a portion of the character object based on a third operation input, and to include the modified character object in the video.

In one aspect, the character object has a face portion that is generated to move based on face motion data indicating the movement of the actor's face, and the distribution server device is configured to display third additional information on the face portion based on the third operation input.

In one aspect, the supporter computer further includes a storage for storing a decoration object displayed in association with the character object, and the supporter computer is configured to store the decoration object when displaying the decoration object in the moving image based on a fourth operation input. The object is configured to display a blind object for at least a portion of the object to be hidden.

In one aspect, the distribution server device is configured to live distribute the video to a second client device used by a second user in addition to the first client device, and the first client device is of a first type. , the second client device is a second type device different from the first type device, and the supporter computer displays a display image of the moving image displayed on the first client device, and A second display device is provided that displays a display image of the moving image displayed on the second client device.

In one embodiment, the supporter computer is configured to prohibit distribution of the video to the first user based on a fifth operation input.

One aspect relates to a video distribution method performed by one or more computer processors executing computer-readable instructions. The video distribution method includes the steps of live-distributing a video including animation of a character object generated based on the movement of an actor to a first display device and a first client device used by a first user; displaying first additional information on the moving image being displayed on the screen, while not displaying the first additional information on the first client device.

A distribution program according to one embodiment causes one or more computer processors to live-distribute a video including animation of a character object generated based on movements of an actor to a first display device and a first client device used by a first user. and a step of displaying first additional information on the video displayed on the first display device while not displaying the first additional information on the first client device.

According to the embodiments of the present invention, it is possible to provide a video distribution system that facilitates dealing with unexpected situations that occur during live distribution.

FIG. 1 is a block diagram illustrating a video distribution system according to an embodiment. FIG. 2 is a schematic diagram schematically showing equipment of a studio that produces videos distributed by the video distribution system of FIG. 1. FIG. 2 is a diagram illustrating a possession list stored in the video distribution system of FIG. 1. 2 is a diagram illustrating a candidate list stored in the video distribution system of FIG. 1. FIG. FIG. 3 is a diagram illustrating an example of a moving image displayed on the client device 10a in one embodiment. FIG. 5 includes an animation of a character object. FIG. 3 is a diagram illustrating an example of a moving image displayed on the client device 10a in one embodiment. FIG. 6 includes normal objects. FIG. 3 is a diagram illustrating an example of a moving image displayed on the client device 10a in one embodiment. FIG. 7 includes decorative objects. FIG. 3 is a schematic diagram showing an example of a decorative object selection screen for selecting a desired selected decorative object from among decorative objects included in a candidate list. FIG. 13 is a diagram showing an example of an image displayed on the display 44 of the supporter computer 40 in one embodiment. 10 is a diagram showing an example of a moving image displayed on the display 39 in one embodiment. FIG 10 shows an example of the first additional information. FIG. 3 is a diagram illustrating an example of a moving image displayed on the client device 10a in one embodiment. FIG. 11a includes second additional information. 11A and 11B are diagrams illustrating an example of a video displayed on a client device 10a in an embodiment, in which second additional information is included. 12A and 12B are diagrams illustrating an example of a video displayed on a client device 10a in an embodiment, in which third additional information is included. FIG. 3 is a diagram illustrating an example of a moving image displayed on the client device 10a in one embodiment. FIG. 12b includes third additional information. 13 is a diagram showing an example of a video displayed on a client device 10a in one embodiment, in which a blind object is included. It is a flow diagram showing the flow of video distribution processing in one embodiment. FIG. 3 is a flow diagram showing the flow of processing for displaying first additional information in one embodiment. It is a flowchart which shows the flow of the process which displays 2nd additional information in one embodiment. It is a flowchart which shows the flow of the process which displays 3rd additional information in one embodiment. FIG. 2 is a flow diagram illustrating a process for displaying blind objects in one embodiment. FIG. 2 is a flow diagram showing a process for prohibiting video distribution to a distribution-prohibited user in one embodiment.

Various embodiments of the present invention will now be described with reference to the drawings. The same reference numerals are used to designate the same or similar components in the drawings.

A video distribution system according to one embodiment will be described with reference to Figures 1 to 4. Figure 1 is a block diagram showing a video distribution system 1 according to one embodiment, Figure 2 is a schematic diagram showing the equipment of a studio where videos to be distributed by the video distribution system 1 are produced, and Figures 3 and 4 are diagrams for explaining information stored in the video distribution system 1.

The video distribution system 1 includes client devices 10a-10c, a server device 20, a studio unit 30, and a storage 60. The client devices 10a-10c, the server device 20, and the storage 60 are connected to each other via a network 50 so as to be able to communicate with each other. The server device 20 is configured to distribute videos including character animations, as described below. The characters included in the videos may be motion-controlled within a virtual space.

This moving image can be distributed from the server device 20 to each of the client devices 10a to 10c. A first viewing user who is the user of the client device 10a, a second viewing user who is the user of the client device 10b, and a third viewing user who is the user of the client device 10c view the distributed video using each client device. be able to. The video distribution system 1 may include fewer than three client devices, or may include more than three client devices. In this specification, client devices 10a to 10c and other client devices may be collectively referred to as "client devices" when there is no need to distinguish them from each other.

The client devices 10a to 10c are information processing devices such as smartphones. In addition to smartphones, the client devices 10a to 10c may be mobile phones, tablet terminals, personal computers, electronic book readers, wearable computers, game consoles, and various other information processing devices capable of playing back moving images. Each of the client devices 10a to 10c includes a computer processor, a memory, a communication I/F, a display, a sensor unit including various sensors such as a gyro sensor, a sound collection device such as a microphone, and a storage for storing various information. Good too.

The viewing user can input a message regarding the distributed video via the input interface of the client devices 10a to 10c, and post this message to the server device 20. Messages posted by each viewing user may be displayed superimposed on the video. This facilitates interaction between viewing users.

In the illustrated embodiment, the server device 20 includes a computer processor 21, a communication I/F 22, and a storage 23.

The computer processor 21 is an arithmetic device that loads an operating system and various programs that implement various functions into memory from the storage 23 or other storage, and executes instructions included in the loaded programs. The computer processor 21 is, for example, a CPU, MPU, DSP, GPU, various other arithmetic devices, or a combination thereof. The computer processor 21 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. Although computer processor 21 is illustrated as a single component in FIG. 1, computer processor 21 may be a collection of multiple physically separate computer processors. In this specification, a program described as being executed by the computer processor 21 or instructions included in the program may be executed by a single computer processor, or may be executed in a distributed manner by a plurality of computer processors. Good too. Further, the program executed by the computer processor 21 or the instructions included in the program may be executed by a plurality of virtual computer processors.

The communication I/F 22 is implemented as hardware, firmware, or communication software such as a TCP/IP driver or a PPP driver, or a combination of these. The server device 20 can send and receive data to and from other devices via the communication I/F 22.

Storage 23 is a storage device accessed by computer processor 21. Storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or any other storage device capable of storing data. Various programs may be stored in storage 23. At least a portion of the programs and various data that may be stored in storage 23 may be stored in storage (e.g., storage 60) that is physically separate from server device 20.

Many of the components of the studio unit 30 are arranged, for example, in studio room R shown in FIG. 2. As shown in the figure, actors A1 and A2 are performing in studio room R. The studio unit 30 is configured to detect the movements and facial expressions of actors A1 and A2 and output the detected information to the server device 20.

Both the actor A1 and the actor A2 are targets whose movements and facial expressions are captured by a group of sensors, which will be described later, provided in the studio unit 30. Actor A1 and actor A2 are, for example, humans who perform, animals, or moving objects. Actor A1 and actor A2 may be robots capable of independent movement, for example. The number of actors in studio room R may be one, or may be three or more.

The studio unit 30 has six motion sensors 31a to 31f attached to the actor A1, a controller 33a held in the left hand of the actor A1, a controller 33b held in the right hand of the actor A1, and a camera 37a attached to the head of the actor A1 via a mounting 37b. The studio unit 30 also has six motion sensors 32a to 32f attached to the actor A2, a controller 34a held in the left hand of the actor A2, a controller 34b held in the right hand of the actor A2, and a camera 38a attached to the head of the actor A2 via a mounting 38b. Each of the mountings 37b and 38b may be equipped with a microphone for acquiring voice data. The microphone can acquire the speech of the actors A1 and A2 as voice data. The microphones may be wearable microphones attached to actor A1 and actor A2 via mounting fixtures 37b and 38b, or may be stationary microphones installed on the floor, wall, or ceiling of studio room R. In addition to the above components, studio unit 30 has base station 35a, base station 35b, tracking sensor 36a, tracking sensor 36b, and display 39. Supporter computer 40 is installed in an adjacent room separated from studio room R by a glass window. Server device 20 may be installed in the same room as supporter computer 40.

The motion sensors 31a to 31f and the motion sensors 32a to 32f cooperate with the base stations 35a and 35b to detect their own positions and orientations. In one embodiment, the base stations 35a and 35b are multi-axis laser emitters. The base station 35a emits a flashing light for synchronization, and then scans the laser light, for example, around a vertical axis. The base station 35b scans the laser light, for example, around a horizontal axis. The motion sensors 31a to 31f and the motion sensors 32a to 32f may each include a plurality of optical sensors that detect the incidence of the flashing light and the laser light from the base stations 35a and 35b. The motion sensors 31a to 31f and the motion sensors 32a to 32f may each detect the time difference between the incidence timing of the flashing light and the incidence timing of the laser light, the light reception time at each optical sensor, the incidence angle of the laser light detected by each optical sensor, and other information as necessary. The motion sensors 31a to 31f and the motion sensors 32a to 32f may be, for example, Vive Trackers provided by HTC CORPORATION. The base station 35a and the base station 35b may be, for example, base stations provided by HTC CORPORATION. Three or more base stations may be provided. The positions of the base stations may be changed as appropriate. For example, in addition to or instead of the base station provided in the upper corner of the space to be detected by the tracking sensor shown in FIG. 2, a base station may be provided in pairs at an upper position and a lower position close to the floor surface.

The detection information (hereinafter, sometimes referred to as "tracking information") detected by each of the motion sensors 31a to 31f and the motion sensors 32a to 32f is transmitted to the server device 20. This detection information may be wirelessly transmitted from each of the motion sensors 31a to 31f and the motion sensors 32a to 32f to the server device 20. Since the base station 35a and the base station 35b emit blinking light and perform laser light scanning at regular intervals, the tracking information of each motion sensor is updated at each interval. Based on the tracking information detected by the motion sensors 31a to 31f and the motion sensors 32a to 32f, the position and orientation of each of the motion sensors 31a to 31f and the motion sensors 32a to 32f can be calculated. For example, the position and orientation of each of the motion sensors 31a to 31f and the motion sensors 32a to 32f may be calculated based on the tracking information in the server device 20.

In the illustrated embodiment, actor A1 is equipped with six motion sensors 31a-31f. The motion sensors 31a, 31b, 31c, 31d, 31e, and 31f are attached to the left wrist, right wrist, left instep, right instep, waist, and top of the head of actor A1, respectively. The motion sensors 31a to 31f may be attached to the actor A1 via attachment tools. Actor A2 is equipped with six motion sensors 32a to 32f. The motion sensors 32a to 32f may be attached to the actor A2 at the same positions as the motion sensors 31a to 31f. The motion sensors 31a-31f and motion sensors 32a-32f shown in FIG. 2 are examples. Motion sensors 31a to 31f can be attached to various parts of actor A1, and motion sensors 32a to 32f can be attached to various parts of actor A2. The number of motion sensors attached to actor A1 and actor A2 may be 5 or less, or may be 7 or more. In this way, the movements of the bodies of actors A1 and A2 are detected by detecting the positions and orientations of the motion sensors 31a to 31f and motion sensors 32a to 32f attached to each part of the bodies of actors A1 and A2. be able to.

In one embodiment, a plurality of motion sensors attached to actor A1 and actor A2 are each equipped with a large number of infrared LEDs, and light from the infrared LEDs is transmitted to a plurality of motion sensors installed on the floor and walls of studio room R. The position and orientation of each of the motion sensors may be detected by detecting with an infrared camera. The position and orientation of each of the motion sensors may be detected by using a visible light LED instead of the infrared LED and detecting the light from the visible light LED with a visible light camera. In this way, each of the plurality of motion sensors attached to the actor is provided with a light emitting part (for example, an infrared LED or a visible light LED), and the light from this light emitting part is transmitted to the light receiving part ( For example, the position and orientation of each of the motion sensors may be detected by detection using an infrared camera or a visible light camera.

In one embodiment, a plurality of reflective markers can be used instead of the motion sensors 31a-31f and the motion sensors 32a-32f. The reflective markers are attached to each of the actors A1 and A2 with adhesive tape or the like. The actors A1 and A2 to which the reflective markers are attached are photographed to generate photographic data, and the photographic data is subjected to image processing to detect the positions and orientations of the reflective markers.

The controller 33a and the controller 33b output control signals to the server device 20 according to the operation of the actor A1. Similarly, the controller 34a and the controller 34b output a control signal to the server device 20 according to the operation of the actor A2.

The tracking sensor 36a and the tracking sensor 36b generate tracking information for determining virtual camera setting information for constructing a virtual space included in a video. The tracking information of the tracking sensor 36a and the tracking sensor 36b is calculated as a position in each three-dimensional orthogonal coordinate system and an angle around each axis. The position and orientation of the tracking sensor 36a can be changed according to an operation by an operator. The tracking sensor 36a transmits tracking information indicating its position and orientation to the tracking information server device 20. Similarly, the position and orientation of the tracking sensor 36b are set according to the operator's operations. The tracking sensor 36b transmits tracking information indicating its position and orientation to the tracking information server device 20. Tracking sensors 36a, 36b may be supported by gimbals or stabilizers. The gimbal may be configured to have a shape that can be held by an actor or a supporter. By supporting the tracking sensors 36a and 36b with gimbals or stabilizers, it is possible to suppress camera shake.

As described above, camera 37a is attached to the head of actor A1. For example, camera 37a is positioned so as to capture an image of actor A1's face. Camera 37a continuously captures images of actor A1's face and acquires image data of actor A1's face. Similarly, camera 38a is attached to the head of actor A2. Camera 38a is positioned so as to capture images of actor A2's face and continuously captures images of actor A2's face and acquires image data of actor A2's face. Camera 37a transmits image data of actor A1's face to server device 20, and camera 38a transmits image data of actor A1's face to server device 20. Cameras 37a and 38a may be 3D cameras capable of detecting the depth of a person's face.

Display 39 is configured to display information received from supporter computer 40 . The information transmitted from the supporter computer 40 to the display 39 may include, for example, text information, image information, and various other information. The display 39 is arranged at a position where it can be viewed by the actors A1 and A2. Display 39 is an example of a first display device.

In the illustrated embodiment, the supporter computer 40 is installed in a room next to the studio room R. The room in which the supporter computer 40 is installed is separated from the studio room R by a glass window, so that the operator of the supporter computer 40 (sometimes referred to as the "supporter" in this specification) can see the actors A1 and A2. In the illustrated embodiment, supporter B1 and supporter B2 are present as operators of the supporter computer 40. In this specification, when there is no need to distinguish between supporter B1 and supporter B2, supporter B1 and supporter B2 may be collectively referred to as the "supporter".

The supporter computer 40 may be configured to be able to change the settings of the components of the studio unit 30 in accordance with the operations of the supporter B1 and the supporter B2. The supporter computer 40 can, for example, set the scanning interval by the base station 35a and the base station 35b, set the position or orientation of the tracking sensor 36a and the tracking sensor 36b, and change various settings of various devices other than these. . At least one of supporter B1 and supporter B2 can input a message into supporter computer 40 and display the input message on display 39.

The components and functions of studio unit 30 shown in FIG. 2 are exemplary. The studio unit 30 applicable to the present invention may include various components not shown. For example, the studio unit 30 may include a projector. The projector can project a moving image distributed to the client device 10a or other client devices onto the screen S.

Next, in one aspect, information stored in storage 23 will be described. In the illustrated embodiment, storage 23 stores model data 23a, object data 23b, possession list 23c, candidate list 23d, and various other information required for generating and distributing distribution videos.

The model data 23a is model data for generating animation of a character. The model data 23a may be three-dimensional model data for generating three-dimensional animation, or may be two-dimensional model data for generating two-dimensional model data. The model data 23a includes, for example, rig data (sometimes called "skeleton data") indicating the skeleton of the character, and surface data indicating the shape and texture of the surface of the character. The model data 23a may include a plurality of model data that are different from each other. The plurality of model data may have different rig data from each other, or may have the same rig data. The plurality of model data may have different surface data from each other, or may have the same surface data. In the illustrated embodiment, in order to generate a character object corresponding to actor A1 and a character object corresponding to actor A2, the model data 23a has at least two different types of model data. The model data for the character object corresponding to actor A1 and the model data for the character object corresponding to actor A2 may have, for example, the same rig data but different surface data.

The object data 23b includes asset data for constructing a virtual space that constitutes a moving image. The object data 23b includes data for drawing the background of the virtual space constituting the video, data for drawing various objects displayed in the video, and data for drawing various objects displayed in the video other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space.

In addition to the above, object data 23b may include gift objects that are displayed in the video based on a display request from a viewing user of client device 10a to 10c. Gift objects may include effect objects, normal objects, and decorative objects. Viewing users can purchase desired gift objects. An upper limit may be set on the number of purchasable objects or the purchase price of the objects.

An effect object is an object that affects the overall impression of the viewing screen of the distributed video, for example an object that resembles confetti. The confetti object may be displayed across the entire viewing screen, thereby changing the overall impression of the viewing screen before and after it is displayed. Effect objects may be displayed so as to overlap with character objects, but differ from decorative objects in that they are not displayed in association with any particular part of the character object.

A normal object is an object that is a digital gift from a viewing user to an actor (for example, actor A1 or actor A2), and is, for example, an object imitating a stuffed animal or a bouquet of flowers. In one aspect, the normal object is displayed on the video display screen so as not to come into contact with the character object. In one aspect, the normal object is displayed on the video display screen so as not to overlap with the character object. The normal object may be displayed so as to overlap objects other than the character object in the virtual space. Normal objects are sometimes displayed so as to overlap with character objects, but are different from decorative objects in that they are not displayed in association with a specific part of the character object. In one aspect, when displaying a normal object overlapping with a character object, the normal object overlaps with a part other than the head including the face of the character object, and does not overlap with the head of the character object. It will be displayed as follows.

A decorative object is an object displayed on a display screen in association with a specific part of a character object. In one embodiment, a decorative object displayed on the display screen in association with a specific part of the character object is displayed on the display screen so as to be in contact with the specific part of the character object. In one embodiment, a decorative object displayed on the display screen in association with a specific part of the character object is displayed on the display screen so as to cover part or all of the specific part of the character object.

Decorative objects include, for example, accessories (headbands, necklaces, earrings, etc.) worn on character objects, clothes (T-shirts etc.), costumes, and other objects that can be worn on character objects. The object data 23b corresponding to a decorative object may include mounting position information indicating which part of the character object the decorative object is associated with. The attachment position information of a certain decorative object can indicate which part of the character object the decoration object is attached to. For example, when the decorative object is a headband, the wearing position information of the decorative object may indicate that the decorative object is worn on the "head" of the character object. When the decorative object is a T-shirt, the wearing position information of the decorative object may indicate that the decorative object is worn on the "torso" of the character object.

Each gift object may have a display time set according to its type. In one embodiment, the display time of a decorative object may be set longer than the display time of an effect object and the display time of a normal object. For example, the display time of a decorative object may be set to 60 seconds, the display time of an effect object may be set to 5 seconds, and the display time of a normal object may be set to 10 seconds.

The holding list 23c is a list showing gift objects held by the video viewing user. An example of the possession list 23c is shown in FIG. As shown in the figure, in the holding list 23c, object IDs that identify gift objects held by the viewing user are stored in association with the viewing user's account information (for example, the viewing user's user ID). These viewing users include, for example, first to third viewing users of the client devices 10a to 10c.

Candidate list 23d is a list of decorative objects for which a viewing user has made a display request. As described below, a viewing user who owns a decorative object can make a display request for the decorative object that he or she owns. In candidate list 23d, an object ID that identifies the decorative object is stored in association with the account information of the viewing user who has made the display request for the decorative object. This candidate list 23d may be created for each broadcaster. Candidate list 23d may be stored, for example, in association with broadcaster identification information that identifies the broadcaster (actor A1, actor A2, supporter B1, and/or supporter B2).

Next, the functions realized by the computer processor 21 will be described in more detail. The computer processor 21 executes computer-readable instructions included in the distribution program to function as a body motion data generating unit 21a, a face motion data generating unit 21b, an animation generating unit 21c, a video generating unit 21d, a video distribution unit 21e, a display request processing unit 21f, a decorative object selecting unit 21g, and an object purchase processing unit 21h. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the video distribution system 1. At least a part of the functions realized by the computer processor 21 may be realized by, for example, a computer processor installed in the supporter computer 40.

The body motion data generation unit 21a generates first body motion data for each part of the body of the actor A1 based on the tracking information detected by each of the motion sensors 31a to 31f, and also generates first body motion data for each part of the body of the actor A1. Based on the tracking information detected by each, second body motion data that is a digital representation of the position and orientation of each part of the actor A2's body is generated. In this specification, the first body motion data and the second body motion data may be collectively referred to simply as "body motion data." Body motion data is generated as time passes. For example, body motion data may be generated every predetermined sampling time interval. In this way, the body motion data can represent the body movements of the actors A1 and A2 in time series as digital data. In the illustrated embodiment, the motion sensors 31a to 31f and the motion sensors 32a to 32f are attached to the left and right limbs, waist, and head of each of the actors A1 and A2. Based on the tracking information detected by the motion sensors 32a to 32f, the positions and orientations of the entire bodies of the actors A1 and A2 can be digitally expressed over time. The body motion data can define, for example, the positions and rotation angles of bones that constitute the rig data included in the model data 23a.

The face motion data generation unit 21b generates first face motion data, which is a digital representation of the facial movement of the actor A1, based on the imaging data of the camera 37a, and also based on the imaging data of each camera 38a, Second face motion data is generated which is a digital representation of the facial motion of actor A2. In this specification, the first face motion data and the second face motion data may be collectively referred to simply as "face motion data." Face motion data is generated as needed over time. For example, face motion data may be generated every predetermined sampling time interval. In this way, the face motion data can digitally express the actor A2 and the facial movements (changes in facial expression) of the actor A2 in a time-series manner.

The animation generation unit 21c applies the body motion data generated by the body motion data generation unit 21a and the face motion data generated by the face motion data generation unit 21b to predetermined model data included in the model data 23a. , is configured to generate an animation of a character object that moves in a virtual space and changes its expression. Specifically, the animation generation unit 21c generates an animation of a character object that moves in synchronization with the body and facial movements of the actor A1, based on the first body motion data and first face motion data regarding the actor A1, Based on the second body motion data and second face motion data regarding actor A2, it is possible to generate an animation of a character object that moves in synchronization with the body and facial movements of actor A2. In this specification, a character object generated based on the movement and facial expression of actor A1 is referred to as a "first character object," and a character object generated based on the movement and facial expression of actor A2 is referred to as a "second character object." ” sometimes happens.

The video generator 21d constructs a virtual space using the object data 23b, and generates a video including this virtual space, an animation of a first character object corresponding to actor A1, and an animation of a second character object corresponding to actor A2. The first character object is placed in the virtual space so as to match the position of actor A1 relative to the tracking sensor 36a, and the second character object is placed in the virtual space so as to match the position of actor A2 relative to the tracking sensor 36a. Therefore, by changing the position or orientation of the tracking sensor 36a, the positions and orientations of the first character object and the second character object in the virtual space can be changed.

In one aspect, the video generation unit 21d constructs a virtual space based on tracking information from the tracking sensor 36a. For example, the video generation unit 21d determines virtual camera setting information (position in virtual space, gaze position, gaze direction, and angle of view) based on the tracking information of the tracking sensor 36a, and based on the virtual camera setting information, A drawing area in the entire virtual space is determined by using the above method, and video information for displaying this drawing area in the virtual space is generated.

The video generating unit 21d may be configured to determine the positions and orientations of the first character object and the second character object in the virtual space, as well as the setting information of the virtual camera, based on the tracking information of the tracking sensor 36b instead of or in addition to the tracking information of the tracking sensor 36a.

The video generation unit 21d can synthesize the voices of the actor A1 and the actor A2 acquired from the microphones of the studio unit 30 into the generated video.

As described above, the video generation unit 21d generates an animation of the first character object that moves in synchronization with the body and facial movements of actor A1, and a second character object that moves in synchronization with the body and facial movements of actor A2. It is possible to generate an animation of the object, and generate a video for distribution by synthesizing the animation with the voices of the actors A1 and A2.

The video distribution unit 21e distributes the video generated by the video generation unit 21d. This moving image is distributed via the network 50 to the client devices 10a to 10c and other client devices. The video distribution unit 21e refers to a list of users who have requested video distribution (distribution destination list), and distributes the video to the client devices of the users included in the list. As will be described later, distribution of a specific video to a specific user may be prohibited based on an instruction from the supporter computer 40. In this case, the video is distributed to users other than the users who are prohibited from distributing the video. The received video is played back on the client devices 10a to 10c.

This moving image may be distributed to a client device (not shown) installed in the studio room R, and projected onto the screen S from this client device via a short throw projector. Further, this video may be distributed to the supporter computer 40. Thereby, supporter B1 and supporter B2 can check the viewing screen of the video being distributed.

FIG. 5 shows a display example of a moving image that is distributed from the server device 20 to the client device 10a and is being played back on the client device 10a. As illustrated, a display image 70 of a moving image distributed from the server device 20 is displayed on the display of the client device 10a. The display image 70 displayed on the client device 10a includes, in the virtual space, a character object 71A corresponding to actor A1, a character object 71B corresponding to actor A2, and an object 72 indicating a table. There is. The object 72 is not a gift object, but one of the objects included in the object data 23b for constructing a virtual space. The character object 71A is generated by applying the first body motion data and first face motion data of the actor A1 to the model data for the actor A1 included in the model data 23a. The motion of the character object 71A is controlled based on the first body motion data and the first face motion data. The character object 71B is generated by applying the second body motion data and second face motion data of the actor A2 to the model data for the actor A2 included in the model data 23a. The motion of the character object 71B is controlled based on the second body motion data and the second face motion data. Therefore, the character object 71A is controlled to move within the screen in synchronization with the movement of the body and facial expression of actor A1, and the character object 71B is controlled to move within the screen in synchronization with the movement of the body and facial expression of actor A2. controlled by.

As described above, the video distributed from the server device 20 may be distributed to the supporter computer 40. The video distributed to the supporter computer 40 is displayed on the supporter computer 40 in the same manner as in FIG. 5. The supporter B1 and the supporter B2 can change the settings of the components of the studio unit 30 by watching the video being played on the supporter computer 40. In one embodiment, when the supporter B1 and the supporter B2 want to change the angle of the character object 71A and the character object 71B in the video being distributed, they can send an instruction signal from the supporter computer 40 to the tracking sensor 36a instructing the change in the orientation of the tracking sensor 36a. The tracking sensor 36a can change its orientation in response to the instruction signal. For example, the tracking sensor 36a is attached to a stand so as to be rotatable around its axis via a rotation mechanism having an actuator built in, and when the tracking sensor 36a receives an instruction signal instructing the change in its orientation, the actuator of the rotation mechanism is driven based on the instruction signal, and the tracking sensor 36a may be rotated by an angle corresponding to the instruction signal. In one embodiment, supporter B1 and supporter B2 may send instructions from supporter computer 40 to tracking sensor 36a and tracking sensor 36b to use tracking information from tracking sensor 36b instead of tracking information from tracking sensor 36a. Tracking sensor 36a and tracking sensor 36b may be configured and installed so as to be movable by an actor or a supporter. This allows the actor or supporter to move tracking sensor 36a and tracking sensor 36b by holding them in their hands.

In one aspect, when the supporters B1 and B2 determine that an instruction to the actor A1 or the actor A2 is necessary after watching the video being played on the supporter computer 40, the supporter B1 and the supporter B2 send a message indicating the instruction to the supporter computer 40. This message can be input and output on the display 39. For example, supporter B1 and supporter B2 can instruct actor A1 or actor A2 to change their standing position through a message displayed on the display 39.

The display request processing unit 21f receives a gift object display request from the viewing user's client device, and performs processing according to the display request. Each viewing user can transmit a gift object display request to the server device 20 by operating his/her own client device. For example, the first viewing user can transmit a gift object display request to the server device 20 by operating the client device 10a. The gift object display request may include the user ID of the viewing user and identification information (object ID) that specifies the object that is requested to be displayed.

As mentioned above, gift objects may include effect objects, regular objects, and decoration objects. Effect objects and normal objects are examples of first objects. Further, a display request requesting display of an effect object or a normal object is an example of a second display request.

In one aspect, when the display request processing unit 21f receives a display request from a viewing user for a specific effect object, the display request processing unit 21f performs processing for displaying the effect object requested to be displayed in the display image 70 of the video based on the display request. For example, when a display request is made for an effect object corresponding to confetti, the display request processing unit 21f displays an effect object 73 representing confetti in the display image 70 based on the display request, as shown in FIG. 6.

In one aspect, upon receiving a display request for a specific normal object from a viewing user, the display request processing unit 21f performs processing for displaying the requested normal object in the video 70 based on the display request. . For example, when a request is made to display a normal object representing a stuffed bear, the display request processing unit 21f displays a normal object 74 representing a stuffed bear in the display image based on the display request, as shown in FIG. 70.

The display request for the normal object 74 may include a display position designation parameter that designates the display position of the normal object 74 in the virtual space. In this case, the display request processing unit 21f can display the normal object 74 at the position designated by the display position designation parameter in the virtual space. For example, the display position designation parameter can designate above the object 72 representing a table as the display position for the normal object 74. By using the display position designation parameter, the viewing user can view the display of the character object 71A, the character object 71B, the gift object, and other objects included in the video 70 and designate the position at which he or she wants to display the normal object.

In one aspect, the normal object 74 may be displayed to move within the display image 70 of the video. For example, the normal object 74 may be displayed to fall from the top to the bottom of the screen. In this case, the normal object 74 may be displayed within the display image 70 from the start of the fall to the floor of the virtual space constituting the video 70, and may disappear from the display image 70 after falling to the floor. The viewing user can see the falling normal object 74 from the start of the fall to the end of the fall. The movement direction of the normal object 74 within the screen may be set arbitrarily. For example, the normal object 74 may be displayed in the display image 70 to move from left to right, right to left, upper left to lower left of the video 70, or in a direction other than these. The normal object 74 can move on various trajectories. For example, the normal object 74 can move on a straight trajectory, a circular trajectory, an elliptical trajectory, a spiral trajectory, or on a trajectory other than these. In addition to or instead of the display position designation parameter, the viewing user can include at least one of a movement direction parameter that designates the movement direction of the normal object 74 and a trajectory parameter that designates the trajectory along which the normal object 74 moves, in the display request for the normal object. In one embodiment, among the effect objects and normal objects, those whose size in the virtual space is smaller than the reference size (for example, the confetti pieces of the effect object 73) may be displayed by overlapping part or all of them with the character objects 71A and 71B. In one embodiment, among the effect objects and normal objects, those whose size in the virtual space is larger than the reference size (for example, the normal object 74 (a teddy bear)) are displayed at a position that does not overlap with the character object. In one embodiment, among the effect objects and normal objects, those whose size in the virtual space is larger than the reference size (for example, the normal object 74 (a teddy bear)) are displayed behind the character object that is displayed in the overlapping state when they overlap at least one of the character objects 71A and 71B.

In one aspect, when the display request processing unit 21f receives a display request for a specific decorative object from a viewing user, it adds the decorative object requested to be displayed to the candidate list 23d based on the display request. The display request for a decorative object is an example of a first display request. For example, the display request processing unit 21f can store in the candidate list 23d, in association with the user ID of the viewing user, identification information (object ID) that identifies the specific decorative object requested to be displayed by the viewing user (see FIG. 4). When multiple display requests for decorative objects are made, the user ID of the viewing user who made the display request and the decorative object ID of the decorative object requested to be displayed by the viewing user are stored in association with each of the multiple display requests in the candidate list 23d.

In one aspect, in response to selection of one or more of the decorative objects included in the candidate list 23d, the decorative object selection unit 21g displays the selected decorative object on the video display image 70. Perform the processing to make it happen. In this specification, a decorative object selected from the candidate list 23d may be referred to as a "selected decorative object."

The selection of the decorative object from the candidate list 23d is performed by, for example, the supporter B1 and the supporter B2 operating the supporter computer 40. In one aspect, a decoration object selection screen is displayed on the supporter computer 40. FIG. 8 shows an example of a decorative object selection screen 80 in one embodiment. This decoration object selection screen 80 is displayed on the display of the supporter computer 40, for example. The decorative object selection screen 80 displays, for example, each of the plurality of decorative objects included in the candidate list 23d in a table format. As illustrated, the decorative object selection screen 80 in one embodiment includes a first column 81 that displays types of decorative objects, a second column 82 that displays images of decorative objects, and character objects with which the decorative objects are associated. and a third column 83 for displaying the parts of the image. Further, the decorative object selection screen 80 displays selection buttons 84a to 84c for selecting each decorative object. In this way, the decorative object selection screen 80 displays decorative objects that are candidates for the selected decorative object.

Supporter B1 and supporter B2 can select one or more of the decorative objects included in this decorative object selection screen 80. For example, supporter B1 and supporter B2 can select a headband by selecting the selection button 84a. When the decoration object selection unit 21g detects that a headband has been selected, the display request processing unit 21f displays a selected decoration object 75 indicating the selected headband on the video display screen 70, as shown in FIG. let The selected decoration object 75 is displayed on the display image 70 in association with a specific part of the character object. The selected decoration object 75 may be displayed so as to be in contact with the specific part of the character object. For example, the selection 75 indicating a headband is associated with the head of the character object and is therefore worn on the head of the character object 71A, as shown in FIG. The decoration object may be displayed on the display screen 70 so as to move along with the movement of the specific part of the character object. For example, when the head of the character object 71A wearing a headband moves, the selected decorative object 75 representing the headband also accompanies the head of the character object 71A, as if the headband were worn on the head of the character object 71A. It moves.

The selected decorative object 75 may be displayed on the display screen 70 in association with the character object 71B, rather than with the character object 71A. The selected decorative object 75 may be displayed on the display screen 70 in association with the character object 71A and the character object 71B.

In one embodiment, the decorative object selection screen 80 may be configured not to include information that can identify the user who owns each decorative object or the user who has requested the display of each decorative object. This can prevent the selector from giving preferential treatment to a specific user when selecting a decorative object.

In one embodiment, the decorative object selection screen 80 may display information about a user who owns each decorative object, or about a user who has made a display request for each decorative object. For example, for each decorative object, the number of times that the user who made the display request for the decorative object has made a display request for the decorative object and the number of times that the decorative object has been selected (for example, information indicating that the user has made a display request for the decorative object five times and selected the decorative object twice), the number of times that the user has viewed at least one of the videos of the character object 71A and the character object B, the number of times that the user has viewed a video (regardless of whether the character object 71A and the character object B appear) in the video distribution system 1, the purchase amount of the gift object by the user, the number of times that the user has purchased an object, the points that the user owns and can use in the video distribution system 1, the level of the user in the video distribution system 1, and various other information about the user who made the display request for each decorative object may be displayed. According to this embodiment, the decorative object can be selected based on the behavior and viewing history of the user who made the display request for the decorative object in the video distribution system 1.

In one embodiment, restrictions may be imposed on the display of decorative objects to prevent overlapping. For example, if a decorative object associated with a specific part of character object 71A has already been selected, selection of other decorative objects associated with the specific part may be prohibited. As shown in the embodiment of FIG. 7, if a headband associated with the "head" of character object 71B has already been selected, other decorative objects associated with the "head" (e.g., a decorative object representing a "hat" associated with the head) may not be displayed on the decorative object selection screen 80, or a selection button for selecting the decorative object representing the hat may be made unselectable on the selection screen 80. According to this embodiment, it is possible to prevent decorative objects from being displayed overlappingly on a specific part of a character object.

The decoration object selection screen 80 may be displayed on another device instead of or in addition to the supporter computer 40. For example, the decorative object selection screen 80 may be displayed on at least one of the display 39 and the screen S in the studio room R. In this case, actor A1 and actor A2 can select a desired decorative object based on the decorative object selection screen 80 displayed on the display 39 or screen S. The selection of decoration objects by the actors A1 and A2 can be performed, for example, by operating the controller 33a, the controller 33b, the controller 34a, or the controller 34b.

In one aspect, the object purchase processing unit 21h transmits, in response to a request from a viewing user of a video, purchase information for each of a plurality of gift objects that can be purchased in relation to the video to the viewing user's client device (e.g., client device 10a). The purchase information for each gift object may include the type of the gift object (effect object, normal object, or decorative object), an image of the gift object, the price of the gift object, and other information necessary for purchasing the gift object. The viewing user can select a gift object to purchase based on the purchase information of the gift object displayed on the client device 10a. The selection of the gift object to be purchased may be performed by operating the client device 10a. When the gift object to be purchased is selected by the viewing user, a purchase request for the gift object is transmitted to the server device 20. The object purchase processing unit 21h performs a payment process based on the purchase request. When this payment process is completed, the purchased gift object is owned by the viewing user. In this case, the object ID of the purchased gift object is stored in the ownership list 23c in association with the user ID of the viewing user who purchased it.

The purchasable gift objects may be different for each video. The purchasable gift objects may be purchasable across multiple videos. That is, the purchasable gift objects may include unique gift objects that are unique to each video and common gift objects that are purchasable across multiple videos. For example, an effect object showing confetti may be a common gift object that is purchasable across multiple videos.

In one aspect, when an effect object is purchased while viewing a predetermined video, the effect object to be purchased is changed to the video being viewed in response to completion of payment processing for purchasing the effect object. It may be displayed automatically. If you purchase a normal object while watching a certain video, the normal gift object for the purchase will be transferred to the video being viewed, as described above, upon completion of the payment process for purchasing the normal object. may be displayed automatically.

In another aspect, in response to the completion of the payment process in the object purchase processing unit 21h for the effect object to be purchased, a payment completion notification may be sent to the client device 10a, and a confirmation screen may be displayed on the client device 10a to ask the viewing user whether or not to make a display request for the purchased effect object. If the viewing user selects to make a display request for the purchased effect object, a display request requesting the display of the purchased effect object may be sent from the viewing user's client device to the display request processing unit 21f, and the display request processing unit 21f may perform processing to display the effect object to be purchased in the video 70. When the object to be purchased is a normal object, a confirmation screen may be displayed on the client device 10a to ask the viewing user whether or not to make a display request for the purchased normal object, as described above.

Next, the supporter computer 40 in one embodiment will be described. In one embodiment, the supporter computer 40 includes a computer processor 41, a communication I/F 42, a storage 43, a display 44, and an input interface 45.

The computer processor 41, like the computer processor 21, is a variety of arithmetic devices such as a CPU. The communication I/F 42, like the communication I/F 22, is a variety of drivers, software, or a combination of these for communicating with other devices. The storage 43, like the storage 23, is a variety of storage devices capable of storing data such as a magnetic disk. The display 44 is a liquid crystal display, an organic EL display, an inorganic EL display, or any other display device capable of displaying images. The input interface 45 is any input interface that accepts input from a supporter such as a mouse or keyboard. The display 44 is an example of a second display device.

Figure 9 shows an example of an image displayed on the display 44 of the supporter computer 40. In the illustrated embodiment, the display image 46 displayed on the display 44 includes a first display area 47a that displays the display screen of the video on a first type of client device (e.g., a personal computer), a second display area 47b that displays the display screen of the video on a second type of client device (e.g., a personal computer), and a number of operation icons 48.

By monitoring the video displayed in the first display area 47a, the supporter can check whether the video distributed to the first type of client device is being displayed correctly, and similarly, by monitoring the video displayed in the second display area 47b, the supporter can check whether the video distributed to the second type of client device is being displayed correctly.

In the illustrated embodiment, the operation icons 48 include a first operation icon 48a, a second operation icon 48b, a third operation icon 48c, a fourth operation icon 48d, and a fifth operation icon 48e. The supporter can select a desired operation icon via the input interface 45. When the supporter selects the first operation icon 48a, the second operation icon 48b, the third operation icon 48c, the fourth operation icon 48d, and the fifth operation icon 48e, the supporter computer 40 accepts the first operation input, the second operation input, the third operation input, the fourth operation input, and the fifth operation input, respectively.

Next, the functions realized by the computer processor 41 will be described in more detail. The computer processor 41 functions as an additional information display unit 41a and a distribution management unit 41b by executing computer-readable instructions contained in a predetermined program. At least a part of the functions realized by the computer processor 41 may be realized by a computer processor other than the computer processor 41 of the video distribution system 1. At least a part of the functions realized by the computer processor 41 may be realized by, for example, the computer processor 21.

The additional information display section 41a is configured to add various additional information to the distributed video in response to various operational inputs from supporters via the input interface 45.

In one embodiment, when the first operation input is received in the supporter computer 40 by selecting the first operation icon 48a, the additional information display section 41a displays the video displayed on the display 39 based on the first operation input. The message 101 is displayed on the client device, while the message 101 is not displayed on the client device. Message 101 is an example of first additional information.

FIG. 10 shows an example of a moving image displayed on the display 39. The display image 100 shown in FIG. 10 includes a message 101. The display image 100 shown in FIG. The display image 100 is the same as the video display image 70 (see FIG. 5) that is being distributed and displayed on the viewing user's client device at the same timing, except that it includes the message 101. In other words, the display image 100 displayed on the display 39 is configured by adding the message 101 to the video being distributed.

Message 101 is, for example, a text message entered by a supporter. Message 101 may include image information in addition to or in addition to text information. Message 101 may include instructions regarding the actor's performance, instructions regarding the actor's remarks, instructions regarding the actor's position, and various other instructions and communications related to the video being distributed.

In one embodiment, when a supporter selects the first operation icon 48a on the supporter computer 40, a window for message input is displayed on the display 44 of the supporter computer 40. The supporter can input a message in the message input area in the window by operating the input interface 45. The input message is sent from the supporter computer 40 to the display 39. The video being distributed by the video distribution unit 21e is displayed on the display 39 and the client devices 10a to 10c. As shown in FIG. 10, the message received from the supporter computer 40 is displayed as a message 101 in a predetermined area of the video being distributed by the video distribution unit 21e on the display 39. On the other hand, the message 101 or information equivalent thereto is not displayed on the video displayed on the client devices 10a to 10c. According to this embodiment, instructions and contact information related to the video being distributed can be transmitted to the actor or other distribution side staff who can see the display 39 without affecting the video being viewed by the viewing user of the client device.

Next, the display of the second additional information will be described with reference to Figures 11a and 11b. In one embodiment, when a second operation input is accepted in the supporter computer 40 by selecting the second operation icon 48b, the additional information display unit 41a is configured to display an interruption image in the video displayed on the display 39 and the video displayed on the client device based on the second operation input. The interruption image is an example of the second additional information. The second operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1, or in response to the operation of the controller 34a or 34b of the actor A2.

Figure 11a shows an example of a display image 110a displayed on the client device 10a when the second operation input is accepted, and Figure 11b shows an example of a display image 110b displayed on the display 39 when the second operation input is accepted. Both display images 110a and 110b include an interruption image 111. The interruption image 111 is set in a layer above the video being distributed. As a result, the interruption image 111 is displayed on the client device and display 39 while the video is being played. There may be slight visual differences between the interruption image 111 displayed on the client device and the interruption image 111 displayed on the display 39 depending on the performance of the device on which they are displayed, but the two are essentially the same image.

In the illustrated embodiment, the interruption image 111 is an emergency display image that is displayed in place of a video including a normal virtual space and a character object when an unexpected event occurs during video distribution. For example, when a malfunction of the equipment used in studio room R causes a problem in controlling the motion of the character object, the supporter can select the second operation icon 48b on the supporter computer 40. In response to the selection of the second operation icon 48b, the additional information display unit 41a performs processing to display the interruption image 111 on each client device and the display 39.

When a first operation input is received by selecting the first operation icon 48a while the interruption image 111 is displayed on the display 39, the additional information display unit 41a may display the message 101 on the display 39, superimposed on the interruption image 111.

According to the above embodiment, when an unexpected situation occurs during video distribution, instead of continuing to distribute the normal video as it is, an interrupt image 111 for emergency display is distributed to the client device and the display 39. Can be done. The distributor side can try to bring the situation under control while this interrupt image 111 is being displayed.

Next, display of the third additional information and generation of a modified character object will be described with reference to FIGS. 12a and 12b. In one embodiment, when the third operation input is received in the supporter computer 40 by selecting the third operation icon 48c, the additional information display unit 41a generates a modified character object and distributes a video including the modified character object. It is configured to perform processing to This third operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1 or in response to the operation of the controller 34a or 34b of the actor A2.

FIG. 12a shows another example of a display image 120a being displayed on the client device 10a, and FIG. 12b shows an example of a display image 120b being displayed on the client device 10a. The display image 120a in FIG. 12a includes a character object 71B, and the display image 120a in FIG. 12b includes a modified character object 171B, which will be described later. As described above, the facial expression of the character object 71B is controlled to change in synchronization with the movement of the facial expression of the actor A2 based on the facial motion data of the actor A2. In this specification, all or part of a part of a character object that changes based on face motion data may be referred to as a "face part." In the illustrated embodiment, motion control is performed so that the eye movements of the character object 71B move in synchronization with the eye movements of the actor A2, so the face portion 121 is set at a position that includes both eyes of the character object 71B. has been done. The face portion 121 may be set to include the entire face of the character object 71B.

In the display image 120a shown in FIG. 12a, the image of the eyes of the character object 71B displayed in the face portion 121 is generated by applying face motion data to model data. In contrast, in the display image 120b shown in FIG. 12b, the eye image displayed within the face portion 121 is an embedded image 122b that has been prepared in advance before the start of video distribution. The embedded image 122b may be stored in the storage 23. The embedded image 122b is an example of third additional information.

In one embodiment, the additional information display unit 41a is configured to display the inlaid image 122b in the face portion 121 of the character object 71B when the third operation input is received. For example, when the additional information display unit 41a receives the third operation input, the additional information display unit 41a transmits a modification instruction to the animation generation unit 21c, and the animation generation unit 21c is configured to generate an animation of the modified character object 171B by embedding the inlaid image 122b, instead of an image generated based on the face motion data, in the face portion 121 of the character object 71B based on the modification instruction from the additional information display unit 41a. In this specification, a character object in which the inlaid image 122b, instead of an image generated based on the face motion data, is embedded in the face portion 121 may be referred to as a modified character object. The modified character object is an object in which a part of the character object generated by the animation generation unit 21c is modified. In the above example, the animation of modified character object 171B is generated by displaying inlay image 122b inlayed on face portion 121 of character object 71B generated by animation generation unit 21c.

The process of applying face motion data to model data to change the facial expression of a character object places a high processing load on the processor. As a result, the facial expression of the character object may not be able to timely follow the facial expression of the actor. Movements other than voice and facial expressions can timely follow the vocalizations and body movements of the actor, so if the facial expression of the character object cannot timely follow the facial expression of the actor, it gives a strange feeling to the viewing user. According to the above embodiment, when the facial expression of the character object cannot timely follow the facial expression of the actor, a modified character object incorporating a previously prepared inset image 122b is generated, and a video including this modified character object is distributed, thereby preventing a decrease in the quality of the video due to the character object's facial expression not being able to follow the facial expression of the actor.

The storage 23 may store a number of different types of images as candidates for the inlay image 122b. The candidates for the inlay image 122b are displayed on the display 44 of the supporter computer 40, and an image selected by the supporter from among the candidates may be used as the inlay image 122b.

Next, the display of blind objects will be described with reference to FIG. 13. In one embodiment, when the fourth operation input is received by selecting the fourth operation icon 48d, the additional information display unit 41a displays a blind object for hiding at least a part of the character object to which the decoration object is attached. It is configured as follows. This fourth operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1 or in response to the operation of the controller 34a or 34b of the actor A2.

In the embodiment of FIG. 13, it is assumed that a T-shirt is selected on the selection screen 80 as the decoration object to be worn on the character object 71B. The additional information display section 41a can display a blind object so as to hide a part of the character object 71B that is associated with the selected decoration object. In the display image 130 of FIG. 13, the blind object 131 is displayed so that at least the torso of the character object 71B associated with the T-shirt, which is the selected decoration object, is hidden.

Next, a process for prohibiting video distribution to a distribution-prohibited user will be described. In one embodiment, when the fifth operation input is received by selecting the fifth operation icon 48e, the distribution management unit 41b performs processing for prohibiting distribution of the video to a specific viewing user or a specific client device. configured to do so. In one embodiment, when the fifth operation icon 48e is selected during distribution of a video, a list of users who are viewing the video is displayed. The distribution management unit 41b performs processing for canceling the distribution of the video to the user selected by the supporter or actor from this list. For example, the distribution management unit 41b can write a flag for identifying the selected user (distribution prohibited user) in the video distribution destination list. The video distribution unit 21e can distribute the video to users who are included in the video distribution destination list and who are not given flags that identify distribution-prohibited users. As a result, distribution of the video to the distribution-prohibited user is stopped. In another embodiment, the distribution management unit 41b disables a user selected by a supporter or actor from a list of users who are watching the video being distributed, from using some of the functions normally available to the user. It may be configured to do so. For example, the distribution management unit 41b may be configured to prohibit a user who has posted an inappropriate message from posting a message. A user who is prohibited from posting messages can continue to watch the video even after the prohibition, but will no longer be able to post messages related to the video.

Next, a video distribution process according to one aspect will be described with reference to FIGS. 14 to 19. FIG. 14 is a flowchart showing the flow of video distribution processing in one embodiment, FIG. 15 is a flowchart showing the flow of processing for displaying first additional information in one embodiment, and FIG. FIG. 17 is a flowchart showing the flow of processing for displaying the second additional information in an embodiment, FIG. 17 is a flowchart showing the flow of processing for displaying the third additional information in an embodiment, and FIG. FIG. 19 is a flow diagram illustrating the flow of a process for displaying a blind object in an embodiment, and FIG. 19 is a flow diagram illustrating a flow of a process for prohibiting video distribution to a distribution-prohibited user in an embodiment.

First, with reference to FIG. 14, video distribution processing in one embodiment will be described. In the video distribution process, it is assumed that actor A1 and actor A2 are performing in studio room R. First, in step S11, body motion data that is a digital representation of the body movements of actors A1 and A2, and face motion data that is a digital representation of facial movements (expressions) of actors A1 and A2 are generated. Ru. Body motion data is generated, for example, by the body motion data generation section 21a, and face motion data is generated, for example, by the face motion data generation section 21b.

Next, in step S12, the body motion data and facial motion data of actor A1 are applied to the model data for actor A1 to generate an animation of a first character object that moves in synchronization with the body and facial movement of actor A1. Similarly, the body motion data and facial motion data of actor A2 are applied to the model data for actor A2 to generate an animation of a second character object that moves in synchronization with the body and facial movement of actor A2. This animation is generated, for example, by the animation generation unit 21c described above.

Next, in step S13, a video is generated that includes an animation of a first character object corresponding to actor A1 and an animation of a second character object corresponding to actor A2. The audio of actors A1 and A2 may be synthesized into this video. The animation of the first character object and the animation of the second character object may be placed in a virtual space. The generation of this video is performed, for example, by the video generation unit 21d described above.

Next, the process moves to step S14, where the video generated in step S13 is distributed. This video is distributed to client devices 10a to 10c and other client devices via network 50. This video is distributed to supporter computer 40, and may also be projected onto a screen S in studio room R. The video is distributed continuously for a specified distribution period. The video distribution period can be set to, for example, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 60 minutes, 120 minutes, or any other time.

Next, the process advances to step S15, and it is determined whether the termination condition for terminating the distribution of the moving image is fulfilled. The termination condition is, for example, that the distribution end time has been reached, that the supporter computer 40 has given an instruction to terminate the distribution, or other conditions. If the termination condition is not met, the processes of steps S11 to S14 are repeatedly executed, and the distribution of the moving image including animation synchronized with the movements of actors A1 and A2 is continued. If it is determined that the termination condition is fulfilled for the video, the distribution process for the video is ended.

Next, with further reference to FIG. 15, the display processing of the first additional information performed during video distribution will be described. During the distribution of the video, it is determined in step S21 whether or not a first operation input has been made. For example, when the first operation icon 48a is selected on the supporter computer 40, it is determined that the first operation input has been made. If the first operation input has been made, the process of displaying the first additional information proceeds to step S22. In step S22, the message 101, which is an example of the first additional information, is displayed on the moving image displayed on the display 39, and the process of displaying the first additional information ends. At this time, the message 101 is not displayed on the client device.

Next, with further reference to FIG. 16, the display process of the second additional information performed during video distribution will be described. During video distribution, in step S31, it is determined whether or not a second operation input has been made. For example, when the second operation icon 48b is selected on the supporter computer 40, it is determined that a second operation input has been made. If a second operation input has been made, the display process of the second additional information proceeds to step S32. In step S32, an interruption image 111, which is an example of the second additional information, is displayed on the client device and the display 39, respectively, and the display process of the second additional information ends.

Next, with further reference to FIG. 17, a description will be given of the third additional information display process performed during video distribution. During the distribution of the video, it is determined in step S41 whether or not a third operation input has been made. For example, when the third operation icon 48c is selected on the supporter computer 40, it is determined that the third operation input has been made. If the third operation input has been made, the third additional information display process proceeds to step S42. In step S42, the embedded image 122b, which is an example of the third additional information, is displayed on the face of the character object, and the third additional information display process ends.

Next, with further reference to FIG. 18, the display process of blind objects performed during video distribution will be described. During video distribution, in step S51, it is determined whether or not a decorative object to be attached to a character object appearing in the video has been selected. As described above, the decorative object to be attached to the character object can be selected from among the decorative object candidates included in the candidate list 23d. The decorative object selection process is executed, for example, by the decorative object selection unit 21g described above.

If it is determined in step S51 that a decorative object has been selected, the process proceeds to step S52. In step S52, it is determined whether or not a fourth operation input has been made. For example, when the fourth operation icon 48d is selected on the supporter computer 40, it is determined that a fourth operation input has been made. If a fourth operation input has been made, the display process of the blind object proceeds to step S53. In other embodiments, it may be determined that a fourth operation input has been made when a decorative object has been selected from the candidate list 23d. In this case, step S52 is omitted, and the display process of the blind object proceeds from step S51 to step S53.

In step S53, the blind object 131 is displayed so as to hide at least a portion of the character object. For example, if the decorative object selected in step S51 is a T-shirt, the blind object 131 is added to the video so as to hide the torso of the character object associated with the T-shirt.

Next, further referring to FIG. 19, the distribution prohibition process for prohibiting distribution of a video to a specific user will be described. During distribution of a video, in step S61, it is determined whether or not a fifth operation input has been made. For example, when the fifth operation icon 48e is selected on the supporter computer 40, it is determined that a fifth operation input has been made.

When the fifth operation input is made, the distribution prohibition process proceeds to step S62. In step S62, users who are prohibited from distributing the video (distribution prohibited users) are designated. For example, a list of users who are watching the video is displayed on the display 44 of the supporter computer 40, and a user selected from this list by the supporter is designated as a distribution prohibited user. The process for designating distribution prohibited users is executed, for example, by the above-mentioned distribution management unit 41b.

Then, the distribution prohibition process proceeds to step S63. In step S63, the video is distributed only to users included in the distribution destination list of the video who are not designated as distribution prohibited users. This video distribution process is executed, for example, by the video distribution unit 21e described above.

According to the video distribution system 1 in the embodiment described above, it becomes easy to deal with unexpected situations that occur during live distribution. For example, by displaying the message 101 on the display 39 without displaying it on the client device, staff on the distribution side can view the display 39 of the actors or others without affecting the video being viewed by the viewing users. You can send instructions and messages related to the videos being distributed.

According to another embodiment, when an unexpected situation occurs during video distribution, an interrupt image 111 for emergency display is distributed to the client device and the display 39 instead of continuing to distribute the normal video as it is. be able to.

According to another embodiment, even if the facial expression of the character object cannot timely follow the facial expression of the actor, a modified character object incorporating a pre-prepared embedding image 122b is generated, and a video including this modified character object is distributed, thereby preventing a decrease in the quality of the video due to the character object's facial expression being unable to follow the facial expression of the actor.

According to another embodiment, even if some viewing users behave inappropriately, such as posting inappropriate messages, the distribution of videos to those viewing users can be stopped.

The embodiments of the present invention are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. For example, the video to be distributed may be filmed and generated outside of the studio room R. For example, filming for generating the video to be distributed may be performed at the home of an actor or supporter.

In the processing procedures described in this specification, particularly in processing procedures described using flow diagrams, it is possible to omit some of the processes (steps) that make up the processing procedures, to add processes that are not explicitly stated as processes that make up the processing procedures, and/or to change the order of the steps, and processing procedures in which such omissions, additions, or changes in order have been made are also included within the scope of the present invention as long as they do not deviate from the spirit of the present invention.

The inventions described in the claims of the original application of this application are set forth below.
[1]
a distribution server device configured to live-distribute a video including an animation of a character object generated based on a movement of an actor to a first client device used by a first user;
a supporter computer configured to prohibit the first user from posting a message to the video based on a fifth operation input;
A video distribution system comprising:
[2]
the supporter computer displays a distribution destination list indicating users who are currently viewing the video in response to the fifth operation input,
When the first user is selected from the distribution destination list, prohibiting the first user from posting a message to the video.
The video distribution system according to [1].
[3]
The first user can watch the video after the posting of messages to the video is prohibited.
A video distribution system according to [1] or [2].
[4]
the supporter computer displays a distribution destination list indicating users who are currently viewing the video in response to the fifth operation input,
When the first user is selected from the distribution destination list, the distribution of the video to the first user is stopped.
A video distribution system according to [1] or [2].
[5]
the supporter computer is configured to generate a modified character object by modifying at least a part of the character object based on a third operation input, and to include the modified character object in the video.
A video distribution system according to any one of [1] to [4].
[6]
the character object has a face portion that is generated to move based on face motion data that indicates a movement of the face of the actor;
the distribution server device is configured to display third additional information on the face portion based on the third operation input.
The video distribution system according to [5].
[7]
Further comprising a storage device for storing a decorative object to be displayed in association with the character object;
the supporter computer is configured to display a blind object for hiding at least a part of the character object when the decorative object is displayed in the video based on a fourth operation input.
A video distribution system according to any one of [1] to [6].
[8]
the distribution server device is configured to live-distribute the video to a second client device used by a second user in addition to the first client device;
the first client device is a device of a first type;
the second client device is a second type of device different from the first type of device;
the supporter computer includes a second display device that displays a display image of the moving image displayed on the first client device and a display image of the moving image displayed on the second client device;
A video distribution system according to any one of [1] to [7].
[9]
the supporter computer is configured to prohibit distribution of the video to the first user based on a fifth operation input.
A video distribution system according to any one of [1] to [8].
[10]
1. A method for streaming video that is executed by one or more computer processors executing computer readable instructions, comprising:
live streaming a video including an animation of a character object generated based on a movement of an actor to a first client device used by a first user;
prohibiting the first user from posting a message to the video based on a fifth operation input to a supporter computer;
A video distribution method comprising:
[11]
one or more computer processors,
live streaming a video including an animation of a character object generated based on a movement of an actor to a first client device used by a first user;
prohibiting the first user from posting a message to the video based on a fifth operation input to a supporter computer;
A distribution program that executes the above.

1 Video distribution system

Claims (8)

A video distribution system that includes one or more computer processors and that live-distributes video including animation of a character object generated based on a movement of an actor, comprising:
the character object has a face portion that is generated to move based on face motion data that indicates a movement of the face of the actor;
The one or more processors execute the computer readable instructions to:
in response to a third operation input being accepted by the controller used by the actor, third additional information is displayed on a part of the face portion of the character object, thereby generating a modified character object by modifying the character object, and including the modified character object in the video;
Video distribution system. the character object has a face portion that is generated to move based on face motion data that indicates a movement of the face of the actor;
the one or more processors display third additional information on the face portion based on the third operation input.
The video distribution system according to claim 1 . Further comprising a storage device for storing a decorative object to be displayed in association with the character object;
the one or more processors display a blind object for hiding at least a part of the character object when the decorative object is displayed in the video based on a fourth operation input;
3. The video distribution system according to claim 1 or 2. a first display device arranged at a position visible to the actor and configured to display the moving image;
The video distribution system according to any one of claims 1 to 3. The video is distributed to a first client device used by a first user,
The one or more processors may cause second additional information to be displayed on the video displayed on the first display device and the video displayed on the first client device based on a second operation input. consists of,
The video distribution system according to claim 4. the one or more processors are configured to prohibit distribution of the video to a first user who is watching the video, based on a fifth operation input;
The video distribution system according to any one of claims 1 to 4. 1. A method for streaming video that is executed by one or more computer processors executing computer readable instructions, comprising:
live streaming a video including an animation of a character object having a face portion generated based on a movement of an actor and generated to move based on face motion data indicating a movement of the face of the actor;
a step of generating a modified character object by modifying the character object by displaying third additional information on a part of the face portion of the character object in response to a third operation input being accepted by the controller used by the actor, and including the modified character object in the video;
A video distribution method comprising: one or more computer processors,
live streaming a video including an animation of a character object having a face portion generated based on a movement of an actor and generated to move based on face motion data indicating a movement of the face of the actor;
a step of generating a modified character object by modifying the character object by displaying third additional information on a part of the face portion of the character object in response to a third operation input being accepted by the controller used by the actor, and including the modified character object in the video;
A distribution program that executes the above.