JP7486283B2 - Video distribution system - Google Patents

Description

The present invention relates to a video distribution system.

Video distribution systems are becoming widespread. Video distribution can be categorized into a streaming system, in which a series of videos are played passively, and an on-demand system, in which videos are played actively and as intended. In the on-demand system, the user is allowed to set the order in which videos are played (see, for example, Patent Document 1).

Patent No. 5350728

However, when streaming video distribution, while it is possible to change the live performance while observing the viewer's reaction when using live footage, when using recorded video, the video's development is fixed and it is difficult to distribute the video in a way that responds to the viewer's reaction.

The present invention was made in light of this background, and aims to provide a technology that can deliver videos that match the viewers' reactions.

The main invention of the present invention for solving the above problems is a video distribution system that includes a material library that stores material data for generating videos, a video distribution unit that transmits the videos based on the material data, a reaction acquisition unit that acquires reaction information indicating reactions from viewers of the videos while the videos are being played back, and an action processing unit that performs operations on the material data according to the reaction information.

Other problems and solutions disclosed in this application will be made clear in the description of the embodiments of the invention and the drawings.

The present invention makes it possible to deliver videos that match the viewer's reactions.

1 is a diagram illustrating an example of the overall configuration of a video distribution system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of a distributor device 3. FIG. 2 is a diagram illustrating an example of a software configuration of a distributor device 3. FIG. 2 is a diagram showing an example of the structure of a material library 331. FIG. 13 is a diagram illustrating an example of the configuration of an action storage unit 333. FIG. 13 illustrates an example of the configuration of an event storage unit 334. FIG. 2 is a diagram illustrating a flow of processing executed by the video distribution system of the present embodiment.

The contents of the embodiment of the present invention will be listed and explained. The video distribution system according to the embodiment of the present invention has the following configuration.

[Item 1]
a material library that stores material data for generating a video;
a video distribution unit that transmits the video based on the material data;
A reaction acquisition unit that acquires reaction information indicating reactions from viewers of the video during playback of the video;
an action processing unit that performs an operation on the material data in response to the reaction information;
A video distribution system comprising:
[Item 2]
Item 1 is a video distribution system according to the present invention,
The reaction acquisition unit acquires an event that has occurred in the video distribution system,
the action processing unit performs an operation on the material data in response to the event;
A video distribution system characterized by the above.
[Item 3]
Item 1 is a video distribution system according to the present invention,
The reaction from the viewer includes at least any of the operation of a device by the viewer, the voice and characters input by the viewer, an image of the viewer, the posture, line of sight and gesture of the viewer analyzed from the image, and measurement data measuring the physical reaction of the viewer;
A video distribution system characterized by the above.
[Item 4]
Item 1 is a video distribution system according to the present invention,
The action includes at least one of appearance, movement, and disappearance of an object, start and stop of an animation, playback and stop of a video, playback and stop of an audio, and change of the material data;
A video distribution system characterized by the above.

The following describes a video distribution system according to one embodiment of the present invention. The video distribution system of this embodiment dynamically changes the video streamed in response to various events.

Events include information acquired about the viewer (hereafter referred to as user event information), such as the viewer operating a device, the viewer inputting voice or text, the viewer being photographed, the viewer's posture, line of sight, gestures, etc. being analyzed from the photographed image, and measurement data that measures the viewer's physical reactions. Events can also include information that can be acquired by the system (hereafter referred to as system event information), such as the number of viewers viewing, the weather, the time, weather, and contact with an object in the video.

The videos that are distributed are generated based on material data such as 3D models, video data, and audio data. Actions can be set for each piece of material data according to events.

Actions are operations on material data, and can include the appearance, movement, and disappearance of objects represented by 3D models, the start and end of animation actions, the playing and stopping of video, the playing and stopping of audio, and the changing (replacing) of material data. In this embodiment, the video to be distributed is created by synthesizing footage or images that are output as a result of the actions.

In the video distribution system of this embodiment, the distributed video can be dynamically changed by initiating an action in response to an event. Details are explained below.

Figure 1 is a diagram showing an example of the overall configuration of a video distribution system according to this embodiment. The video distribution system according to this embodiment includes a distributor device 3, which is connected to a streaming server 2 so as to be able to communicate with the streaming server 2, and which is connected to a viewer terminal 1 so as to be able to communicate with the streaming server 2. A large number of viewer terminals 1 may be connected. The distributor device 3 is a computer that transmits video, such as a personal computer, tablet computer, or smartphone. The distributor device 3 may be a home computer operated by a general user who wishes to distribute video. Video data is sent from the distributor device 3 to the streaming server 2 by a streaming method. The streaming server 2 is a computer such as a workstation, a personal computer, or a virtual computer provided by cloud computing, which distributes the video data received from the distributor device 3 to the viewer terminal 1. The streaming server 2 can simultaneously distribute video data to one or more viewer terminals 1. The viewer terminal 1 is a computer operated by the viewer, such as a personal computer, tablet computer, or smartphone. The viewer operates the viewer terminal 1 to access the streaming server 2, and the viewer terminal 1 can receive and play the video data sent from the streaming server 2. The viewer terminal 1 can also collect various types of event information, such as photographing the viewer, recording the viewer's voice, and accepting comments from the viewer. Events from one or more viewers are collected by the streaming server 2, and the distributor device 3 can obtain events from the viewers by, for example, calling an API (Application Programming Interface) provided by the streaming server 2.

In the following explanation, the entire video that is distributed by streaming and played linearly is specifically referred to as "distributed video." Distributed video can be in a format where images and audio captured by a camera or microphone are encoded into video data in real time and sent out, but in this embodiment, it is assumed that the distributed video is generated as video data synthesized by combining material data.

2 is a diagram showing an example of the hardware configuration of the distributor device 3. The distributor device 3 includes a CPU 301, a memory 302, a storage device 303, a communication interface 304, an input device 305, and an output device 306. The storage device 303 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 304 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 305 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 306 is, for example, a display, a printer, a speaker, etc. for outputting data.

Figure 3 is a diagram showing an example of the software configuration of the distributor device 3. As shown in the figure, the distributor device 3 includes a video distribution unit 311, an event acquisition unit 312, an action processing unit 313, a material library 331, an action storage unit 333, and an event storage unit 334.

The video distribution unit 311, event acquisition unit 312, and action processing unit 313 are realized by the CPU 301 of the distributor device 3 reading out a program stored in the storage device 303 into the memory 302 and executing it, and the material library 331, action storage unit 333, and event storage unit 334 are realized as part of the storage area provided by at least one of the memory 302 and storage device 303 of the distributor device 3.

The material library 331 stores various types of data (material data) that can be incorporated into videos. Examples of material data include 3D models, still images, video images, audio data, MIDI data, and text data. Figure 4 shows an example of the structure of the material library 331. As shown in the figure, the material library 331 stores various types of material data in association with information (material ID) for identifying the material data.

The action storage unit 333 stores information about operations (actions) on material data (hereinafter, referred to as action information). Figure 5 is a diagram showing an example of action information stored in the action storage unit 333. As shown in the figure, the action information includes an action ID indicating the action, a judgment period, a condition, and the action.

The judgment period is information that indicates the period during which it is determined whether an event satisfies a condition. The judgment period may be specified relatively, for example, from the present time up to five minutes ago, or it may be specified absolutely, for example, from 00:00 to 00:10 on December 25th.

The condition is for an event, and may be, for example, whether or not an event occurs, or whether or not an aggregate value relating to the event from multiple viewers exceeds a threshold.

For user event information, for example, conditions can be whether or not a viewer pressed a specific key while playing the distributed video, whether or not a viewer clicked or tapped a specific area on the video, whether or not the number of viewers who clicked or tapped a specific area on the video exceeded a threshold, etc. Based on the material data used to generate the distributed video, conditions can also be whether or not an object displayed in the distributed video was clicked or tapped, and whether or not the number of viewers who clicked or tapped the object exceeded a threshold.

Conditions can also be whether or not a viewer has input a specific character (keyword) into the viewer terminal 1, for example, whether or not a specific keyword was included in a comment posted to a distributed video, or whether or not the number of viewers who input a keyword exceeded a threshold. Conditions can also be that the number of comments or viewers tallied by region of viewers who posted comments, for example, the total amount posted by viewers in response to a question in a video about their opinions on amounts (average, median, maximum, minimum, etc.), the number of viewers who are sad based on the viewers' emotions as determined by syntactic analysis of the comments, exceed a specific threshold, or that no comments containing a specific keyword have ever been posted.

In addition, by performing voice recognition, it is possible to set conditions such as whether or not a particular pronunciation was made by the viewer or the viewer's mood, such as whether or not a particular word was spoken, whether or not clapping, whistling, stomping, etc., and whether or not the number of viewers who made such particular pronunciations exceeded a threshold. In this case, the voice recognition process may be performed by the viewer terminal 1, and the recognition results may be transmitted to the streaming server 2 or the distributor device 3, or the captured voice data may be transmitted from the viewer terminal 1 via the streaming server 2 or directly to the distributor device 3, and the voice recognition process may be performed by the distributor device 3.

In addition, the conditions may be whether or not the viewer has taken a selfie image or a photo of the viewer's mood on the viewer terminal 1, whether or not such an image has been posted to the streaming server 2, and whether or not the number of viewers who have posted images has exceeded a threshold. In addition, the conditions may be whether or not an image of a specific object (for example, an image of a specific sign or flyer around the viewer) has been posted by image analysis, and whether or not the number of viewers who have posted such an image has exceeded a threshold. Furthermore, the conditions may be whether or not the viewer's posture, line of sight, gesture, etc. has been recognized by image analysis, and whether or not the viewer has taken a specific posture, line of sight, gesture, and whether or not the number of viewers who have taken a specific posture, line of sight, gesture, etc. has exceeded a threshold. In this case, the image analysis process may be performed by the viewer terminal 1, and the analysis results may be transmitted to the streaming server 2 or the distributor device 3, or the image data captured by the viewer terminal 1 may be transmitted to the distributor device 3 via the streaming server 2 or directly, and the image analysis process may be performed by the distributor device 3.

For system event information, a condition can be, for example, whether the number of viewers viewing the distributed image exceeds a threshold. A condition can also be whether a predetermined time has passed since the distributed image was played. If the weather in the location of the viewer or distributor or in a predetermined area is obtained as system event information, a condition can be whether the weather is a predetermined weather (for example, sunny, raining, snowing, wind blowing at a predetermined strength or higher, temperature above a predetermined temperature, etc.). If the material data is a 3D model, a condition can also be whether an object collides during the distributed video.

As described above, actions are various operations on material data. Actions can also include parameters, and it is possible to set in detail what specific operations are to be performed on the material data. Furthermore, in this embodiment, actions can include not only operations on material data, but also processes such as executing or stopping other actions, and waiting for the execution of other actions to be completed. Actions can also be stored, for example, as programs written in a scripting language.

The video distribution unit 311 distributes videos. Multiple actions may be executed from an action. The video distribution unit 311 can generate a distribution video by synthesizing (and encoding as necessary) the output from the actions executed by the action processing unit 313 described below. The video distribution unit 311 can also distribute the video to the viewer terminal 1 by transmitting the generated distribution video to the streaming server 2.

The event storage unit 334 stores various events that have occurred. FIG. 6 is a diagram showing an example of the configuration of the event storage unit 334. As described above, the event storage unit 334 stores user event information and system event information. Hereinafter, the user event information and the system event information are collectively referred to as event information. The user event information includes an event ID indicating an event, a viewer ID indicating a viewer related to the event, the date and time when the event occurred, and the content of the event. The system event information includes an event ID, the date and time when the event occurred, and the content of the event. The content of the event varies depending on the event. For example, in the case of user event information, when a viewer posts a comment on a distributed video, the posted comment can be set as the event content. In addition, when the viewer terminal 1 analyzes an image captured of a viewer and obtains the viewer's gaze and gestures, information indicating these gazes and gestures can be set as the event content. In the case of system event information, the number of viewers, the weather, the time, and the like can be set as the event content.

The event acquisition unit 312 acquires an event and registers the contents of the acquired event as event information in the event storage unit 334. Note that the type of event that the event acquisition unit 312 acquires can be set by the user. The setting information indicating the type of event to be acquired can be stored, for example, in the memory 302 or the storage device 303.

The event acquisition unit 312 can register events such as keystrokes, taps, and clicks entered by the viewer as user event information. For example, when a distributed video is being viewed on a web browser, a script can be set to transmit a click event that occurs on the web browser to the streaming server 2 or the distributor device 3, and the event acquisition unit 312 can acquire the click event via the streaming server 2 or directly.

In addition, for example, if the streaming server 2 provides an API for acquiring posted comments, the event acquisition unit 312 can acquire comments posted by viewers by calling the API. The event acquisition unit 312 may also analyze the syntax of the comment and register the inclusion of a predetermined keyword as user event information.

The event acquisition unit 312 can receive voice data captured from the viewer terminal 1, analyze the received voice data, and register the occurrence of a specific pronunciation as user event information. Note that the voice analysis process may be performed in the viewer terminal 1, and the event acquisition unit 312 may receive the analysis result directly from the viewer terminal 1 or via the streaming server 2 and register it as user event information. The event acquisition unit 312 can also receive an image of a viewer from the viewer terminal 1, analyze the received image, and acquire the posture, line of sight, gesture, and the like of the viewer. Alternatively, the image may be analyzed in the viewer terminal 1, and the result may be received. The event acquisition unit 312 may periodically count the number of viewers watching and register it in the system event information. Note that the image analysis process may be performed in the viewer terminal 1, and the event acquisition unit 312 may receive the analysis result directly from the viewer terminal 1 or via the streaming server 2 and register it as user event information.

The event acquisition unit 312 can also acquire a timeout event from a clock, for example, and register the passage of a predetermined time from the start of playback of a distributed image as system event information. The event acquisition unit 312 can also register the arrival of a predetermined time (for example, the date changing, the arrival of 3:00 p.m., etc.) as system event information.

The event acquisition unit 312 can also acquire weather information, for example, from a weather company's server, and register weather elements in a specific area (for example, the location of each viewer or the location of the distributor, or a pre-set area) as system event information.

In addition, if the material data is a 3D model, for example, the event acquisition unit 312 can determine that an object has collided during the distributed video and register this as system event information.

The action processing unit 313 executes an action on the material data. The action processing unit 313 accepts the specification of the action to be executed first, and executes the accepted action. Since an action may include the launch of other actions, in this case the action processing unit 313 can continue to launch actions sequentially. Note that an action can also branch into multiple actions and be executed in parallel.

The action processing unit 313 also executes actions according to events. For each piece of action information stored in the action storage unit 333, the action processing unit 313 reads out event information corresponding to a period from the event storage unit 334, tallys it up, determines whether a condition is met, and if the condition is met, executes the action set in the action information. This makes it possible to dynamically change the progression of the video according to the event.

Figure 7 is a diagram explaining the flow of processing executed by the video distribution system of this embodiment.

The action processing unit 313 accepts the specification of the action to be executed first (S421), and executes the specified action (S422). The video and audio output as a result of the execution of the action are synthesized and transmitted as video data to the streaming server 2 by the video distribution unit 311.

The event acquisition unit 312 periodically acquires events (S423). For example, the event acquisition unit 312 can acquire comments posted on a distributed video by calling an API provided by the streaming server 2. The event acquisition unit 312 can also acquire the lapse of a predetermined time period using a timer function.

For each piece of action information stored in the action storage unit 333, the action processing unit 313 reads out event information for a date and time that falls within the judgment period from the event storage unit 334 (S424), and determines whether the event content of the read event information satisfies the conditions of the action information (S425). If the event satisfies the conditions (S426: YES), the action processing unit 313 executes the action included in the action information (S427). Here, the action processing unit 313 may stop the previously executed action, or may execute a new action in parallel with the previously executed action, or may make this behavior specifiable within the new action.

If the action in the action information does not end the streaming video (S428), the process is repeated from step S423.

In this way, the video distribution system of this embodiment distributes video to the viewer terminal 1 via the streaming server 2, and can make changes to the video being played back in response to various events that occur during playback of the video. Therefore, while using pre-recorded video, it is possible to add various developments to the video in response to the viewer's reaction, and it is possible to distribute videos without losing the live feeling that comes with streaming video distribution.

Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the interpretation of the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

For example, in this embodiment, the video sent from the distributor device 3 is distributed to the viewer terminal 1 via the streaming server 2, but it is also possible to distribute the video directly from the distributor device 3 to each viewer terminal 1.

In addition, in this embodiment, even if the video is in the middle of playing, if the comment satisfies the condition, the transition to the next part video data is made (step S427 in FIG. 7), but the execution of the current action may be completed before the execution of the next action.

REFERENCE SIGNS LIST 1 Viewer terminal 2 Streaming server 3 Distributor device 311 Video distribution unit 312 Event acquisition unit 313 Action processing unit 331 Material library 333 Action storage unit 334 Event storage unit

Claims (3)

a material library that stores material data for generating a video;
a video distribution unit that transmits the video based on the material data;
A reaction acquisition unit that acquires reaction information indicating reactions from viewers of the video during playback of the video;
an action processing unit that performs an operation on the material data during playback of the video in accordance with a total value of the reaction information;
Equipped with
The operation on the material data includes at least one of appearance, movement, and disappearance of an object represented by a 3D model;
A video distribution system characterized by the above. The video distribution system according to claim 1,
The reaction acquisition unit acquires an event that has occurred in the video distribution system,
the action processing unit performs an operation on the material data in response to the event;
A video distribution system characterized by the above. The video distribution system according to claim 1,
The reaction from the viewer includes at least any one of the operation of a device by the viewer, the voice and characters input by the viewer, an image of the viewer, the posture, line of sight and gesture of the viewer analyzed from the image, and measurement data measuring the physical reaction of the viewer;
A video distribution system characterized by the above.