JP6948934B2 - Content processing systems, terminals, and programs - Google Patents

Description

The present invention relates to content processing systems, terminal devices, and programs.

Technology for distributing video and audio of live performances (various events, stages, music, etc.) via communication lines (Internet, etc.) has become widespread. In the prior art, devices such as editing equipment, switchers, mixers, etc. are used for editing and processing live video and audio. Then, in the prior art, editing and processing are performed centrally in the first stage of the distribution process.

For example, Non-Patent Document 1 describes a system configuration when a broadcaster actually distributes images of all competitions and all events by live streaming for a large-scale sporting event via the Internet. According to this document, IPVandA's SD-quality video resources were transmitted from the center of the local city (Brazil) where the event was held to the broadcasting center in Tokyo using an international line. The SD image quality video is based on a bit rate of about 2.5 Mbps. Then, at the broadcasting center, the IPVandA video was recoded into a lower bit rate video and distributed via the Internet. In addition, for the video of some competitions, an audio booth for audio processing was constructed in the above broadcasting center, and the commentary and live commentary unique to online distribution were added and distributed.

Kenji Shimanishi, Mune Endo, Yuki Kokubo, Shinya Orishita, Shunichi Sakai, Aya Maeda, "Rio de Janeiro Olympic Digital Content Production", Broadcasting Technology, November 2016, p. 104-106.

However, if it is possible to add or replace content with a configuration that can be realized at a lower cost based on the content that has already been delivered live, it will be possible to generate a variety of content.

The present invention has been made based on the above-mentioned problem recognition, and intends to provide a content processing system, a terminal device, and a program capable of easily processing the distributed content with an inexpensive device configuration. To do.

[1] In order to solve the above problems, the content processing system according to one aspect of the present invention includes a manifest acquisition unit that acquires an original manifest file included in the original content of streaming using the hypertext transfer protocol, and the original content. Acquires a segment acquisition unit that acquires the original segment file included in the file, a decoder unit that decodes and outputs the original segment file, and additional content newly added in a form associated with the decoded original segment file. An interface unit, an encoder unit that encodes the additional content, a segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file, and the segmentation unit. Based on the original manifest file, a manifest generator that generates a processed manifest file so that the original segment file and the additional segment file are synchronized, the original segment file, the additional segment file, and the processed manifest file. As a processed content, a redistribution unit that distributes the file using the hypertext transfer protocol is provided.

[2] Further, in one aspect of the present invention, in the above content processing system, the manifest generation unit includes all of the acquired original segment files and reproduces additional processed contents including the additional segment files. It is characterized in that the processing manifest file for the purpose is generated.

[3] Further, in one aspect of the present invention, in the above content processing system, the manifest generation unit includes only a part of the acquired original segment file and is a replacement type including the additional segment file. It is characterized in that the processing manifest file for reproducing the processed content is generated.

[4] Further, one aspect of the present invention is a content processing system including a server device and a terminal device, and the terminal device acquires an original manifest file included in the original content of streaming using the hypertext transfer protocol. A new manifest acquisition unit, a segment acquisition unit that acquires the original segment file included in the original content, a decoder unit that decodes and outputs the original segment file, and a new form associated with the decoded original segment file. An additional segment file by segmenting the interface part that acquires the additional content added to the file, the encoder part that encodes the additional content, and the encoded additional content so as to synchronize with the time of the original segment file. The server device includes a segmentation unit that generates a processing manifest file based on the original manifest file so that the original segment file and the additional segment file are synchronized with each other. A content processing system including a redistribution unit that distributes the original segment file, the additional segment file, and the processing manifest file as processing content using a hypertext transfer protocol.

[5] Further, in one aspect of the present invention, in the above content processing system, the original segment file stores data obtained by encoding at least one of video and audio, and is output by the decoder unit. It further includes a content generation unit that automatically generates the additional content based on the original segment file by analyzing the video or audio to be generated.

[6] Further, in one aspect of the present invention, in the above content processing system, the content generation unit includes the text data obtained by performing a video or audio recognition process output by the decoder unit. It produces content.

[7] Further, in one aspect of the present invention, in the above content processing system, the manifest generation unit has a function of generating a processing manifest file for reproducing only the original segment file based on an instruction from the outside. The redistribution unit is provided with a function of distributing only the original segment file and the processing manifest file based on the instruction from the outside.

[8] Further, one aspect of the present invention is a manifest acquisition unit that acquires an original manifest file included in the original content of streaming using the hypertext transfer protocol, and a segment that acquires the original segment file included in the original content. Encodes the acquisition unit, the decoder unit that decodes and outputs the original segment file, the interface unit that acquires the additional content newly added in a form associated with the decoded original segment file, and the additional content. The terminal device includes an encoder unit and a segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.

[9] Further, in one aspect of the present invention, a computer is provided with a manifest acquisition unit that acquires an original manifest file included in the original content of streaming using the hypertext transfer protocol, and an original segment file included in the original content. A segment acquisition unit to be acquired, a decoder unit that decodes and outputs the original segment file, an interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and the additional content. Based on the encoder section that encodes the A manifest generator that generates a processing manifest file, the original segment file, the additional segment file, and the processing manifest file are used as processing contents so that the original segment file and the additional segment file are synchronized. It is a program for functioning as a content processing system including a redistribution unit that distributes using a hypertext transfer protocol.

[10] Further, in one aspect of the present invention, a computer is provided with a manifest acquisition unit that acquires an original manifest file included in the original content of streaming using the hypertext transfer protocol, and an original segment file included in the original content. A segment acquisition unit to be acquired, a decoder unit that decodes and outputs the original segment file, an interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and the additional content. To function as a terminal device including an encoder unit for encoding and a segmentation unit for generating an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file. It is a program for.

According to the embodiment, only the content to be added can be encoded and distributed, and the original content can be distributed as the original segment file. This makes it possible to process and redistribute streamed content with a small device configuration.

It is a block diagram which shows the schematic functional structure of the distribution system including the content processing system 1 by embodiment of this invention. It is a block diagram which shows the more detailed functional structure of the terminal apparatus 3 in the same embodiment. It is the schematic which shows the architecture of the processing by the content processing system 1 by the same embodiment. It is a schematic diagram which shows the structure of the segment file when the content processing system 1 by the same embodiment performs processing to add content. It is a schematic diagram which shows the structure of the segment file when the content processing system 1 by the same embodiment performs processing which replaces a content. FIG. 5 is a schematic diagram showing an example of an original manifest file distributed from the web server device 7 and received by the content processing system 1 in the same embodiment. It is the schematic which shows the example of the manifest file generated by the manifest generation part 22 of the server apparatus 2 by the same embodiment. It is a schematic diagram which shows the structural example of the processing content management information used for management by the content processing system 1 by the same embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a schematic functional configuration of a distribution system according to the present embodiment. In this figure, reference numeral 0 is a distribution system. As shown in the figure, the distribution system 0 includes a content processing system 1, an encoder device 6, a web server device 7, and a receiving terminal 8. The web server device 7 and the content processing system 1 are connected by the Internet 100, and communication between the two is possible. Further, the content processing system 1 and the receiving terminal 8 are also connected by the Internet 100, and communication between the two is possible. The content processing system 1, the encoder device 6, the web server device 7, and the receiving terminal 8 may each use dedicated devices or may be realized by using a computer.

The distribution system 0 is for distributing contents (video, audio, text, etc.) from the web server device 7 side to the receiving terminal 8.
The content processing system 1 receives the content distributed from the web server device 7, processes it, and redistributes the processed content. Here, the processing of the content is, for example, the addition of the content (video, audio, text, etc.) or the replacement of a part of the content.
The encoder device 6 is a device that encodes content such as video and audio.
The web server device 7 distributes the content encoded by the encoder device 6 in the HTTP live streaming format. For HTTP live streaming, for example, techniques such as HLS (HTTP Live Streaming) and MPEG-DASH (Dynamic Adaptive Streaming over HTTP) can be used. HLS and MPEG-DASH itself are existing technologies. In addition, "HTTP" means a hypertext transfer protocol.
The receiving terminal 8 receives and reproduces the content distributed from the content processing system 1.

As shown in FIG. 1, the content processing system 1 includes a server device 2 and a terminal device 3. The server device 2 and the terminal device 3 operate in cooperation with each other to appropriately process the content distributed from the web server device 7 side. The server device 2 may be composed of a plurality of computers. Further, the server device 2 may be a so-called cloud server. Further, although one terminal device 3 is shown in the figure, a plurality of terminal devices 3 may be included in the content processing system 1.

The server device 2 includes a manifest acquisition unit 21, a manifest generation unit 22, a segment acquisition unit 25, a segment selection unit 26, and a web server unit 28.
The terminal device 3 includes a decoder unit 41, a reproduction unit 42, an encoder unit 48, and a segmentation unit 49. A more detailed configuration of the terminal device 3 will be described later with reference to FIG.
Each functional unit listed here is realized by using, for example, an electronic circuit. Further, each functional unit may be provided with a storage means such as a semiconductor memory or a magnetic hard disk device, if necessary. In addition, each function may be realized by a computer and software.

In the configuration shown in FIG. 1, the terminal device 3 receives the content distributed from the web server device 7 side directly from the Internet 100, but the terminal device 3 receives the content from the server device 2. May be good.

Here, the configuration of the content distributed by the content processing system 1 in the present embodiment will be described.
Content consists of one or more materials.
The material is video, audio, text, and so on. Here, the text as the material includes, for example, text displayed on the screen of the receiving device such as subtitle text, and text processed by the receiving program. The text is, for example, plain text, data in XML format, or the like.
The video and audio are appropriately encoded.
Materials such as video, audio, and text are appropriately segmented. A segment is a fragment of a content material cut for a predetermined time length. The length of the segment is typically on the order of a few seconds. Content is distributed as segment-based files, and is stored and managed as needed. A file for one segment may be called a segment file. Each segment of content has a start time (presentation start time) and a time length associated with it. The start time is expressed as an absolute time or a relative time. Note that each segment may be associated with some timing information instead of the start time. This timing information is, for example, clock reference information (also referred to as “time stamp”) in the system on the side of sending the content. Such timing information does not have to exactly match the time at the time of distribution or reproduction. However, the relative time relationship between consecutive segments can be grasped from such timing information. In the following, it will be referred to as the "start time" of the segment, including the case where this kind of timing information is used.
The segment is also called a fragment, a chunk, a fragment, or the like.
If the content consists of multiple materials, the multiple materials are synchronized by the start time information associated with each segment.
Content consists of one or more materials. That is, the content may include a plurality of materials at one time point. For example, some content may include video material and audio material at some point in time. Further, a certain content may include a plurality of audio materials or a plurality of video materials at a certain point in time. Some content may contain any number of materials at any given time.

The data for indexing a plurality of materials and a plurality of segments in one content is a manifest. Manifest data is kept in the manifest file. The manifest data is managed by associating the start time of the segment, the type of material, and the location information of the segment file having the content data of the segment. The location information of a file is a file name, a URI (Uniform Resource Identifier), or similar information. In other words, the manifest data indicates from when (start time) how many seconds (time length), what kind of material (video, audio, etc.) should be read from which file and presented. That is, the manifest file contains information about the playback procedure of the delivered content. Specifically, the manifest file in HLS is an m3u8 file. The manifest file in MPEG-DASH is an MPD file. The content playback device (or playback program) reads the segment file from a predetermined location and presents it at an appropriate timing by referring to the data in the manifest. Here, the presentation means displaying a video on a display device or outputting audio from a speaker or the like.

That is, the web server device 7 distributes the segment file including the content by using HLS or MPEG-DASH. In addition, the web server device 7 distributes a manifest file that describes a procedure for reproducing the segment file and the like. These files sent from the web server device 7 side are called "original" for convenience. On the terminal device 3 side, as will be described later, a segment file synchronized with the time information (start time (also referred to as “start time”) and time length (also referred to as “duration”)) of the original segment file is generated. NS. The generated segment file is, for example, a file that stores contents such as video, audio, video plus audio, and text. The file generated on the terminal device 3 side is called "addition" for convenience.

On the server device 2 side, an additional segment file is added to the original segment file to create new content. Alternatively, the server device 2 may replace at least a part of the original segment file with an additional segment file to provide new content. Further, the server device 2 may add an additional segment file and replace at least a part of the original segment file with the additional segment file. Note that "replacement" is equivalent to adding an additional segment file and deleting at least a part of the original segment file (that is, not flowing to the downstream side). One of the features of this embodiment is that in any of the above cases, the new additional content generated in the content processing system 1 is added to the original content distributed from the web server device 7 side. Specifically, the segment selection unit 26 selects a segment to be redistributed toward the receiving terminal 8 from the segment acquired by the segment acquisition unit 25 and the segment passed from the terminal device 3 side.

In the server device 2, an additional segment file is added as described above. However, for example, in response to a request from the receiving terminal 8 on the end user side, a form in which an additional segment file is not added may be selected. That is, it may be possible to choose not to add or replace the segment file. This is called "pass-through" for convenience.
When instructed to pass through the content, the manifest generation unit 22 generates a processed manifest file for reproducing only the original segment file. Further, in that case, the web server unit 28 delivers only the original segment file and the processing manifest file.

Further, the manifest generation unit 22 of the server device 2 generates a manifest file according to the segment file selected by the segment selection unit 26.

As a result, the content processing system 1 can perform processing such as adding or replacing contents. In addition, the content processing system 1 can also select an operation of passing through and redistributing the original content. As a result, the processed content can be received and viewed on the receiving terminal 8 side. Even if the content is processed by the content processing system 1, the receiving terminal 8 can dynamically replace and add the content such as video and audio without being aware of the stream switching and the like.

Next, the functions of each functional unit of the server device 2 will be described.
The manifest acquisition unit 21 acquires the manifest file transmitted from the web server device 7. That is, the manifest acquisition unit 21 acquires the original manifest file included in the original content of the streaming using the hypertext transfer protocol.
The manifest generation unit 22 generates a new manifest file based on the manifest file acquired by the manifest acquisition unit 21 and the segment file selected by the segment selection unit 26, and passes the new manifest file to the web server unit 28. The manifest file generated by the manifest generation unit 22 corresponds to the processing content performed by the content processing system 1. That is, the manifest generation unit 22 synchronizes the original segment file (segment file of the original content) and the additional segment file (segment file of the additional content) based on the original manifest file, so that the processed manifest file (processed content segment file) is synchronized. Manifest file) is generated.

The manifest generation unit 22 can generate a processing manifest file that includes all of the acquired original segment files and that reproduces the additional processing content including the additional segment files. This is for adding content. In addition, the manifest generation unit 22 can generate a processing manifest file for reproducing the replacement type processed content including only a part of the acquired original segment file and including the additional segment file. This is for content replacement.

The segment acquisition unit 25 acquires the segment file transmitted from the web server device 7.
The segment selection unit 26 selects a segment file to be distributed from the segment file acquired by the segment acquisition unit 25 and the segment file passed from the terminal device 3. The segment selection unit 26 passes the segment file selected as the distribution target to the web server unit 28.

The web server unit 28 distributes the segment file passed from the segment selection unit 26 and the manifest file generated by the manifest generation unit 22 via the Internet 100. That is, the web server unit 28 distributes the original segment file, the additional segment file, and the processing manifest file as processed contents by using the hypertext transfer protocol. The web server unit 28 redistributes the content by using, for example, a method such as HLS or MPEG-DASH. The web server unit 28 is also referred to as a "redistribution unit".

Since each part inside the terminal device 3 will be described with reference to FIG. 2, description thereof will be omitted here.

FIG. 2 is a block diagram showing a more detailed functional configuration of the terminal device 3. As shown in the figure, the terminal device 3 includes a manifest acquisition unit 31, a manifest analysis unit 32, a segment acquisition unit 33, a time analysis unit 35, a decoder unit 41, a playback unit 42, a content generation unit 43, and the like. It includes an A / V interface unit 44, a mixer unit 45, an encoder unit 48, a segmentation unit 49, and an upload unit 50.

The manifest acquisition unit 31 acquires the manifest file transmitted from the web server device 7. That is, the manifest acquisition unit 31 acquires the original manifest file included in the original content of the streaming using the hypertext transfer protocol. The manifest acquisition unit 31 passes the acquired manifest file to the manifest analysis unit 32.
The manifest analysis unit 32 analyzes the manifest file passed from the manifest acquisition unit 31. That is, the manifest analysis unit 32 analyzes the structure of the content distributed from the web server device 7. Specifically, the manifest analysis unit 32 extracts the location information and the start time of the segment file to be acquired from the manifest file. The manifest analysis unit 32 passes the segment access information to the segment acquisition unit 33 based on the analysis result. Specifically, the manifest analysis unit 32 passes the location information and the start time information of the segment file to be acquired to the segment acquisition unit 33. Further, the manifest analysis unit 32 may extract the time length information of the segment file and pass it to the segment acquisition unit 33. In addition, the manifest analysis unit 32 extracts information on the configuration of the segment file and time information on each segment file from the manifest file. The manifest analysis unit 32 passes the extracted information (time information of each segment file, etc.) to the segmentation unit 49.

The segment acquisition unit 33 receives information regarding the segment file to be acquired from the manifest analysis unit 32. Then, the segment acquisition unit 33 acquires the segment file transmitted from the web server device 7 based on the information received from the manifest analysis unit 32. That is, the segment acquisition unit 33 acquires the original segment file included in the original content. The segment acquisition unit 33 passes the acquired segment file to the decoder unit 41. Further, the segment acquisition unit 33 passes information regarding at least the time of the acquired segment file to the time analysis unit 35.

The time analysis unit 35 receives information about the time of the segment file or the segment file from the segment acquisition unit 33. Then, the time analysis unit 35 analyzes the time information for each segment file. The time analysis unit 35 outputs at least information on the start time and time length for each segment file. That is, the time analysis unit 35 passes information on the start time and time length of each segment to the segmentation unit 49.

The decoder unit 41 decodes the segment file acquired by the segment acquisition unit 33. That is, the decoder unit 41 decodes the original segment file and outputs it. Specifically, the decoder unit 41 decodes the video and audio data stored in the segment file. In addition, the decoder unit 41 extracts other data (text data, etc.) stored in the segment file. The decoder unit 41 passes the decoded data to the reproduction unit 42, the content generation unit 43, and the A / V interface unit 44.

The reproduction unit 42 reproduces the video and audio decoded by the decoder unit 41 based on the designated time information. The playback unit 42 displays the video on the display device and outputs the sound from the speaker or the like. Further, the reproduction unit 42 may appropriately process data other than the video and audio passed from the decoder unit 41 by a predetermined method. As an example, the reproduction unit 42 displays the text data passed from the decoder unit 41 on the screen at a specified timing and by a specified method. This text data is, for example, timed text, and more specifically, superimpose or subtitle data. Further, the reproduction unit 42 may display the image data passed from the decoder unit 41 on the screen. Further, the reproduction unit 42 may output the synthetic voice so as to read out the text data passed from the decoder unit 41. Further, the reproduction unit 42 may give other data passed from the decoder unit 41 as an input to the program running on the reproduction unit 42. Further, the reproduction unit 42 may run the program passed from the decoder unit 41 on the reproduction unit 42. When executing a program, the reproduction unit 42 includes a program execution environment. An example of a program execution environment is a Javascript interpreter, but the description language or form of the program is not limited to this.

The content generation unit 43 performs processing based on the data as a result of decoding by the decoder unit 41. Then, as a result of the processing, the content generation unit 43 generates content (or content material) different from the segment file acquired by the segment acquisition unit 33.
For example, the content generation unit 43 can automatically generate additional content based on the original segment file by analyzing the video or audio output by the decoder unit 41.
In addition, the content generation unit 43 can generate the additional content including the text data obtained by performing the video or audio recognition process output by the decoder unit.
Further, the content generation unit 43, for example, performs voice recognition of the utterance sentence included in the voice decoded by the decoder unit 41, and outputs the text data transcribed from the sentence.
Further, the content generation unit 43 performs language translation processing of the utterance sentence included in the voice decoded by the decoder unit 41, and outputs the translated sentence as text or voice.
Further, the content generation unit 43 generates, for example, a response to the utterance sentence included in the voice decoded by the decoder unit 41 by using artificial intelligence or the like, and outputs the generated response sentence as a text or a voice.
Further, the content generation unit 43 performs, for example, a Fourier analysis process of the voice decoded by the decoder unit 41, and outputs the data of the result of the Fourier analysis.
Further, the content generation unit 43 performs recognition processing (image recognition, character recognition, etc.) based on the video decoded by the decoder unit 41, and outputs the result of the recognition processing.
Further, the content generation unit 43 performs various image processing on the video decoded by the decoder unit 41, for example, and outputs the result of the image processing.
Further, the content generation unit 43 performs recognition processing regarding the contents of the video and audio decoded by the decoder unit 41, and when a specific scene is detected in the video and audio, a sound effect or a specific video / audio is detected. Output the image.
Further, the content generation unit 43 may perform various processes based on the contents passed from the decoder unit 41 in addition to the processes illustrated above to generate new contents.
The processing itself such as voice recognition processing, language translation processing, response processing by artificial intelligence, Fourier transform processing, recognition processing, and image processing included in the processing by the content generation unit 43 exemplified above can be realized by existing technology. It is a thing.

The A / V interface unit 44 receives the content material decoded by the decoder unit 41. Further, the A / V interface unit 44 passes the content material passed from the decoder unit 41 to the mixer unit 45. Further, the A / V interface unit 44 receives a new content material from the mixer unit 45 and passes it to the encoder unit 48. That is, the A / V interface unit 44 acquires the newly added additional content in a form associated with the decoded original segment file. The A / V interface unit 44 may pass at least a part of the content material passed from the decoder unit 41 to the encoder unit 48.
The A / V interface unit 44 is also simply referred to as an "interface unit".

The mixer unit 45 appropriately mixes and outputs the content passed from the A / V interface unit 44 and the content input from the outside. The content output by the mixer unit 45 is passed to the encoder unit 48 via the A / V interface unit 44. The process of mixing the mixer unit 45 is, for example, mixing audio and audio, mixing video and video, and the like. When the video content material contains audio inside, the mixer unit 45 may perform a process of mixing the video and audio. The mixing ratio when the mixer unit 45 mixes can be arbitrarily set. Further, the mixer unit 45 may independently pass the content passed from the content generation unit 43, the video or audio input from the outside, or the like to the A / V interface unit 44 side.

That is, the decoder unit 41 decodes the original content, and the mixer unit 45 acquires the new content at the timing when the reproduction unit 42 reproduces the original content. Alternatively, the mixer unit 45 acquires new content at the timing when the decoder unit 41 decodes the original content and the content generation unit 43 processes the original content (analysis processing or the like).
The new content is content for addition or content for replacement.

The encoder unit 48 encodes the content material passed from the A / V interface unit 44 and passes it to the segmentation unit 49. That is, the encoder unit 48 encodes the additional content. That is, the encoder unit 48 re-encodes and outputs the content (video, audio, video plus audio, etc.) added on the terminal device 3 side. The parameters when the encoder unit 48 performs the encoding process are dynamically set according to the original video / audio distributed from the web server device 7 side.

The segmentation unit 49 segments new content input by the mixer unit 45 so as to synchronize with the segment file acquired by the segment acquisition unit 33. That is, the original content and the newly added content are synchronized. Specifically, the segmentation unit 49 appropriately divides the output from the encoder unit 48 and segments it according to the time information passed from the time analysis unit 35. That is, the segmentation unit 49 generates the additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file. Then, the segmentation unit 49 passes the generated segment file to the upload unit 50.
The upload unit 50 uploads the segment file passed from the segmentation unit 49 to the server device 2.

By the above processing, the segmentation unit 49 makes the segment start time and the segment time length the same between the original segment file and the additional segment file. Therefore, the segmentation unit 49 acquires the information of the head time stamp of the original segment file first received by the time analysis unit 35. Then, the segmentation unit 49 calculates the start time of the first segment and the subsequent segments based on the predetermined segment time length, starting from the start time of the segment acquired from the time analysis unit 35. The segmentation unit 49 determines the time stamp information of the additional segment file to be generated by using the start time of each segment calculated in this way. As a result, the contents can be easily integrated on the server device 2 side.

As described above, in the terminal device 3, the content (original content) reproduced by the reproduction unit 42 and the content to be added (video, audio, etc.) are mixed by the mixer unit 45. As a result, there is almost no video / audio synchronization shift between the original content and the mixed content.
For example, when the terminal device 3 acquires the original content including video and audio and adds additional audio content to the content, the following three contents are redistributed from the server device 2. That is, there are three contents: the original sound which is the sound included in the original content, the added sound which is the sound added on the terminal side, and the original video included in the original content. There is no synchronization shift between these three contents. Therefore, for example, when the above-mentioned original video and the original audio are reproduced, or when the above-mentioned original video and the additional audio are reproduced, there is no synchronization shift between the video and the audio.

FIG. 3 is a schematic diagram showing a processing architecture by the content processing system 1. The cloud processing 202 shown in FIG. 3 corresponds to the processing by the server device 2 in FIG. That is, the architecture shown in FIG. 3 is for using a so-called cloud server as the server device 2 in FIG. Further, the terminal process 203 shown in FIG. 3 corresponds to the process by the terminal device 3 in FIG. The stream 200 is an original stream delivered from the web server device 7 side.

The cloud process 202 includes a process of adding content and a process of replacing content. The content for addition and replacement is generated on the terminal processing 203 side. The terminal processing 203 refers to the stream 200, and based on the stream 200, generates the material of the content for addition or replacement and provides it to the cloud processing 202 side. The control 201 controls which material is added and which material is replaced with which material. That is, the control 201 selects the segment file of the material and controls to generate the manifest file that matches the selected segment. In addition, the segment file for addition and the segment file for replacement are controlled to be synchronized with the original segment file. That is, the additional content material and the replacement content material are segmented so as to be consistent with the original segment file. Then, time information (start time, time length) that synchronizes with the original segment file is added to the additional segment file and the replacement segment file. Time information given in consideration of synchronization is written in the manifest file. That is, on the receiving side, by referring to the manifest file, the original content and the added or replaced content are played back in synchronization.

The selection 210 is a process of selecting what kind of stream output from the cloud process 202 is to be viewed on the receiving side.
The stream 211 is a stream for adding content. That is, the stream 211 is a stream that further includes the additional content generated in the terminal processing 203 while maintaining the content included in the original stream 200.
The stream 212 is a content replacement stream. That is, the stream 212 is a stream in which at least a part of the contents included in the original stream 200 is replaced with the additional contents generated in the terminal processing 203.
Stream 213 is a pass-through stream. That is, the stream 213 redistributes the stream 200 as it is without adding content to the original stream 200 or replacing the content of the stream 200.

Next, a specific example of the segment file when the content is added or replaced will be described.

FIG. 4 is a schematic diagram showing the structure of a segment file when the content processing system 1 adds content. In the figure, the horizontal direction is the time axis. Further, the figure includes segment files having each of the times t1, t2, ..., T7 as the start time. The description is omitted after the time t8. The illustrated C11, C12, ..., C17 are a series of segment files of a specific material (for example, video or audio) included in the content. Further, C21, C22, ..., C27 are a series of segment files of other materials (for example, video or audio) included in the content. These two series, that is, the series of C11, C12, ..., C17 and the series of C21, C22, ..., C27 are included in the original content. That is, the segment files belonging to these two series are shown by solid rectangles. On the other hand, C35 whose start time is time t5 and C36 and C37 following it are contents added by the content processing system 1. That is, C35, C36, and C37 are not included in the original content. The segment files of the contents added by the content processing system 1 are shown by broken lines. As shown, the segment files C35, C36, and C37 of the added content are synchronized with the segment files C15, C16, C17 and C25, C26, and C27 included in the original content, respectively. That is, the added content is divided so as to be synchronized with the segment file of the original content, and the time information is added.

As shown in FIG. 4, the series of the added contents (segment files C35, C36, C37) is segmented so as to be synchronized with the segment file of the original contents. That is, the start time of the segment file C35 is the same as the start time of the segment files C15 and C25. Further, the time length of the segment file C35 may be the same as the time length of the segment files C15 and C25. The same applies to the subsequent segment files. Further, in the manifest file (playlist file), the time information is described so that each segment file is synchronized. The segmentation unit 49 and the manifest generation unit 22 control the time information of the output file so that the segment files are synchronized between the series as described above.

FIG. 5 is a schematic diagram showing the structure of a segment file when the content processing system 1 replaces the content. Also in this figure, the horizontal direction is the time axis. Further, as in the case of FIG. 4, the illustrated segment files C11, C12, ..., C17 belong to a specific material included in the content. Then, in the case of this figure, the sequence of the segment files C21, C22, ..., C27 is replaced with the segment files C45, C46, C47 of another content from the middle.

As shown in FIG. 5, the series of the contents to be replaced (segment files C45, C46, C47) is segmented so as to be synchronized with the segment file of the original contents. That is, the start time of the segment file C45 is the same as the start time of the segment file C15. Further, the time length of the segment file C45 may be the same as the time length of the segment file C15. The same applies to the subsequent segment files. Further, in the manifest file (playlist file), the time information is described so that each segment file is synchronized. The segmentation unit 49 and the manifest generation unit 22 control the time information of the output file so that the segment files are synchronized between the series as described above.

Next, an example of a manifest file will be described. As described above, the content processing system 1 processes the original content received from the web server device 7 side and redistributes it as the processed content. At this time, the content processing system 1 receives the manifest file of the original content. In addition, the manifest generation unit 22 in the content processing system 1 generates a manifest file for the processed content to be redistributed. 6 and 7 show a manifest file of the original content and a manifest file of the processed content, respectively. The manifest files shown in FIGS. 6 and 7 are also referred to as "master playlists". The playlist file defined in the master playlist file contains information about the segment file that should actually be played.

FIG. 6 is a schematic diagram showing an example of an original manifest file distributed from the web server device 7 and received by the content processing system 1. The illustrated manifest file is an M3U8 file (M3U file). In the figure, each line of the file is given a line number for reference. The contents of the original manifest file are as follows.
The first line "# EXTM3U" is the header of the file, indicating that this file is an extended M3U file.
The second line "# EXT-X-VERSION: 3" indicates that the version number of the manifest file is "3".
The third line "# EXT-X-INDEPENDENT-SEGMENTS" indicates that the content in one segment in all playlists referenced from the master playlist is independent of the information in other segments. It is a tag. That is, the content in one segment can be decrypted without the information in the other segment.
The 4th to 17th lines are information defining seven streams. The pair of lines 4 and 5 defines the first stream. The pair of lines 6 and 7 defines the second stream. The pair of 8th and 9th lines defines the third stream. The pair of lines 10 and 11 defines the fourth stream. The pair of lines 12 and 13 defines the fifth stream. The pair of lines 14 and 15 defines the sixth stream. The pair of lines 16 and 17 defines the seventh stream. The information of each stream is composed of the "# EXT-X-STREAM-INF" tag and the information of the file name (m3u8 file name) of the playlist file. The "# EXT-X-STREAM-INF" tag is a parameter of "BANDWIDTH", "AVERAGE-BANDWIDTH", "CODECS", "RESOLUTION", "FRAME-RATE", and "CLOSED-CAPTION". have. "BANDWIDTH" represents bandwidth. "AVERAGE-BANDWIDTH" represents the average bandwidth. "CODECS" represents coding and decoding information. "RESOLUTION" represents the resolution of the image. "FRAME-RATE" represents a frame rate (number of frames per unit time). "CLOSED-CAPTION" indicates the presence or absence of closed captions.
The m3u8 file of each stream is described in the manifest file as relative position information, for example, "test2_270.m3u8" on the fifth line.

FIG. 7 is a schematic view showing an example of a manifest file generated by the manifest generation unit 22 of the server device 2. The manifest generation unit 22 generates the manifest file of FIG. 7 based on the manifest file of FIG. 6 received by the server device 2. The illustrated manifest file is an M3U8 file (M3U file). In the figure, each line of the file is given a line number for reference. The contents of the manifest file of the processed contents generated by the contents processing system 1 are as follows.
The first line "# EXTM3U" (header) and the second line "# EXT-X-VERSION: 3" (version information) are the same as those described in FIG.

The third and fourth lines are information added by the manifest generation unit 22, and define an audio (audio) m3u8 file. The "# EXT-X-MEDIA" tags on the third and fourth lines associate two mutually substitutable media. The "# EXT-X-MEDIA" tags on the 3rd and 4th lines have the group IDs "TYPE = AUDIO" (type is voice) and "GROUP-ID =" audio ""("audio"). Has a common description.
However, while the "# EXT-X-MEDIA" tag on the third line has the description "NAME = mixed" (mixed voice), the "# EXT-X-MEDIA" tag on the fourth line has a description. It has a description of "NAME = original" (original voice). As described above, the names of the audio media are different between the third line and the fourth line. Further, the third line has the description "DEFAULT = YES" (default voice), while the fourth line has the description "DEFAULT = NO" (not the default voice). The URL of the playlist file specified by the "# EXT-X-MEDIA" tag on the third line is "mixed / playlist.m3u8". On the other hand, the URL of the playlist file specified by the "# EXT-X-MEDIA" tag on the fourth line is "original / playlist.m3u8".

As described above, the manifest generation unit 22 on the server device 2 side generates a manifest file according to the content for addition or replacement (series of segment files) generated by the terminal device 3. Specifically, in the manifest file generated by the manifest generation unit 22, two types of substitutable audio contents are described in the third line and the fourth line. They also have different names and specify different playlist URLs. Also, the voice specified in the third line is the default (that is, implicitly selected), whereas the voice specified in the fourth line is not the default.

Further, as shown in FIG. 7, the manifest generation unit 22 sets the group ID (GROUP-ID = "audio") of the audio content defined in the third and fourth lines from the sixth line to the third line. Rewrite so that the 19th line is referenced. That is, the sixth to 19th lines of FIG. 7 are descriptions corresponding to the fourth to 17th lines of FIG. With respect to the seven streams described in the manifest file of FIG. 7, the manifest generation unit 22 adds the parameter specification "AUDIO =" audio "" to the "# EXT-X-STREAM-INF" tag. The "audio" written in this lowercase letter is the group ID defined in the third and fourth lines. That is, the seven streams described in FIG. 7 are inherited from the file of FIG. 6, and the manifest generation unit 22 associates a group called "audio" with each of these streams.

Due to this manifest file generated by the manifest generation unit 22, not only the original audio (playlist URL is original / playlist.m3u8) but also processed mixed audio (playlist URL is mixed) on the receiving terminal 8 side. /Playlist.m3u8) can be reproduced.
The mixed audio (mixed) is, for example, the audio from the original distribution source (for example, the event venue audio) and the audio input by the content processing system 1 (for example, Japanese commentary, etc.) in the live video of a sports event. ) And is mixed.

FIG. 8 is a schematic view showing a configuration example of processed content management information used by the content processing system 1 for management. When the control unit (not shown) of the terminal device 3 processes the content, the processed content management information shown in FIG. 8 is generated and passed to the server device 2. The control unit (not shown) of the server device 2 receives and stores the processed content management information from the terminal device 3. One terminal device 3 generates one or a plurality of processed contents for one original content, and passes the segment file to the server device 2. At that time, the terminal device 3 generates the processed content management information of FIG. 8 for each processed content. Further, a plurality of terminal devices 3 may generate processed contents for one original content, and pass the segment file to the server device 2. At that time, each terminal device 3 that generates the processed content generates the processed content management information of FIG. 8 for each processed content.

As shown in the figure, the processed content management information includes an original content ID, a processed content ID, a replacement series list, an additional series list, and a processor ID (FIG. (A)). Here, "ID" means identification information (identifier).

The original content ID is information for uniquely identifying the original content to be processed. In the illustrated example, an 8-digit numerical value is used as the original content ID, but the data format is not particularly limited.
The processed content ID is information for uniquely identifying the processed content generated by the content processing system 1. In the illustrated example, the original content ID and the branch number (four-digit numerical value) are used as the processed content ID, but the data format is not particularly limited. The processed content ID may be assigned on the terminal device 3 side or on the server device 2 side.
The replaced series list is information on the series of segment files that existed in the original content but were replaced by this processed content. The number of series included in the replaced series list is arbitrary. In the illustrated example, the substituted sequence list (FIG. (B)) has two sequences. This means that two of the plurality of series included in the original content are replaced series. The replaced series list is tabular data, and has each item of series ID, series type, start time, and end time. The series ID is information for uniquely identifying the series of the segment fill. The series type is information representing the type of series (for example, video, audio, video plus audio, subtitles, etc.). The start time is information representing the time when the replacement of the replaced series is started in the form of a serial number of a year, month, day, hour, minute, second, and less than a second. The end time is information representing the time when the replacement of the replaced series is completed in the same format as the start time.
The additional series list is information on the series of segment files that do not exist in the original content and are added in this processed content. The number of series included in the additional series list is arbitrary. In the illustrated example, the additional sequence list (FIG. C) has two sequences. This means that two series are added in the processed content. The additional series list is tabular data, and has each item of series ID, series type, start time, and end time, like the replaced series list. The start time represents the time when the addition of the additional series is started. The end time represents the time when the addition of the additional series ends.
The processor ID is information for uniquely identifying a person (user, business operator, etc.) who generates the processed content.

The series included in the above-mentioned additional series list may be a series that replaces the replaced series or a series that is simply added to the original content.

It should be noted that at least a part of the functions of the server device 2 and the terminal device 3 in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. Say that. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Next, an example of a service that can be realized by using the content processing system 1 will be described.

First service example: Video synchronization sharing application For example, video and audio of a sports competition are distributed as original contents from the web server device 7. A user who is watching the sports competition at a stadium or the like launches an application program for content processing (hereinafter, may be referred to as an "application") on a terminal device 3 (for example, a smartphone or the like). By the user's operation, the terminal device 3 can shoot a moving image in the stadium and upload it to the server device 2 as additional content. In addition, the terminal device 3 can upload the comment text to the server device 2 by a user input operation or the like. The processed content including the additional content is distributed from the content processing system 1. The processed content is received by the terminal device 3 of another user, and another content (video, audio, text, etc.) can be added. Such content may be added in multiple stages. In addition, the processed content can be processed / edited by a content distribution company or the like and distributed. That is, it is possible for a large number of users to add content to the original content and enjoy the content in multiple ways.

Second service example: Holy fire relay video distribution service For example, in connection with a sporting event, the pattern of the holy fire relay and related video and audio are distributed from the web server device 7 as original content of the video and audio of the sports competition. Will be done. For example, a small camera capable of shooting 360 degrees is attached to the torch torch carried by the torch relay. In addition, a small terminal device 3 is attached to the torch torch. The terminal device 3 attached to the torch torch uploads a 360-degree moving image of the torch viewpoint to the server device 2 as additional content. In addition, a general user who supports the torch relay along the road takes a picture of the relay pattern with the terminal device 3 (for example, a smartphone or the like) and uploads it to the server device 2 as additional content. The server device redistributes the video content including the additional content from the web server unit 28.
This makes it possible to enjoy content using images from the torch viewpoint and images taken by general users. Anyone who supports the torch relay along the road can participate as a content creator.

Third service example: Generation and distribution of VR video For example, the video is distributed from the web server device 7 as original content. A single or a plurality of users who are close to the video shooting site use the terminal device 3 (for example, a smartphone or the like) to shoot the objects reflected in the original video from different angles. The terminal device 3 of each user uploads the video projected from each viewpoint to the server device 2 as additional content. For example, the same object is photographed by three or more terminal devices 3 from different angles. The video showing the same object from these plurality of viewpoints can be redistributed from the web server unit 28 as a VR (virtual reality) video.

Fourth service example: Addition of metadata related to the face recognition function For example, video and audio of a sports competition are distributed as original contents from the web server device 7. The user who is a viewer registers a favorite player in advance in the terminal device 3 (for example, a smartphone or the like). The terminal device 3 stores in advance the face image of the registered favorite player or the feature amount data of the face image. By executing the face recognition function, the terminal device 3 detects a scene in which a favorite player appears in the video of the original content. When a scene in which a favorite player appears is detected, the terminal device 3 notifies the user, for example, adds a chapter mark on the timeline, and automatically edits only the scene in which the player appears. Generate and display highlight videos. As a result, it is possible to enjoy a scene of a specific baseball player's turn at bat, a scene of a specific golf player's shot, a scene of a specific soccer player playing, and the like in a video.
The terminal device 3 can further add information to such contents. For example, the terminal device 3 can add (correct) the appearance scene of a favorite player that could not be detected by automatic face recognition by the user's operation. The terminal device 3 adds this additional information (metadata), for example, as text content. In addition, the terminal device 3 can give a scene name to each of the scenes extracted by face recognition by the user's operation. The terminal device 3 adds this additional information (text representing the scene name) as content. In addition, the terminal device 3 can add a "like" attribute to each of the scenes extracted by face recognition. The terminal device 3 adds this additional information (metadata representing "like"), for example, as text content. As described above, the terminal device 3 has a metadata modification function. The content processing system 1 can collect these metadata from the terminal device 3 and redistribute the processed content to which the metadata is added. In addition, the creator of the original content can also capture the collected metadata in real time and reflect it in the original content.

Fifth service example: Shift-synchronized video generation For example, in speed skating, skiing, athletics, swimming, etc., time is competed, but the timing at which the athlete starts may be different. The web server device 7 distributes the video of the athlete starting at that time as original content. On the other hand, the terminal device 3 receives in advance the video of the athlete who has already started, and records and saves the video. Then, the terminal device 3 synchronizes with the competition video currently distributed from the web server device 7, and adds the video recorded and saved by itself. At this time, control is performed so that the start timings of a plurality of competitors are synchronized. The server device 2 can simultaneously deliver the video (real-time video) delivered from the web server device 7 and the video (time-shift recorded video) added by the terminal device 3. As a result, on the receiving terminal 8 side, it is possible to superimpose and watch the athletes having different start timings in time.

Sixth service example: Road and public space watching application For example, a video is distributed from the web server device 7 as original content. On the other hand, the terminal device 3 is installed in advance on a road (for example, a school road) or at a plurality of places such as a public space such as a park. Each terminal device 3 receives the original content from the web server device 7, captures an image of the place where the terminal device 3 is installed (road, park, etc.), and provides the image as additional content to the server device 2. The web server unit 28 of the server device 2 redistributes the video provided by the terminal device 3 including the video as additional content. On the receiving terminal 8 side, images from a plurality of terminal devices 3 can be synchronized and viewed at the same time.

Seventh service example: Automatically generated commemorative photo, etc. Original content is distributed from the web server device 7. For example, a plurality of terminal devices 3 are installed in tourist spots and the like. Each terminal device 3 photographs the installed location and provides the server device 2 with a high-quality image (or image) as additional content. The server device 2 distributes the content including the additional content. Further, the terminal device 3 searches for a video (or image) using a specific place (sightseeing spot or the like) and time as a key. The server device 2 delivers the searched video (or image). Here, for example, an on-demand distribution mechanism may be used. The terminal device 3 can create a commemorative image (or a commemorative photo) by synthesizing the distributed image (or image) and the image (or image) taken by the photographing means owned by the terminal device 3. The terminal device 3 provides the video (or image) obtained by the compositing process to the server device 2 as additional content. The server device 2 further distributes processed content including the added content.

Eighth service example: Replace row insertion For example, a video of a sports competition or the like is distributed as original content from the web server device 7. The content generation unit 43 of the terminal device 3 generates a simple slow video by thinning out the frames of the received video. The terminal device 3 inserts the generated slow video into the live stream and provides it to the server device 2. The server device 2 redistributes the video in which the slow video is inserted.
Further, the terminal device 3 receives the re-distributed video (with slow video) and generates the text of the live comment by the user's input operation or the like. The terminal device 3 provides the text content of this live comment to the server device 2. The server device 2 redistributes the processed content to which the passed text content is added.

Ninth service example: Virtual camera switching For example, a video of a sports broadcast, a live music broadcast, or a stage (drama, etc.) broadcast is distributed from the web server device 7 as original content. The video delivered as the original content is a video shot with a high resolution of, for example, 8K (width 7680 pixels x height 4320 pixels). The video delivered as the original content is typically a video shot using a fixed camera and a wide-angle lens. The terminal device 3 generates a video stream obtained by cropping a part of the original content. That is, the cropping image has a smaller number of pixels than the original image. Further, the terminal device 3 may perform switching between a plurality of cropping images. That is, virtual camera switching is realized. The content generation unit 43 in the terminal device 3 generates the video stream by the cropping and switching. The server device 2 redistributes the video generated by the terminal device 3.
The content generation unit 43 may automatically select the area to be cropped or automatically perform virtual switching of the camera. For automatic cropping and automatic virtual switching, for example, AI (artificial intelligence) technology is used. Specifically, the learning process is performed by assigning an evaluation value to the video obtained as a result of cropping or switching in advance. Then, cropping and virtual switching are performed by the learned AI.
Further, the terminal device 3 may receive the re-distributed cropping video. Then, for example, the terminal device 3 acquires the text data of the live comment by the operation of the user or the like. The terminal device 3 provides the acquired text data as additional content to the server device 2. The server device 2 also redistributes the text data of the live comment provided by the terminal device 3 as content.

Example of 10th service: Insertion of clip video by virtual switching For example, a press conference, a broadcast from a news site, or a video of a sports broadcast is delivered from the web server device 7 as original content. While playing back the original content, the terminal device 3 inserts a clip video stored in advance based on a user's operation or the like as a replacement image to generate new video content. At this time, for example, the sound included in the original content is not replaced, and the sound is retransmitted (pass-through) as it is. The clip video to be inserted is, for example, a VTR video related to the press conference, news, or sports broadcast. The terminal device 3 provides the generated new content to the server device 2. The server device 2 redistributes new video content accompanied by the replacement video.
As a result, at the site (press conference, news broadcast, sports broadcast, etc.), it is only necessary to perform live distribution of camera images. In other words, the only equipment required in the field is a camera and an encoder device for small live distribution. Then, the terminal device 3 can perform an operation of inserting a necessary video clip. On the content processing system 1 side, it is possible to replace and insert the video without decoding or re-encoding the live streaming of the video.
Further, the terminal device 3 may receive the re-distributed video (video in which the clip video is inserted as a replacement). Then, for example, the terminal device 3 acquires the text data of the live comment by the operation of the user or the like. The terminal device 3 provides the acquired text data as additional content to the server device 2. The server device 2 also redistributes the text data of the live comment provided by the terminal device 3 as content.

Eleventh service example: Live distribution commentary addition application For example, video and audio of a sports broadcast are distributed from the web server device 7 as original contents. The terminal device 3 (for example, a smartphone) receives and reproduces the distributed original content. Further, the terminal device 3 acquires the user's spoken voice and generates additional voice content while playing back the original content. The terminal device 3 passes the generated audio content to the server device 2. The server device 2 redistributes the content including at least the video included in the original content and the audio generated by the terminal device 3. As a result, a social commentary service can be realized.
Further, the server device 2 may synchronize the audio contents passed from the plurality of terminal devices 3 with each other, that is, integrate them on the server device 2 and redistribute them together with the original video. As a result, commentary by a plurality of users can be added to the original video and distributed.
Further, the terminal device 3 receives the commentary added content (processed content). Then, the terminal device 3 acquires commentary evaluation information (for example, rating numerical value information) based on the user's operation or the like, and generates this evaluation information as, for example, additional text content. The terminal device 3 passes the additional text content to the server device 2. The server device 2 can appropriately process the evaluation information as necessary and redistribute it together with the original content. As a result, user evaluation information can be added to each of a large number of audio tracks (commentary content). This makes it possible to efficiently select high-quality commentaries that are popular with users.

12th service example: Virtual stadium For example, a live image such as a sports broadcast is distributed from the web server device 7 as original content. The terminal device 3 of an unspecified number of users receives the content, acquires a comment by the user's voice, and generates the voice content. The terminal device 3 passes the generated audio content to the server device 2. The server device 2 receives the audio content passed from the terminal device 3. The server device 2 may receive the respective audio contents from a large number of terminal devices 3 and integrate them. The server device 2 redistributes the processed content including at least the video included in the original content and the audio received from the terminal device 3.
The receiving terminal 8 can synchronize and reproduce the video of the original content and the comments of a large number of users generated by the terminal device 3. That is, on the receiving terminal 8, it is possible to appreciate the original video together with the sound mixed in the atmosphere as if the sports competition is being held. In other words, the viewing user can enjoy the sense of unity in the field.
Further, when the server device 2 collects the contents of the voice comments from the plurality of terminal devices 3, only the voice comments from the terminal devices 3 owned by the plurality of users of friends may be integrated and redistributed. .. Further, the server device 2 may integrate and redistribute only the voice comments from the terminal device 3 owned by the user who favors a specific team or player.

13th service example: Social orchestra application For example, a video including a music performance sound is distributed from the web server device 7 as original content. The terminal device 3 receives and reproduces the video of the original content, and also acquires the singing by the user of the terminal device 3 and the sound of the musical instrument performance by the user. The terminal device 3 generates audio content. The terminal device 3 passes the added audio content to the server device 2. The server device 2 synchronizes and redistributes the audio content passed from the terminal device 3 and the video of the original content (audio may be added). The server device 2 may synchronize the audio content passed from the plurality of terminal devices 3 with the video of the original content and redistribute it. This makes it possible to realize a virtual music session by a plurality of performers and singers.
Further, the terminal device 3 may acquire an image of a user singing or a user playing a musical instrument as additional content. The terminal device 3 passes the acquired video to the server device 2. The server device 2 also synchronizes and distributes the video passed from the terminal device 3 with the original content.
Further, the plurality of terminal devices 3 do not necessarily have to have a session at the same time. That is, the timing of reproducing the original content may be different between a certain terminal device 3 and the other terminal device 3, and as a result, the timing of acquiring the sound or video of singing or playing a musical instrument may be different. In that case, the server device 2 acquires the additional content (video or audio) acquired at each timing from the terminal device 3, and re-distributes all the additional content in synchronization with the timing of the original content. As a result, the content of the session can be generated without the user of each terminal device 3 singing or playing at the same time.
Further, when the server device 2 distributes the processed content to the receiving terminal 8, for example, based on a request from the receiving terminal 8, only a specific audio track or a specific video track may be selected and distributed. good. Alternatively, the receiving terminal 8 may select and play back only a specific audio track or a specific video track.
Further, when the server device 2 distributes the processed content to the receiving terminal 8, the volume level of each audio track may be mixed at an arbitrary ratio. As a result, it becomes possible to deliver a plurality of patterns of content such as audio content that emphasizes musical instruments and audio content that emphasizes singing.

14th service example: Video work dubbing application For example, a video such as a movie or an animation is distributed as original content from the web server device 7. The terminal device 3 receives and reproduces the video of the original content. The terminal device 3 acquires the user's voice spoken at the timing of the video of the original content. The terminal device 3 generates audio content and passes it to the server device 2. The server device 2 synchronizes and redistributes the additional audio content passed from the terminal device 3 and the video which is the original content.
The server device 2 can also receive additional audio content from the plurality of terminal devices 3 and redistribute the plurality of audio content in synchronization with the original content. This makes it possible for a plurality of users to share different roles and perform dubbing. The server device 2 may receive additional audio contents from the plurality of terminal devices 3 at the same time, or may receive them at different timings.
Further, the server device 2 may receive additional audio content a plurality of times from one terminal device 3 and redistribute the plurality of audio contents in synchronization with the original video content. As a result, one user can perform dubbing while playing a plurality of roles.
Further, the terminal device 3 may receive the processed content redistributed from the server device 2, and the user of the terminal device 3 may vote for the processed content, input an evaluation value, or the like. As a result, the terminal device 3 acquires popular text data of various processed contents as additional contents. The terminal device 3 passes this text data to the server device 2. The server device 2 can redistribute the text data content indicating the popularity of each processed content. With this mechanism, for example, it is possible to hold an event to compete for which processed content (video) was interesting.

Fifteenth service example: Multilingual live commentary and explanation For example, a live video of a sporting event is distributed as original content from the web server device 7. By the operation of the commentator or the commentator of the sporting event, the terminal device 3 receives and reproduces the video (which may be accompanied by audio) of the original content. The commentator and commentator may be in the same place or in different places. In addition, the terminal device 3 acquires the audio of the commentator and the commentator, and generates additional audio content. The terminal device 3 passes the generated audio content to the server device 2. The server device 2 synchronizes and distributes the video which is the original content and the audio content generated by the terminal device 3. As a result, the commentator and the commentator can add the commentary sound and the commentary sound in a specific language to the content. In addition, for each language (for example, Japanese, English, Chinese, French, etc.), the commentator and the commentator may emit the content of the commentary audio and the commentary audio to generate additional content. .. In addition, even if the commentator and the commentator are separated from each other, it is possible to generate the content by the dialogue voice. As a result, for example, a large-scale sporting event can be played in multiple languages.

Example of 16th service: Audio commentary for live streaming For example, some kind of video content (arbitrary content including music live, talk live, etc.) is distributed from the web server device 7 as original content. The terminal device 3 receives and reproduces the original content. In addition, the terminal device 3 acquires the user's voice and generates additional content by voice. The terminal device 3 passes the generated audio content to the server device 2. The server device 2 synchronizes the received additional content with the original content and redistributes it.
The server device 2 may receive additional content (audio) from the plurality of terminal devices 3, mix them, and redistribute them.
With this mechanism, users can share voice comments in the video distribution service.

17th Service Example: Distribution of Audience Audio at Public Viewing Venue For example, a live video (including audio) of a large-scale sporting event is distributed from the web server device 7 as original content. At the so-called public viewing venue, the terminal device 3 (PC or the like) reproduces the video and audio of the original content. The video is displayed on the large screen of the public viewing venue, and the audio is output from speakers and the like. The terminal device 3 acquires the audio of the audience at the public viewing venue and generates audio content. The terminal device 3 passes the generated additional content to the server device 2. The server device 2 synchronizes the additional audio content passed from the terminal device 3 with the original content and redistributes them. As a result, the receiving terminal 8 can receive and play back the content including not only the sound of the venue where the sporting event is held but also the sound of the public viewing venue.
The public viewing venue may be located in the same area as the venue of the sporting event, or may be located in a remote location (including different countries). When the venue of the sporting event and the venue of the public viewing are separated from each other, the audience of the audience at the venue of the sporting event and the audience of the audience at the venue of the public viewing may be different. For example, the spectators at both venues may have different favored players, favored teams, cultural backgrounds, and so on. In this case, on the receiving terminal 8 side, it is possible to experience the realism of the public viewing venue, which is different from the realism of the voice of the sports event venue, through the processed content to be delivered.

18th service example: Directions video generation application The terminal device 3 transmits a search keyword to the server device of the search engine based on the user's operation or the like. The search keyword includes the place name of the departure point and the destination of the movement, for example, "from Shiodome to Nihonbashi". The intention of the user at this time is to obtain information on directions. Then, as the original content, the basic moving video corresponding to the search keyword is distributed from the web server device 7. When the search keyword is "from Shiodome to Nihonbashi", the basic movement video is the video on the route of movement between those points. The basic moving image is stored in the database in advance. In addition, position information (longitude, latitude), time information, and information related to the means of transportation are associated with the basic moving image. The basic moving image may be prepared in advance for each time zone. In that case, the basic moving video in the time zone closest to the actual time zone is delivered as the original content. Further, the basic moving image may be a combination of a plurality of images. For example, the basic moving image from Shiodome to Nihonbashi may be a combination of the moving image from Shiodome to Ginza 4-chome and the moving image from Ginza 4-chome to Nihonbashi. The terminal device 3 receives and reproduces the basic moving video which is the original content. Further, the content generation unit 43 in the terminal device 3 extracts data associated with the basic moving video. For example, the content generation unit 43 extracts moving position information (coordinate information). Then, the content generation unit 43 searches for another database (for example, the database owned by the terminal device 3 itself or the database on the server accessed via the Internet) based on the extracted location information, and the information related to the location. To get. Information related to the location includes, for example, tourist spot information (location, highlights, historical background, etc.), restaurant information (location, menu, etc.), and other store information (location, business format, sales). Etc.) and so on. Position-related information is given, for example, in textual information. The content generation unit 43 generates composite content based on the moving video, which is the original content, and the information on the above-mentioned tourist spots, restaurants, and stores. In this composite content, information on tourist spots, restaurants, and stores includes, for example, images and texts displayed at specific places in the video. The terminal device 3 passes the newly generated content (composite content of moving video and information such as a store) to the server device 2. The server device 2 redistributes the processed content. The redistributed content can be viewed by the terminal device 3 itself, or can be viewed by another receiving terminal 8.
As a result, information on tourist spots, stores, etc. can be provided to tourists in an easy-to-understand manner. Further, the advertisement information may be provided in association with the basic movement information.

19th Service Example: Timeline Search Key Generation For example, arbitrary video or audio content (which may include both) is distributed from the web server device 7 as original content. The terminal device 3 receives the original content, and the decoder unit 41 of the terminal device 3 decodes the content and provides video, audio, or the like to the content generation unit 43. The content generation unit 43 includes a video recognition engine and a voice recognition engine. The video recognition engine of the content generation unit 43 performs the video recognition process supplied from the decoder unit 41, recognizes what the person, object, or scene contained in the video is, and outputs the text data of the recognition result. do. Further, the voice recognition engine of the content generation unit 43 performs voice recognition processing of the voice supplied from the decoder unit 41, recognizes what the words and sentences included in the voice are, and outputs the text data of the recognition result. Further, the content generation unit 43 edits and outputs the text data obtained as a result of the video recognition or the voice recognition in a form that can be easily used as the metadata for the search key. In the metadata for the search key, the search term and the time position in the video content or the audio content (for example, the relative time from the start of the content) are associated with each other. Further, the search term and the scene such as a video may be related to each other. The video content can be divided at the point of switching the scene, and the audio content can be divided into silent sections having a predetermined length by using the existing technology. The terminal device 3 passes the text data generated as the search key metadata to the server device 2. The server device 2 redistributes the text data passed from the terminal device 3 together with the original content. On the receiving terminal 8 side that receives the processed content re-distributed from the content processing system 1, by referring to the metadata for the search key, the part corresponding to the search term in the video content or the audio content is quickly searched ( For example, it is possible to cue). That is, the viewer who operates the receiving terminal 8 can easily view a desired scene or the like.

20th service example: Subtitle generation For example, arbitrary video or audio content (which may include both) is distributed from the web server device 7 as original content. The terminal device 3 receives the original content, and the decoder unit 41 of the terminal device 3 decodes the content and provides video, audio, or the like to the content generation unit 43. The content generation unit 43 includes a voice recognition engine. The voice recognition engine of the content generation unit 43 performs the voice recognition process of the voice supplied from the decoder unit 41, and outputs the transcription text data from the voice. In addition, the content generation unit 43 automatically or at least partially generates subtitle text data from the text data based on the operation of the reviewer. For example, the subtitle text data is output as timed text or timeline type text data. The terminal device 3 passes the generated subtitle text data to the server device 2. The server device 2 redistributes the text data passed from the terminal device 3 in synchronization with the original content. On the receiving terminal 8 side that receives the processed content, the subtitle text can be displayed together with the original content. Alternatively, the receiving terminal 8 side can output the result of performing a predetermined process on the subtitle text in association with the original content.

21st service example: Automatic highlight video generation (1)
For example, arbitrary video or audio (which may include both) content is delivered from the web server device 7 as original content. The terminal device 3 receives the original content, and the decoder unit 41 of the terminal device 3 decodes the content and provides video, audio, or the like to the content generation unit 43. The content generation unit 43 includes a video recognition engine and a voice recognition engine. These recognition engines may be those that utilize artificial intelligence technology. The content generation unit 43 extracts the main events in the original content by recognizing and analyzing the video and audio. This major event is extracted by learned artificial intelligence, for example, based on audio content, sound pressure level, video content, a specific pattern in the video, and the like. The content generation unit 43 recognizes a fragment including a main event in the original content as a highlight scene, and generates a highlight video (or audio) by, for example, cutting out and connecting only a plurality of highlight scenes. Highlight images and the like are also called "highlight clips". The terminal device 3 passes the content of the highlight clip to the server device 2. The server device 2 redistributes the highlight clip passed from the terminal device 3 together with the original content. Alternatively, the server device 2 replaces the original content and redistributes only the highlight clip passed from the terminal device 3. This mechanism makes it possible to easily deliver only the highlights of video and audio contents.

Although a plurality of service examples have been described above, the content processing system 1 may be implemented by combining a plurality of the service examples described above.

In the above embodiment, the content processing system 1 is configured by using the server device 2 and the terminal device 3. The functional configuration of each device is as shown in FIGS. 1 and 2. However, as a modification, the arrangement of functions in each of the server device 2 and the terminal device 3 may be arbitrarily changed. Further, the content processing system 1 may be configured by distributing the functions to three or more devices. Further, the contents processing system 1 may be configured as one device by integrating the functions of the server device 2 and the terminal device 3.

According to the present embodiment and its modification, it is possible to encode and distribute only the content to be added, and distribute the original content as the original segment file. This makes it possible to process and redistribute streamed content with a relatively small device configuration.
Further, by delivering the above-mentioned additional content in the form of replacing a part of the original segment file, the replacement of the content can be realized with a relatively small device configuration.
In addition, a wide variety of services can be realized by using this embodiment.

Although the embodiments, modifications, and service examples of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment and does not deviate from the gist of the present invention. Design etc. are also included.

The present invention can be used, for example, in a content distribution business (including a broadcasting business). However, the scope of use of the present invention is not limited to the business exemplified here.

0 Distribution system 1 Content processing system (content processing equipment)
2 Server device 3 Terminal device 6 Encoder device 7 Web server device 8 Receiving terminal 21 Manifest acquisition unit 22 Manufest generation unit 25 Segment acquisition unit 26 Segment selection unit 28 Web server unit (redistribution unit)
31 Manufest acquisition unit 32 Manufest analysis unit 33 Segment acquisition unit 35 Time analysis unit 41 Decoder unit 42 Playback unit 43 Content generation unit 44 A / V interface unit (interface unit)
45 Mixer section 48 Encoder section 49 Segment section 50 Upload section

Claims (10)

A manifest acquisition unit that acquires the original manifest file included in the original content of streaming using the hypertext transfer protocol, and
A segment acquisition unit that acquires the original segment file included in the original content,
A decoder unit that decodes and outputs the original segment file, and
An interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and an interface unit.
The encoder section that encodes the additional content and
A segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.
A manifest generation unit that generates a processing manifest file so that the original segment file and the additional segment file are synchronized based on the original manifest file.
A redistribution unit that distributes the original segment file, the additional segment file, and the processing manifest file as processed contents using a hypertext transfer protocol.
Content processing system equipped with. The manifest generation unit generates the processing manifest file that includes all of the acquired original segment files and that reproduces the additional processing content including the additional segment file.
The content processing system according to claim 1, wherein the content processing system is characterized in that. The manifest generation unit generates the processing manifest file for reproducing the replacement type processing content including only a part of the acquired original segment file and including the additional segment file.
The content processing system according to claim 1, wherein the content processing system is characterized in that. A content processing system that includes a server device and a terminal device.
The terminal device is
A manifest acquisition unit that acquires the original manifest file included in the original content of streaming using the hypertext transfer protocol, and
A segment acquisition unit that acquires the original segment file included in the original content,
A decoder unit that decodes and outputs the original segment file, and
An interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and an interface unit.
The encoder section that encodes the additional content and
A segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.
Equipped with
The server device
A manifest generation unit that generates a processing manifest file so that the original segment file and the additional segment file are synchronized based on the original manifest file.
A redistribution unit that distributes the original segment file, the additional segment file, and the processing manifest file as processed contents using a hypertext transfer protocol.
Equipped with
Content processing system. The original segment file stores data obtained by encoding at least one of video and audio.
A content generation unit that automatically generates the additional content based on the original segment file by analyzing the video or audio output by the decoder unit.
The content processing system according to any one of claims 1 to 4, further comprising. The content generation unit generates the additional content including text data obtained by performing a video or audio recognition process output by the decoder unit.
The content processing system according to claim 5. The manifest generation unit has a function of generating a processed manifest file for reproducing only the original segment file based on an external instruction.
The redistribution unit has a function of distributing only the original segment file and the processing manifest file based on the instruction from the outside.
The content processing system according to any one of claims 1 to 6. A manifest acquisition unit that acquires the original manifest file included in the original content of streaming using the hypertext transfer protocol, and
A segment acquisition unit that acquires the original segment file included in the original content,
A decoder unit that decodes and outputs the original segment file, and
An interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and an interface unit.
The encoder section that encodes the additional content and
A segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.
A terminal device comprising. Computer,
A manifest acquisition unit that acquires the original manifest file included in the original content of streaming using the hypertext transfer protocol, and
A segment acquisition unit that acquires the original segment file included in the original content,
A decoder unit that decodes and outputs the original segment file, and
An interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and an interface unit.
The encoder section that encodes the additional content and
A segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.
A manifest generation unit that generates a processing manifest file so that the original segment file and the additional segment file are synchronized based on the original manifest file.
A redistribution unit that distributes the original segment file, the additional segment file, and the processing manifest file as processed contents using a hypertext transfer protocol.
A program for functioning as a content processing system equipped with. Computer,
A manifest acquisition unit that acquires the original manifest file included in the original content of streaming using the hypertext transfer protocol, and
A segment acquisition unit that acquires the original segment file included in the original content,
A decoder unit that decodes and outputs the original segment file, and
An interface unit that acquires additional content newly added in a form associated with the decoded original segment file, and an interface unit.
The encoder section that encodes the additional content and
A segmentation unit that generates an additional segment file by segmenting the encoded additional content so as to synchronize with the time of the original segment file.
A program for functioning as a terminal device equipped with.

Images