JP6700957B2 - Subtitle data generation device and program - Google Patents

Description

The present invention, subtitle data generation apparatus, and a program.

  There is a demand for the spread of a system for delivering moving image content such as television broadcast programs simultaneously with broadcasting via a communication line (such as the Internet). The widespread availability of such a system is expected to enable viewing of programs in good condition in various broadcast receiving environments.

  One of the components of a system for delivering moving image content such as broadcast programs via a communication line is an encoder device. In particular, an encoder device that receives a broadcast signal from a television and outputs a moving image file or the like that can be distributed in real time is required. Such an encoder device is also called a "live encoder". When delivering the content of a broadcast program via a communication line, with respect to video (including audio), a file to be delivered can be generated by the encoder device (live encoder) described above.

  According to the standard defined by ARIB (Association of Radio Industries and Businesses), subtitle data for television broadcasting is put on a broadcast signal in the form of text data in addition to video, and sent from a transmitting device. On the side of the television receiver, the video and the subtitle text are extracted from the broadcast signal, and they are displayed while synchronizing them according to a predetermined presentation time. The subtitle text is presentation-controlled by the presentation start time and the presentation end time that are set.

Regarding the display of subtitles, the technologies are described in the following documents.
Patent Document 1 describes a technique for simultaneously displaying a current subtitle and a past subtitle. Specifically, in the technique of the document, the display unit has two screens. Then, on the first screen (current screen), the current subtitle synchronized with the video of the program extracted from the received broadcast signal is displayed. On the other hand, on the second screen (past screen), past captions displayed at a timing a predetermined time before the current video and subtitles displayed on the current screen are displayed.

  Patent Document 2 describes a technique for controlling stream output by using captions displayed in the past. Specifically, the information is separated from the stream having the multiplexed time information stored in the storage device. The separated information is subtitles, video, and audio. If the separated information is a subtitle, the subtitle is held in the subtitle list holding memory. Then, the subtitles are combined with the video corresponding to the time information and displayed and output. Then, when a specific caption of the caption history stored in the caption list holding memory is selected, the stream output is controlled based on the time information corresponding to the caption. The technique of the document is presumed to have contents recorded in advance on a recording medium on the viewer side, such as a DVD player device and a hard disk recorder device.

JP, 2009-177720, A JP, 2003-018491, A

As described above, it is possible in the related art that the encoder device (live encoder) encodes video and audio based on the broadcast signal and outputs it as a file. However, the conventional encoder device cannot encode subtitle data in real time. Therefore, even if an attempt is made to deliver a television program through communication simultaneously with broadcasting, there is a problem that only a video without subtitles can be delivered. Even in the content distribution currently being carried out via communication lines, subtitle data is not distributed.
The techniques described in Patent Document 1 and Patent Document 2 enable the viewer to view past subtitles and control the output stream based on the past subtitles. It does not generate distributable subtitle data in real time.
In the future, it is desired that subtitles can be viewed on the receiving side (viewer side) even when the content of a broadcast program is distributed via a communication line.

  The present invention has been made based on the above-mentioned problem recognition, and is intended to provide a caption data generation device and a program thereof for generating caption data that can be distributed in real time based on a broadcast signal. is there. Further, the present invention aims to provide a content display device and a program for displaying subtitle data and the like generated by such a subtitle data generation device or program.

  [1] To solve the above problem, a caption data generation device according to an aspect of the present invention acquires caption text and information on the presentation time of the caption text from caption data extracted from a broadcast signal acquired from the outside. Then, the subtitle conversion unit that outputs the subtitle text in association with the presentation time, the storage unit that stores the video file encoded based on the broadcast signal, and the presentation time of the video file that the storage unit stores. A data generation unit that generates a subtitle file including the subtitle text so as to be synchronized, and an output unit that outputs the moving image file read from the storage unit and the subtitle file generated by the data generation unit. To have.

  [2] Further, according to an aspect of the present invention, in the caption data generation device, the moving image file is a plurality of moving image files divided into segments each having a predetermined length of time, and the storage unit includes: The moving image playlist file, which is data of a playlist including presentation time information of each moving image file for presenting the plurality of moving image files in an appropriate order, is further stored. While generating a plurality of the subtitle files respectively corresponding to the plurality of video files while referring to the list file, a subtitle playlist file which is data of a playlist for presenting the generated subtitle files in an appropriate order is further created. The output section further outputs the moving picture playlist file and the subtitle playlist file.

  [3] Further, according to an aspect of the present invention, in the above subtitle data generation device, when the subtitle conversion unit includes an external character, the location information of a font corresponding to the external character is included. Is output as a subtitle text in a form associated with the external character.

  [4] Further, according to one aspect of the present invention, a computer acquires a caption extraction unit that extracts caption data from a broadcast signal acquired from the outside, and acquires the caption text and information of the presentation time of the caption text from the caption data. , A subtitle conversion unit that outputs the subtitle text in association with the presentation time, a storage unit that stores a video file encoded based on the broadcast signal, so as to synchronize with the presentation time of the video file stored by the storage unit And a program for functioning as a data generation unit that generates a subtitle file including the subtitle text, and an output unit that outputs the moving image file read from the storage unit and the subtitle file generated by the data generation unit. Is.

  [5] Further, the content display device according to one aspect of the present invention includes a communication unit that receives a moving image file and a subtitle file corresponding to the moving image file, and an image and the image by decoding the received moving image file. And a subtitle processing unit that outputs, from the received subtitle file, subtitle text and subtitle presentation time information that is the presentation time of the subtitle text. , The video display area, which is an area for displaying the video, does not overlap while synchronizing the timing of presenting the video and the subtitle text based on the information on the video presentation time and the information on the subtitle presentation time. A presentation control unit that displays the subtitle text in the subtitle display area.

  [6] Further, according to an aspect of the present invention, in the content display device described above, the information on the caption presentation time includes information on a caption presentation start time and a caption presentation end time. , Starting display of the subtitle text corresponding to the subtitle presentation start time at the subtitle presentation start time, and not ending display of the subtitle text even when the subtitle presentation end time corresponding to the subtitle text arrives The subtitle text is controlled so as to be displayed at a position different from the display position of the subtitle text in the subtitle display area.

  [7] Further, according to an aspect of the present invention, in the above content display device, when the presentation control unit receives an operation of selecting the displayed subtitle text, the presentation control unit corresponds to a presentation time of the subtitle text. It is characterized by performing a quick rewind to the position and restarting the presentation of the moving image file from the position.

  [8] Further, according to an aspect of the present invention, in the content display device described above, the presentation control unit acquires, from the subtitle file, speaker identification information that identifies a speaker corresponding to the subtitle text. The subtitle text is displayed in a form associated with the speaker identification information.

  [9] Further, according to one aspect of the present invention, a computer is provided with a decoding unit that outputs a video and information of a video presentation time, which is a presentation time of the video, by decoding a video file received by the communication. From the received subtitle file, a subtitle processing unit that outputs subtitle text and information of subtitle presentation time that is the presentation time of the subtitle text, the video based on the video presentation time information and the subtitle presentation time information. And a presentation control unit that displays the subtitle text in a subtitle display area that does not overlap with the video display area that is an area for displaying the video, while synchronizing the presentation timing of the subtitle text. It is a program.

  According to the present invention, subtitles can be distributed when a broadcast program is simultaneously distributed in real time on a communication line. In addition, subtitles can be displayed in time series on the content display device side, which makes it easy to view the distributed content even in a mobile environment.

It is a block diagram showing functional composition of a title data generation device by an embodiment of the present invention. FIG. 3 is a block diagram showing a schematic functional configuration of a distribution system including the caption data generation device according to the same embodiment. It is a schematic diagram showing an example of a play list file (moving picture m3u8) which a caption data generation device by the embodiment acquires from an encoder device and memorize|stores in a memory|storage part. It is the schematic which shows the example of the master (master.m3u8) of the play list file which the subtitle data generation apparatus by the same embodiment acquires from an encoder apparatus and memorize|stores in a memory|storage part. It is a schematic diagram showing composition of subtitle data listed by a subtitle conversion part by the embodiment. It is a schematic diagram showing an example of composition of a subtitles file which a data generation part by the embodiment generates. It is a schematic diagram showing an example of composition of a playlist file (subtitle m3u8) of a subtitle which a data generation part by the embodiment generates. It is a schematic diagram showing an example of composition of a master (master.m3u8) of a play list file which a data generation part by the embodiment. FIG. 3 is a block diagram showing a schematic functional configuration of a client device according to the same embodiment. It is a schematic diagram showing an example of composition of a contents presentation screen in the client device side by the embodiment. FIG. 9 is a schematic diagram showing an example in which the client device according to the same embodiment displays subtitles in a mode different from the previous mode.

Next, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing the functional configuration of the caption data generation device according to the present embodiment. In this figure, reference numeral 1 is a caption data generation device. As illustrated, the caption data generation device 1 is configured to include a caption extraction unit 11, a caption conversion unit 12, a data generation unit 13, an output unit 14, and a storage unit 20. Each of these units is realized by using an electronic circuit and processes an electric signal representing information. As will be described later, a computer may be used to realize the function of each unit. The function of each unit will be described below.

The caption extraction unit 11 takes in a broadcast signal input from the outside and extracts caption data from the acquired broadcast signal. The broadcast signal is transmitted to the caption data generation device 1 by SDI (Serial Digital Interface). SDI is a standard interface used for broadcasting equipment. The format of the broadcast signal is based on the standard established by ARIB (Association of Radio Industries and Business). The subtitle data is also superimposed on the input broadcast signal according to the ARIB regulations. The caption data is stored in the vertical blanking area of HD-SDI or SD-SDI, and the caption extraction unit 11 extracts this caption data. The subtitle data may be stored in another area of the broadcast signal.
The caption extraction unit 11 passes the extracted caption data to the caption conversion unit 12.
The function of the caption extraction unit 11 may exist in a device outside the caption data generation device 1. In that case, the caption extraction unit existing outside the caption data generation device 1 extracts the caption data from the broadcast signal, and passes the extracted caption data to the caption conversion unit 12 of the caption data generation device 1.

  After converting the extracted subtitle data into a text format, the subtitle conversion unit 12 lists the text of the subtitle text and its additional information. The additional information includes information on presentation time (presentation start time and presentation end time), information on display position of subtitles on the screen, and information on character decoration. That is, the subtitle conversion unit 12 acquires the subtitle text and the information on the presentation time of the subtitle text from the subtitle data, and outputs the subtitle text and the presentation time in association with each other. The data listed by the subtitle conversion unit 12 will be described later with reference to FIG.

In addition, when the extracted subtitle data includes an external character in the text, the subtitle conversion unit 12 determines the external character as information on an external character font (also referred to as “web font”) that can be obtained on the web (that external character font is URL to retrieve). The URL is an abbreviation for Uniform Resource Locator. Here, the external character is a character that is not included in the character set defined as a standard. Examples of the standard of character codes are JIS (Japanese Industrial Standard) code and Unicode (Unicode). As the external character code, the code of the empty area of the standard character code system is assigned. Note that pictograms may also be treated as a type of external character. The subtitle conversion unit 12 holds in advance data of the correspondence relationship between each external character and the location information (URL, etc.) of the web font, and when the external character exists in the subtitle text, the external character corresponding to that external character is stored. The font location information is associated and output. By performing such conversion, the client device side that has received the subtitle data distribution can acquire an appropriate web font and display the external character.
The subtitle conversion unit 12 passes the listed data to the data generation unit 13.

  The data generation unit 13 generates a subtitle file 23 based on the subtitle data converted by the subtitle conversion unit 12 and the playlist file 22 read from the storage unit 20, and writes the subtitle file 23 in the storage unit 20. In addition, the data generation unit 13 adds the information regarding the subtitles to the read playlist file 22 to generate the playlist file 24 and writes the playlist file 24 in the storage unit 20. The data generation unit 13 does not modify the read playlist file 22 (moving image playlist file) itself. That is, the data generation unit 13 generates the subtitle file 23 including the subtitle text so as to be synchronized with the presentation time of the moving image file 21 stored in the storage unit 20.

  The moving image file 21 and the playlist file 22 are data files acquired from the outside and written in the storage unit 20. The moving image file 21 and the playlist file 22 correspond to the broadcast signal input to the subtitle extraction unit 11. That is, the moving image file 21 and the playlist file 22 are files generated by an external encoder device (described later) based on the broadcast signal. Specifically, the data generation unit 13 generates a plurality of subtitle files respectively corresponding to a plurality of video files with reference to a video playlist file described later, and presents the generated subtitle files in an appropriate order. A subtitle playlist file that is playlist data is further generated.

  The output unit 14 outputs the distribution moving image file 21, the subtitle file 23, and the playlist file 24 to the outside. Specifically, the output unit 14 passes the moving image file 21, the subtitle file 23, and the playlist file 24 to an external content distribution server device (described later). The playlist file 24 output by the output unit 14 includes a moving image playlist file, a subtitle playlist file, and a playlist master.

  The storage unit 20 temporarily stores a moving image file 21, a playlist file 22, a subtitle file 23, and a playlist file 24. The storage unit 20 is internally provided with a storage medium such as a hard disk device or a semiconductor disk device for storing these files.

Here, each file stored in the storage unit 20 will be further described.
The moving image file 21 is a file of moving image content (including video and audio). The moving image file 21 holds moving image content obtained by encoding with an external encoder device based on a broadcast signal. The moving image file 21 exists by being divided into files having a predetermined segment length. The length of the segment is appropriately determined, but is, for example, 5 seconds or 10 seconds. That is, there are a plurality of moving image files 21 for each segment.
The playlist file 22 is a file that describes the overall structure of the moving image content. The playlist file 22 is also a file generated by an external encoder device. The playlist file 22 holds data describing when (presentation time) and in what order the moving image files 21 existing in plural for each segment should be reproduced. In the present embodiment, the play list file 22 includes a moving picture m3u8 file (moving picture play list file), a master. It consists of m3u8 file. master. The m3u8 file is a master file that serves as a basis for showing the configuration of the entire multimedia content. The moving image m3u8 file is play list data including information on the presentation time of each moving image file for presenting a plurality of moving image files in an appropriate order.
A specific example of the playlist file 22 will be described later with reference to the drawings.

The subtitle file 23 and the playlist file 24 are generated by the data generation unit 13 and written in the storage unit 20.
The subtitle file 23 is a file including subtitle text data corresponding to the moving image file 21. The subtitle file 23 holds data of subtitle text extracted from the broadcast signal acquired by the subtitle extraction unit 11. This broadcast signal is the same as the broadcast signal on which the moving image file 21 is based. The subtitle file 23 is also divided into a plurality of files each having the length of the segment, and a plurality of subtitle files 23 exist. That is, for a certain segment, one subtitle file 23 exists corresponding to one moving image file 21.
The playlist file 24 is a file in which information regarding subtitles is added to the playlist file 22. In the present embodiment, the playlist file 24 includes the moving image m3u8 file described above, a subtitle m3u8 file that is a playlist related to subtitles, and a master. Including m3u8 file. Here, the master. Information about subtitles is additionally written in the m3u8 file.

  That is, on the side that receives the playlist file 24, the moving image file 21, and the subtitle file 23, by referring to the playlist file 24, the moving image file 21 and the subtitle file 23 are generated for each segment corresponding to the presentation time. It is possible to present it. That is, the moving image file 21 and the subtitle file 23 are presented in synchronization with each other.

As the format of the moving image file, HLS (HTTP Live Streaming, HTTP live streaming) or MPEG-DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming over HTTP) can be used. It is not limited to. Further, as the format of the subtitle file, WebVTT (Web Video Text Track, Web Video Text Track), TTML (Timed Text Markup Language, Timed Text Markup Language), or ARIB-TTML can be used. However, it is not limited thereto. ARIB-TTML is TTML defined as a standard of ARIB.
In the following description, a description will be given assuming a specific format, but naturally, the present embodiment can also be applied to the case where a moving image or subtitle is distributed in another file format.

Next, a distribution system using the caption data generation device 1 will be described.
FIG. 2 is a block diagram showing a schematic functional configuration of a distribution system including the caption data generation device 1. That is, the figure shows the relationship between the caption data generation device 1 and the peripheral devices. As illustrated, the distribution system 100 is configured to include a caption data generation device 1, a time code inserter 51, a distributor 52, an encoder device 53, a content distribution server device 61, and a client device 71. It is something. The client device 71 may be called a “content display device”. A signal including data can be exchanged between the respective devices shown in FIG. In particular, the content distribution server device 61 and the client device 71 are connected by a communication line such as the Internet. Although only one client device 71 is shown in the figure, in actuality, the distribution system 100 may be configured using a large number of client devices 71.

  The time code inserter 51 inserts a time code into the input broadcast signal (SDI). The time code inserter 51 delivers the broadcast signal in which the time code has been inserted to the distributor 52. The SDI transmission is also performed between the time code inserter 51 and the distributor 52.

  The distributor 52 distributes the broadcast signal passed from the time code inserter 51 into two systems. The broadcast signal of the first system output from the distributor 52 is passed to the encoder device 53. Further, the broadcast signal of the second system output from the distributor 52 is passed to the caption extraction unit 11 of the caption data generation device 1. That is, the same broadcast signal is distributed by the distributor 52, the broadcast signal of the first system is used for encoding a moving image, and the broadcast signal of the second system is used for extracting caption data.

The encoder device 53 extracts video and audio included in the input broadcast signal and outputs a moving image file (including video and audio). The moving image file is segmented into a predetermined length. The length of the segment is arbitrary, but has a predetermined length such as 5 seconds or 10 seconds. That is, each moving image file output by the encoder device 53 corresponds to a segment and has a predetermined segment length.
In addition, the encoder device 53 also outputs a playlist file describing the moving image file.
The process itself of inputting a broadcast signal and encoding video and audio can be performed by using an existing technology. That is, the encoder device 53 itself is based on the existing technology.
The moving image file and the playlist file generated by the encoder device 53 are passed to the subtitle data generation device 1. Then, these files are written in the storage unit 20 in the subtitle data generation device 1 as a moving image file 21 and a playlist file 22.

  The subtitle data generation device 1 acquires the moving image file and the playlist file output from the encoder device 53 and temporarily stores them in the internal storage means. Further, the caption data generation device 1 directly acquires a broadcast signal (SDI) from the distributor 52 and extracts caption data from the broadcast signal. Then, the caption data generation device 1 generates a caption file that matches the acquired moving image file. In addition, the subtitle data generation device 1 adds information about the subtitle file to the playlist file acquired from the encoder device 53 so that the above-mentioned moving image file and the subtitle file generated by itself can be presented in synchronization and output. To do. Then, the subtitle data generation device 1 passes these moving image file, subtitle file, and playlist file to the content distribution server device 61.

  The content distribution server device 61 distributes content data to the client device 71. Specifically, the content distribution server device 61 transmits a moving image file, a subtitle file, or a playlist file to the client device 71 in response to a request from the client device 71.

The client device 71 requests content data from the content distribution server device 61, receives the content data distributed from the content distribution server device 61, and presents it to the viewer. For example, the client device 71 receives a moving image file, a subtitle file, and a playlist file from the content distribution server device 61. Then, the client device 71 reads the moving image file and the subtitle file at a predetermined timing according to the received playlist file, and displays the moving image and the subtitle on the screen while synchronizing them. The client device 71 also outputs the sound included in the moving image file from a sound output unit such as a speaker or an earphone terminal.
The details of how the client device 71 displays the subtitle text will be described later.

With the above configuration, the caption data generation device 1 extracts caption data from a broadcast signal and outputs a caption file using the extracted caption data. The subtitle file is configured as data in a format that is easily used by the client device 71 of the delivery destination. Also, the subtitle data generation device 1 processes the playlist file so that the subtitle data and the moving image file are synchronized. Then, the subtitle data generation device 1 passes the playlist file to the content distribution server device 61 together with the moving image file and subtitle file for distribution.
The content distribution server device 61 can distribute these files. Also, the client device 71 side that has received the distribution acquires the moving image file and the subtitle file to be displayed based on the playlist file. As a result, the client device 71 that has received the distribution can reproduce and display the content including the subtitles.

  FIG. 3 is a schematic diagram showing an example of a playlist file that the subtitle data generation device 1 acquires from the encoder device 53 and stores in the storage unit 20. Specifically, this figure shows a moving image m3u8 file. "M3u8" is one of the formats of the multimedia playlist, and the "m3u8" file is described as text data. In the figure, line numbers are added for convenience. Hereinafter, the data example shown in the figure will be described.

"#EXT3U" in the first line is the header of the moving image m3u8 file.
“#EXT-X-VERSION:3” on the second line indicates that the compatible version of the moving image m3u8 file is “3”.
“#EXT-X-TARGETDURATION:5” on the third line indicates the maximum length (time length) of the media file (moving image file or the like) in seconds. In this example, since "5" is designated, it indicates that the maximum length of the media file is 5 seconds.
"#EXT-X-MEDIA-SEQUENCE: 0" on the fourth line indicates the sequence number of the first URL appearing in this playlist file. The sequence number is a number sequentially assigned to each segment of the medium. In this example, since "0" is designated as the sequence number, it indicates that the sequence number of the URL (the 7th line) that appears first is 0.

The fifth line to the seventh line correspond to one segment.
"#EXT-X-PROGRAM-DATE-TIME:2015-06-09T08:42:05.635Z" on the fifth line indicates the date and time associated with the start portion of the segment. Specifically, in this example, the start portion of this segment is “June 9, 2015 08:42:05.635”. In addition, "Z" indicates that the time is expressed in Coordinated Universal Time.
“#EXTINF:5.0” on the sixth line indicates the length of the segment in seconds. Specifically, in this example, the segment length is 5.0 seconds.
The 7th line describes the URL indicating the location of the moving image file (also called “chunk file” or “TS file”) of the segment.

The segments from the fifth line to the seventh line have been described above.
In the following parts of the file, a description of the segments follows. The 8th line to the 10th line describe the second segment in the file. The 11th to 13th lines describe the third segment in the file. The 14th line to the 16th line describe the fourth segment in the file.
As described above, in the example shown in the figure, the content is distributed as a continuous segment having a length of 5 seconds.
In this example, the moving image m3u8 file has 16 lines, but the moving image m3u8 file may include a description regarding a segment that follows further.
Although the location of the moving image file is described by the URL in this example, the location is not limited to this as long as it indicates the location, and the relative path may be described, for example.

FIG. 4 is also a schematic diagram showing an example of a playlist file that the subtitle data generation device 1 acquires from the encoder device 53 and stores in the storage unit 20. This figure shows the master of the playlist file (master.m3u8 file). In the figure, line numbers are added for convenience. Hereinafter, the data example shown in the figure will be described.
"#EXTM3U" in the first line is the header of the file.
The second and fourth lines show information about streaming. “PROGRAM-ID=” indicates information for identifying the program. In addition, “BANDWIDTH=” indicates a streaming bandwidth at the time of distribution.
The 3rd and 5th lines indicate the name of the moving image m3u8 file.

FIG. 5 is a schematic diagram showing the structure of the caption data converted and listed by the caption conversion unit 12.
The data shown in the figure is tabular data, and has items such as start time, end time, subtitle text, display position and character decoration.
Each row of this table corresponds to the segment defined in the file shown in FIG. That is, the subtitle extraction unit 11 extracts the subtitle text and the time code in association with each other from the broadcast signal including the time code. Then, the subtitle conversion unit 12 divides the extracted subtitle text for each segment and distributes it while referring to the playlist file 22 (that is, the moving image m3u8 file illustrated in FIG. 3) stored in the storage unit 20.

The start time is the presentation start time of subtitles corresponding to each segment. The start time of each segment shown in FIG. 5 corresponds to the start time of each segment described in the playlist file (moving image m3u8 file) shown in FIG.
The end time is the presentation end time of the subtitle. The end time data may be omitted. When the end time data is omitted, the start time of the next segment is treated as the end time of the segment.
Subtitle text is the text of the subtitle presented within that segment. The caption text is extracted from the broadcast signal by the caption extraction unit 11.
The display position/character decoration is information on the position where the subtitle text is displayed (coordinate information on the screen) and the decoration when the characters of the subtitle text are displayed (character size, font, underline, etc.). The display position and character decoration are also information extracted from the broadcast signal.
Incidentally, other data items may be further included.

  FIG. 6 is a schematic diagram showing a configuration example of a subtitle file generated by the data generation unit 13. The data generation unit 13 generates one segment of subtitle data as one subtitle file. That is, the data generation unit 13 generates many subtitle files corresponding to the segments. The figure shows a subtitle file corresponding to one segment. In the figure, line numbers are added for convenience. The subtitle file will be described below.

“WEBVTT” in the first line is header information and indicates that this file is a WEBVTT format file.
“X-TIMESTAMP-MAP:MPEGTS=1522260, LOCAL:00:00:00.000” on the second line is information indicating time mapping. Specifically, in this example, “MPEGTS:1522260” is information acquired from the moving image file (chunk file) and corresponds to the time stamp “1522260” in the moving image file. Further, "LOCAL: 00:00: 00.000" means that the above-mentioned time stamp "1522260" is "00:00: 00.000" (0:00: 00: 00) in local (relative) time notation. 000).

The data on the third line and the data on the fourth line are a pair. The third line shows the start time (00:00:00.000) and the end time (00:00:02.203) of subtitle presentation. The time described here is the relative time described above. "Aiueo" on the fourth line is the text that should be presented between the start time and the end time shown on the third line. This subtitle text "Aiueo" corresponds to the fourth line of the data shown in FIG.
The data on the fifth line and the data on the sixth line are a pair. The fifth line shows the start time (00:00:02.203) and end time (00:00:06.041) of subtitle presentation. The time described here is the relative time described above. "Kakikuke" on the sixth line is the text to be presented between the start time and the end time shown on the fifth line. This subtitle text "Kakikuke" corresponds to the fifth line of the data shown in FIG.

  FIG. 7 is a schematic diagram showing a configuration example of a playlist file of subtitles generated by the data generation unit 13. The subtitle playlist file is also generated as an m3u8 file similarly to the moving picture playlist file. Here, the subtitle playlist file is also referred to as a “subtitle m3u8 file”. In the figure, line numbers are added for convenience. Hereinafter, each line of the data shown in the figure will be described.

"#EXTM3U" in the first line is the header of the subtitle m3u8 file.
“#EXT-X-VERSION:3” on the second line indicates that the compatible version of the subtitle m3u8 file is “3”.
"#EXT-X-TARGETDURATION:5" in the third line indicates the maximum length (time length) of the media file in seconds. In this example, “5 seconds” is designated as the maximum length in accordance with the moving image m3u8 file (FIG. 3).
In the 4th line, "#EXT-X-MEDIA-SEQUENCE:0" is similar to the sequence number in the moving image m3u8 file (FIG. 3), and the first URL appearing in this playlist file has a sequence number of Indicates whether it is a thing.

The fifth and sixth lines correspond to one segment.
“#EXTINF:5.0” on the fifth line indicates the length of the segment in seconds. Specifically, in this example, the segment length is 5.0 seconds.
In the 6th line, the URL indicating the location of the subtitle file of the segment (the file described in FIG. 6. The file whose extension name is “.vtt”.) is described.
The subtitle m3u8 file does not include the description of the "#EXT-X-PROGRAM-DATE-TIME" attribute. The segment of the moving image file and the segment of the subtitle file have a one-to-one correspondence, and the start time of the segment is the same between the moving image file and the subtitle file. Therefore, the value of the “#EXT-X-PROGRAM-DATE-TIME” attribute described in the moving image m3u8 file also represents the start time of the corresponding subtitle file.
The subtitle m3u8 file may include a description of the attribute “#EXT-X-PROGRAM-DATE-TIME”.

The segments from the fifth line to the sixth line have been described above.
In the following part of the subtitle m3u8 file, the description about the segment sequentially follows. The 7th line to the 8th line describe the second segment in the file. The ninth to tenth lines describe the third segment in the file. The 11th line to the 12th line describe the fourth segment in the file.
In this way, in the example shown in the figure, a subtitle file exists in a form corresponding to a moving image file as a continuous segment of 5 seconds in length.
In this example, the number of lines of the subtitle m3u8 file is 12, but the subtitle m3u8 file may include a description regarding a segment subsequent to the moving image m3u8 file.
Although the location of the subtitle file is described by the URL in this example, the location is not limited to this as long as it indicates the location, and may be described by a relative path, for example.

  FIG. 8 is a schematic diagram showing a configuration example of the master (master.m3u8 file) of the playlist file generated by the data generating unit 13. The data generation unit 13 reads the master of the playlist file shown in FIG. 4 and adds information regarding subtitles to this master to generate the master of FIG. The master shown in FIG. 8 is a part of the playlist file 24 written in the storage unit 20. In the figure, line numbers are added for convenience. In addition, the underlined portion in the figure is information that is not included in the original master and is information that is added by the data generation unit 13. The details of this master will be described below.

"#EXTM3U" in the first line is the header of the file.
The entire second line is the information regarding the subtitle data added by the data generation unit 13. The description contained in the second line is as follows. “#EXT-X-MEDIA” indicates a definition related to media.
“TYPE=SUBTITLES” indicates that the media type is subtitles. “GROUP-ID=“xxxx”” indicates that the group ID of the media is “xxx”. “NAME=“Japanease”” indicates that the name of the medium is “Japanease”. “DEFAULT=YES”, “AUTOSELECT=YES”, and “FORCED=NO” respectively describe setting values for the medium. "LANGUAGE="ja"" indicates that the language used is Japanese. “URI=“sub/jpn/sub.m3u8”” describes a URI (uniform resource identifier) indicating the location of the subtitle m3u8 (playlist file). By specifying the URI, the subtitle playlist is associated with the master.

The third and fifth lines show information about streaming. The keyword parameters “PROGRAM-ID” and “BANDWIDTH” are the same as those described in the second and fourth lines of FIG. “SUBTITLES=“xxxx”” is added by the data generation unit 13 as a description regarding subtitles.
The 4th and 6th lines indicate the name of the moving image m3u8 file, and are the same as those described in the 3rd and 5th lines of FIG.

  As illustrated in FIG. 8, the data generation unit 13 adds the description related to the subtitle data generated by itself to the master (master.m3u8) and outputs it. By distributing such a master, the client device 71 receiving the distribution can allocate the subtitle file associated with the moving image file of each segment.

Next, the presentation of subtitles and the like on the client device side that receives distribution will be described.
FIG. 9 is a block diagram showing a schematic functional configuration of the client device. As illustrated, the client device 71 includes a communication unit 79, a storage unit 80, a decoding unit 81, a caption processing unit 82, a presentation control unit 83, and an output unit 84. The client device 71 is specifically a device such as a personal computer (PC), a smartphone (smartphone), or a wearable terminal. The wearable terminal is, for example, a wristwatch-type terminal or a spectacle-type terminal, but is not limited to these forms.

The communication unit 79 communicates with the outside. The communication unit 79 communicates with the content distribution server device 61 via the Internet, for example. Through this communication, the communication unit 79 receives the content data distributed from the content distribution server device 61. Specifically, the communication unit 79 receives the moving image file 21, the subtitle file 23, and the playlist file 24, and writes them in the storage unit 80.
The storage unit 80 stores data. Specifically, the storage unit 80 at least temporarily stores the moving image file 21, the subtitle file 23, and the playlist file 24. The storage unit 80 internally includes a storage medium such as a magnetic hard disk device or a semiconductor disk device.

The decoding unit 81 reads the moving image file 21 from the storage unit 80 and decodes the moving image file 21 under the control of the presentation control unit 83. As a result of the decoding process, the decoding unit 81 passes the video and audio to the presentation control unit 83. Also, the decoding unit 81 acquires information regarding the presentation time of the video from the moving image file and passes it to the presentation control unit 83.
Under the control of the presentation control unit 83, the subtitle processing unit 82 reads the subtitle file 23 from the storage unit 80 and acquires the subtitle text and the like from the subtitle file 23. If the subtitle text extracted from the subtitle file 23 includes an external character, the subtitle processing unit 82 also reads from the subtitle file 23 a URL indicating the location of the font of the external character (web font). Then, the subtitle processing unit 82 accesses the location of the URL via the communication unit 79 and acquires the font data of the external character. Then, the subtitle processing unit 82 outputs the acquired font data in a form associated with the external character. The subtitle processing unit 82 passes the data such as the subtitle text to the presentation control unit 83. Further, the subtitle processing unit 82 extracts information regarding the presentation time of the subtitle text from the subtitle file 23 and passes it to the presentation control unit 83.

The presentation control unit 83 controls presentation of moving image content. Specifically, the presentation control unit 83 controls the decoding unit 81 so as to decode the moving image file 21 for each segment at a predetermined timing while referring to the playlist file 24 stored in the storage unit 80. .. The presentation control unit 83 also controls the subtitle processing unit 82 so as to read and process the subtitle file 23 for each segment while also referring to the playlist file 24. In addition, the presentation control unit 83 passes the video and the subtitle text to the output unit 84 so as to be displayed on the screen while synchronizing with the appropriate presentation timing. Specifically, the presentation control unit 83 synchronizes the timing of presenting the video and the subtitle text based on the information of the video presentation time and the information of the subtitle presentation time, and a video display that is an area for displaying the video. Subtitle text is displayed in a subtitle display area that does not overlap the area. Further, the presentation control unit 83 starts displaying the subtitle text corresponding to the subtitle presentation start time at the subtitle presentation start time, and displays the subtitle text even when the subtitle presentation end time corresponding to the subtitle text arrives. It is also possible to control to display the subsequent subtitle text at a position in the subtitle display area different from the display position of the subtitle text without terminating the above. The configuration and layout of the screen at this time will be described later. In addition, the presentation control unit 83 passes the sound acquired from the moving image file 21 to the output unit 84 so as to be output from the sound output unit (speaker, earphone terminal, or the like) in synchronization with the video or the like.
The output unit 84 displays the image (video) passed from the presentation control unit 83 on the screen or the like. Further, the output unit 84 outputs the voice passed from the presentation control unit 83 from the voice output means.

  FIG. 10 is a schematic diagram showing a configuration example of a content presentation screen on the client device side. As illustrated, a display unit 101 is provided on the display surface side of the client device 71. As the display unit 101, for example, a liquid crystal display device, an organic EL display device, or the like can be used. Note that “EL” is an abbreviation for “electroluminescence”. In the illustrated example, the display unit 101 is divided into a plurality of areas, and the video and subtitle text are displayed in each area. That is, the display unit 101 is provided with the video display area 102 and the subtitle display area 103. The display unit 101 is composed of a large number of pixels, and a display control unit (not shown) appropriately divides the pixel region to enable control in each of the plurality of regions. In the figure, what is displayed in the image display area 102 is one frame of an image obtained by decoding the moving image file 21. Also, what is displayed in the caption display area 103 is a history of caption texts presented up to the present time (the time when the frame is presented).

  That is, in the example of the display method shown in FIG. 10, the presentation control unit 83 does not superimpose the subtitle text on the screen of the video (video display area 102) and displays the subtitle only area (subtitle display area 103). ) Is displayed. Further, the presentation control unit 83 does not erase the subtitle text from the subtitle display area 103 even when the presentation end time of the subtitle text arrives, and continues to display it as a history of the subtitle text. Then, the new subtitle text when the presentation start time has arrived is additionally and sequentially displayed. In this example, new subtitle text is additionally displayed below the subtitle text that has already been displayed. As a result, the user can visually recognize the subtitles by going back in time from the currently reproduced scene. Further, the presentation control unit 83 may perform a display method such that the caption display area 103 can be scrolled up and down by a user's operation. This allows the user to refer to the history of subtitle text in a wider range over time.

  FIG. 11 is a schematic diagram showing an example in which the client device displays subtitles in a mode different from the previous mode. Also in this figure, subtitle text is displayed in the subtitle display area 103. However, in the example of the figure, the presentation control unit 83 displays the subtitle in a form associated with the icon corresponding to the speaker. In this example, the icons of two speakers are displayed. The presentation control unit 83 also controls the subtitle text to be displayed in the balloon graphic. Further, in this example, the presentation control unit 83 displays the subtitle text in time series in the vertical direction of the screen. What is displayed on the upper side of the screen is older (past direction) subtitle text at the presentation time, and what is displayed on the lower side of the screen is newer subtitle text at the presentation time. Further, in the display example shown in the figure, a scroll bar 111 is also displayed so that it can be moved up and down in the subtitle display area 103 by a user operation. An example of the subtitle text shown in the figure is as follows. First, the subtitle text "AIUEO" is displayed in association with the icon (left side) of the first speaker. Next, the subtitle text "Kakikuke" is displayed in association with the icon (right side) of the second speaker. Next, the subtitle text "See also tomorrow" is displayed in association with the icon (left side) of the first speaker. Next, the subtitle text “See also tomorrow” is displayed in association with the icon (right side) of the second speaker.

In order to perform the display of FIG. 11, the presentation control unit 83 uses the metadata sent from the content distribution server device 61 side. For example, this metadata accompanies the subtitle text and includes speaker identification information for identifying the speaker. Further, this metadata includes the data of the icon image or the data of the URL from which the icon image can be obtained in association with the speaker identification information. This metadata is sent from the content distribution server device 61 to the client device 71 while being stored in the subtitle file 23. Further, this metadata is included in the original broadcast signal, and the data generation unit 13 of the caption data generation device 1 generates the caption file 23 so as to inherit the metadata.
Note that the display position in the subtitle display area 103 may be used as the metadata instead of the speaker identification information described above.

  That is, in order to realize the display method shown in the figure, the presentation control unit 83 uses the speaker identification information (speaker ID or speaker) that specifies the speaker corresponding to the caption text from the information included in the caption file. (The icon image location information of the speaker's icon image, etc.) is acquired, and the subtitle text is displayed so as to be associated with the speaker identification information.

Next, a method for realizing a content fast rewind operation in the client device 71 will be described.
In the client device 71, for example, when the user (viewer of the content) designates (selects) any subtitle text displayed on the screen, the video is rewound to the point in time of the presentation time of the subtitle text. You can In order to instruct the subtitle text, the user can operate a pointing device such as a mouse or touch the touch panel. When these operations are performed by the user, the presentation control unit 83 acquires information on the coordinates of the designated position on the screen. Then, the presentation control unit 83 identifies which subtitle text is instructed from the coordinates. Then, the presentation control unit 83 refers to the playlist file 24 to acquire the presentation time of the instructed subtitle text, and specifies the moving image file of the same segment as the subtitle file. In this way, the presentation control unit 83 rewinds the content, and restarts the presentation of the moving image file 21 and the subtitle file 23 from the rewinded position. That is, when the presentation control unit 83 receives an operation of selecting the displayed subtitle text, the presentation control unit 83 rewinds to a position corresponding to the presentation time of the subtitle text, and restarts the presentation of the moving image file from the position. Control. In this case, the presentation control unit 83 may instruct the decoding unit 81 to decode again the moving image file corresponding to the fast-returned position. Alternatively, the decoded video may be stored in the temporary storage means, and the video may be reread from the temporary storage means to realize the fast-reverse reproduction.
The fast-reverse operation described here can be performed both in the subtitle display in the form shown in FIG. 10 and in the subtitle display in the form shown in FIG.

  According to the client device 71 of the present embodiment, the user can later read and confirm the captions displayed in the past. At this time, the presentation time can be traced back by appropriately scrolling the subtitle display area. Furthermore, by linking the position of the moving image corresponding to the position of the subtitle text from the subtitle text portion, it is possible to perform fast-rewind reproduction of a missed scene.

  The functions of the caption data generation device, the content distribution server device, the client device, and the like in the above-described embodiments may be realized by a computer. In that case, the program for realizing these functions may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read by a computer system and executed. The “computer system” here includes an OS and hardware such as peripheral devices. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, the "computer-readable recording medium" means that a program is dynamically held for a short time like a communication line when the program is transmitted through a network such as the Internet or a communication line such as a telephone line. In this case, it is possible to include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be one for realizing some of the functions described above, and may be one that can realize the above functions in combination with a program already recorded in the computer system.

[Modification]
In the above embodiment, the playlist file is generated and distributed together with the moving image file and the subtitle file. However, as a modification, it may be implemented without using the playlist file. When the playlist file is not used, information for specifying the sequence number of the moving picture file is stored inside the moving picture file. In addition, information for identifying the sequence number of the subtitle file is stored inside the subtitle file. By generating such a file and distributing it from the content distribution server device, each of the moving image file and the subtitle file can be presented in the correct order on the client device side. When the playlist file is not used, the information regarding the presentation timing is stored inside the moving image file and the subtitle file. As a result, on the client device side, it is possible to synchronize the moving image file and the subtitle file and present them at an appropriate timing. In this case, the data generation unit in the caption data generation device does not generate the playlist file, and the output unit does not output the playlist file. Further, the content distribution server device does not distribute the playlist file.

  Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes a design and the like within a range not departing from the gist of the present invention.

  INDUSTRIAL APPLICABILITY The present invention can be used in a business of distributing contents, a business of manufacturing and selling devices for the same, and other businesses.

1 Subtitle Data Generation Device 11 Subtitle Extraction Unit 12 Subtitle Conversion Unit 13 Data Generation Unit 14 Output Unit 20 Storage Unit 21 Video File 22 Playlist File 23 Subtitle File 24 Playlist File 51 Time Code Inserter 52 Distributor 53 Encoder Device 61 Content Distribution server device 71 Client device (content display device)
79 communication unit 80 storage unit 81 decoding unit 82 caption processing unit 83 presentation control unit 84 output unit 100 distribution system

Claims (4)

The subtitle text and the presentation time information of the subtitle text are obtained from the subtitle data extracted from the vertical blanking area of the HD-SDI or SD-SDI of the broadcast signal obtained from the outside, and the subtitle text is set as the presentation time. A subtitle conversion unit that outputs in association with
A storage unit that stores a moving image file encoded based on the broadcast signal,
A data generation unit that generates a subtitle file including the subtitle text so as to be synchronized with the presentation time of the video file stored in the storage unit,
An output unit that outputs the moving image file read from the storage unit and the subtitle file generated by the data generation unit;
A subtitle data generation device comprising: The moving image file is a plurality of moving image files divided into segments of a predetermined length of time,
The storage unit further stores a moving picture playlist file that is data of a playlist including information of presentation time of each moving picture file for presenting the plurality of moving picture files in an appropriate order,
The data generation unit generates a plurality of subtitle files respectively corresponding to the plurality of video files while referring to the video playlist file, and a playlist for presenting the generated subtitle files in an appropriate order. It further generates a subtitle playlist file that is data,
The output unit further outputs the video playlist file and the subtitle playlist file,
The caption data generation device according to claim 1, wherein When the subtitle text includes an external character, the subtitle conversion unit outputs subtitle text in a form in which the location information of a font corresponding to the external character is associated with the external character.
The caption data generation device according to claim 1 or 2, characterized in that. Computer
A caption extraction unit for extracting caption data from a vertical blanking area of HD-SDI or SD-SDI of a broadcast signal acquired from the outside,
A subtitle conversion unit that acquires subtitle text from the subtitle data and information about the presentation time of the subtitle text, and outputs the subtitle text in association with the presentation time.
A storage unit that stores a moving image file encoded based on the broadcast signal,
A data generation unit that generates a subtitle file including the subtitle text so as to be synchronized with the presentation time of the moving image file stored in the storage unit,
An output unit that outputs the moving image file read from the storage unit and the subtitle file generated by the data generation unit,
Program to function as.

Images